Why Sponsor Oils? | blog | oilshell.org

The Simplest Explanation of Oil

I'm writing a series of blog posts about Oil, and I want new readers understand what it is. I've tagged past explanations #why-a-new-shell, but they require background knowledge.

This tweet by Vanessa McHale is more relatable. It shows the frustration I felt working with shell for the first time 15 years ago, which many users still feel.

not wrong pic.twitter.com/h1DTpeXiWJ

— Vanessa McHale (@vamchale) November 2, 2019

(bigger image)

Productive Programmers Secretly Use Shell

For context, my second job was at Google, and when I joined I had never used Unix! I grew up with Windows, and my first job was in console game development. In college, I liked theory more than programming.

So I had never seen a shell script. But in the next decade I noticed that many programmers at Google leaned heavily on Unix and shell. Learning those dark arts helped me "get things done".

Oil Might Look Familiar

But shell is unnecessarily hostile, and Oil aims to fix that.

The glossary says that the Oil language is a new dialect of shell that's parsed and evaluated like Python or Javascript, as opposed to being a "macro processor".

Responding to the frustrations in the tweet will make that more concrete.

Assignment: Oil has keywords and expressions

Whitespace matters in shell, so these two lines have totally different meanings:

sh$ a=b    # assign variable 'a' to string value 'b'

sh$ a = b  # run command 'a' with 2 args: '=' and 'b'
a: command not found

I thought this was ugly 15 years ago. It's still annoying, if only because it's inconsistent with every other language I use.

Oil looks more familiar:

oil$ var a = 'b'            # declare and initialize
oil$ setvar a = "hello $a"  # mutate it

The keywords var and setvar change the lexer mode and invoke an expression parser:

sh$  x=$(( 42 + a[i] ))  # shell looks funny

oil$ var x = 42 + a[i]                # Oil is familiar
oil$ const n = len(L) + D['key'] + 1  # and richer

More about Python-like expressions below.

Oil uses curly braces rather than if, fi

To delimit blocks, shell uses do and done, if and fi, etc.

for path in /bin /tmp; do
  if test -d "$path"; then
    echo "$path is a directory"
  fi
done
# =>
# /bin is a directory
# /tmp is a directory

Oil uses curly braces everywhere, but otherwise it's still shell-like:

for path in /bin /tmp {
  if test --dir $path {  # It also has long flags
    echo "$path is a directory"
  }
}

Prefer Oil's function calls to ${var%arg}

In shell, cryptic punctuation within ${} is used for simple string manipulation:

sh$ path=/bin/foo.py

sh$ echo ${path#/bin/}  # strip a prefix with #
foo.py

sh$ echo ${path%.py}    # strip a suffix with %
/bin/foo

Oil still uses $path or ${path} to refer to a string variable, but operations are expressed with function call syntax:

oil$ var path = '/bin/foo.py'

oil$ echo $lstrip(path, '/bin/')   # strip a prefix
foo.py

oil$ echo $rstrip(path, '.py')     # strip a suffix
/bin/foo

See Appendix A for more problems with ${}.

(The previous examples work with the latest release, but this one doesn't. Function calls are parsed and evaluated, but the strip functions don't exist yet.)

Words aren't split by default, so they don't need quotes

Oil fixes what I call the "!QEFS problem". Shell forces you to quote everything:

sh$ path='my blog post.txt'

sh$ ls $path     # wrong: ls will be passed 3 args, not 1
ls: cannot access 'my': No such file or directory
ls: cannot access 'blog': No such file or directory
ls: cannot access 'post.txt': No such file or directory

sh$ ls "$path"   # quoting disables word splitting
-rw-rw-r-- 1 andy andy  2091 Jan 13 01:51 my blog post.txt

Oil doesn't require quotes:

oil$ var path = 'my blog post.txt'

oil$ ls $path  # no splitting occurs
-rw-rw-r-- 1 andy andy  2091 Jan 13 01:51 my blog post.txt

But you can explicitly call @split() if you want:

oil$ lines @split(path)  # function that prints one arg per line
my
blog
post.txt

The @ sigil means "array". In shell, word splitting is a poor substitute for arrays, but Oil has proper arrays.

The array literal syntax is %(...), and you splice words into a command with @myarray:

oil$ var posts = %('my blog post.txt' 'other post.txt')

oil$ ls -l @posts
-rw-rw-r-- 1 andy andy  2091 Jan 13 01:51 my blog post.txt
-rw-rw-r-- 1 andy andy    74 Jan 19  2019 other post.txt

More: Oil Doesn't Require Quoting Everywhere (April 2021)

Python-like expressions inside () (not -lt and -gt)

As mentioned this in 2019 FAQ, Oil has Python-like expressions. Shell's style is unfamiliar to Python and JavaScript users:

if [ "$x" -gt 0 ]; then
  echo "$x is positive"
fi

Oil is more familiar:

if (x > 0) {
  echo "$x is positive"
}

I'm still looking for feedback and contributions to Oil!

Oil Is For Programmers Who Avoid Shell

This small exchange on lobste.rs also explains the project:

Oil is also aimed at people who know say Python or JavaScript, but purposely avoid shell

I’m one of those people! I didn’t realize I was in the target group for Oil until now :)

And For Non-Programmers

Many Unix users learn shell before they learn to program. It's a gateway into programming:

1. The REPL is a proven interface for fast experimentation.
2. Shell programs are often concrete, making them easy to read and trace. Your first programs don't need any abstraction, and you can print shell's simple data types.

If shell is useful in these ways, then Oil will be even more useful. It gives more errors, which aids learning, and those errors are more precise. I wrote about syntax errors a few years ago and runtime errors over the summer, but there are many more examples (#real-problems).

It's odd that shell is the first language you encounter on a Unix machine, but you're usually encouraged to "skip over" it and launch a different one!

Related: Shouldn't scripts over 100 lines be rewritten in Python or Ruby?

Summary

Vanessa's tweet is only a couple months old, but these problems have been bothering shell users for decades.

If you want to read more about Oil, subscribe to /r/oilshell or follow @oilshellblog. The next posts will review what the Oil project has achieved.

Thanks to Eric Higgins and Aaron Sokoloski for reviewing drafts of this post.

Appendix A: Problems with ${}

I want a smooth transition path from shell to Oil, so I implemented the syntax within ${} to run existing shell scripts. This led to several posts tagged #shell-the-bad-parts:

• The Five Meanings of #. And What Does ${####} Mean?
• Four Slashes and Three Meanings

Appendix B: It's Not Just Syntax

I wrote about other syntactic issues, like static vs. dynamic parsing, but shell has many semantic problems too.

I want to write a post called Shell Has Context-Sensitive Evaluation based on this thread. Summary: it's hard to explain what "$@" and "${myarray[@]}" mean in shell because it depends on where they're used.

There are many more syntactic and semantic problems on the Shell WTFs wiki page. I'll write about some of them, but it's more important to me to get Oil done.