source | all docs for version 0.8.1 | all versions | oilshell.org

Warning: Work in progress! Leave feedback on Zulip or Github if you'd like this doc to be updated.

Oil Language Design Notes

These informal notes may help you understand Oil.

It Mostly Borrows From Other Languages

Oil's design is more conservative than that of other alternative shells. It

• Preserves what works best about shell (processes and files, pipelines)
• Cleans up the sharp edges like quoting, ad hoc parsing and splitting
• Integrates features from:
  • Related languages like awk and xargs (in-progress)
  • Popular dynamic languages like from Python and JavaScript.

Influences

POSIX Shell

The command and word syntax comes from shell:

ls | wc -l                       # pipeline
echo $var "${var} $(hostname)"   # variable and command sub
echo one; echo two               # sequence of commands
test -d /tmp && test -d /tmp/foo

Oil's own shell-like extensions:

echo $[1 + 2*3]                  # Expression substitution
echo $strfunc(x, y)              # Inline Function Calls

bash and ksh

We implement many bash semantics, like "named references" for out variables:

f() {
  local -n out=$1    # -n for named reference
  out=bar
}

myvar=foo
f myvar
echo $myvar          # prints "bar"

But clean up the syntax:

proc f(:out) {       # "out param" declared with :
  setref out = 1
}

var myvar = 'foo'
f :myvar             # caller prefixes the var name with :
echo $myvar          # prints "bar"

Historical note: Usenix 93. korn shell was used for GUIs and such!

Python

Oil's expression language is mostly Python compatible. Expressions occur on the RHS of =:

var a = 1 + a[i]
var b = fib(10)
var c = 'yes' if mybool else r'no'

Proc signatures resemble Python:

proc cp(src, dest='/tmp') {  # Python-like default value
  cp --verbose $src $dest
}

Differences:

• Oil uses shell-like composition with "procs", not Python- or JavaScript-like functions.
• Oil doesn't overload operators as much:
  • a + b is for addition, whle a ++ b is for concatenation.
  • a < b is only for numbers. cmp() could be for strings.
• No "accidentally quadratic"
  • Strings and +=
  • No in for array/list membership. Only dict membership.
• s[i] returns an integer "rune", not a string?
• Singleton tuples like 42, are disallowed, in favor of the more explicit tup(42).

JavaScript

Oil uses JavaScript's dict literals:

# Unquoted keys.  Sigil to avoid confusion with blocks.
d = {name: 'Bob', age: 10}

d = {[mystr]: 'value'}  # key expressions in []

And "structured programming" looks like C / Java / JavaScript:

if (x > 0) {
  echo 'positive'
} else {
  echo '0 or negative'
}

var x = 5
while (x > 0) {
  echo $x
  setvar x -= 1
}

Ruby

Oil has Ruby-like blocks:

cd /tmp {
  echo $PWD  # prints /tmp
}
echo $PWD

proc foo (&block) { run(block) }
var myblock = &(echo $PWD)

(Julia has something like blocks too.)

And the syntax for references to variable names:

read :line                   # populate $line variable
push :myarray one two three  # append to array

Perl

The @ character (which PowerShell also uses) comes from Perl:

var myarray = %(one two three)
echo @myarray          # @ is the "splice" operator

echo @arrayfunc(x, y)

for i in @(seq 3) {    # split command sub
  echo $i
}

Perl can be viewed as a mixture of shell, awk, and sed. Oil is a similar agglomeration of related languages.

Go (library)

The builtin flags syntax comes from Go:

 mybuiltin --show=0  # turn a flag that's default true

It also uses a native UTF-8 representation of strings.

Awk

Oil gets its regex match operator from Awk:

if (mystr ~ /digit+/) {
  echo 'Number'
}

(We don't use Perl's =~ operator.)

make

TODO

find and xargs

Features influenced by these languages are planned, but not implemented.

Minor Influences

Tcl

Oil uses proc and set, which makes it look something like Tcl:

 proc p(x) {
   set y = x * 2
   echo $y
 }

 p 3  # prints 6

But Oil isn't homoiconic like Tcl is. It avoids dynamic parsing, and has a lot of syntax.

So this is superficial, an instance of "convergent evolution".

However, Data Definition and Code Generation in Tcl (PDF) shows how Tcl can be used a configuration language:

change 6/11/2003 {
  author "Will Duquette"
  description {
    Added the SATl component to UCLO.
  }
}

Oil's blocks (also inspired Ruby) would allow this to be expressed very similarly:

change 6/11/2003 {
  author  = "Will Duquette"
  description = '''
    Added the SATl component to UCLO.
  '''
}

(This mechanism is still in progress.)

Lua

Oil also uses a leading = to print expressions in the REPL.

= 1 + 2

PHP

PHP has global variables like _REQUEST and _POST.

Oil may have _argv, _match(), _start(), etc. These are global variables that are "silently" mutated by the interpreter, and functions to access such global data.

Paradigms and Style

Shell is already mix of:

• dataflow: concurrent processes and files, pipelines
  • instead of Clojure's "functions and data", we have "processes and files". Simple. Functional. Transforming file system trees is a big part of containers.
• imperative: the original Bourne shell added this.
  • "functions" are really procedures; return
  • iteration constructs: while / for / break / continue
  • conditional constructs: if / case

Oil is:

• getting rid of: ksh. Bourne shell is good; ksh is bad because it adds bad string operators.
  • ${x%%a} ${x//} getting rid of all this crap. Just use functions.
  • korn shell arrays suck. Replaced with python-like arrays
• Add Python STRUCTURED DATA.
  • the problem with PROCESSES AND FILES is that it forces serialization everywhere.
  • Structured Data in Oil
• Add declarative paradigm to shell.
  • Package managers like Alpine Linux, Gentoo need declarative formats. So do tools like Docker and Chef.
• Language-Oriented -- internal DSLs.