Why Sponsor Oils? | blog | oilshell.org

Four More Posts in "Shell: The Good Parts"

This is an outline of the most important series in this blog. As I wrote on Hacker News, I started this project nearly 4 years ago with a plan:

1. Explain the problems with shell, motivating the need for a new one. I tagged these posts #shell-the-bad-parts and #why-a-new-shell.
2. Write a new but compatible shell called Oil.
3. Explain the good parts of shell with #shell-the-good-parts. Circulate these posts and let people try the good parts without the bad parts.

Unfortunately there are only two posts in that series, written over 3 years ago. Step 2 is taking longer than expected!

So while I'm chipping away at Oil's implementation, here are the main ideas. These ideas should be useful for experienced shell users, and I hope to explain them more carefully to non-shell users later in 2020.

Thanks to reader enriquto for reminding me to write this.

Each section below describes an article I'd like to write. The heading is the proposed title, and the body has the main ideas and important URLs.

Semi-automation With run.sh Scripts

• Every time I "fuck up", I add a few lines to a shell script. A comment on How to fuck up software releases.
• Shell makes problems go away. Semi-automation is how you gradually achieve automation. It's a mistake to try to automate too much too soon.
• run.sh scripts consist of a bunch of shell functions followed by "$@". That idiom runs the function $0 with arguments $1, $2, etc.
  • Typical functions are build, test, and deploy. There are dozens of shell scripts in the Oil repo that follow this pattern.
  • They're similar to Makefiles with .PHONY targets, but I prefer shell.

I used to call these scripts do.sh, but that name shares a prefix with the common doc/ directory, creating a speed bump for autocompletion.

2/26 update: I'm not the only one who advocates this pattern:

• mbland/go-script-bash is a framework for such test, deploy, etc. functions, offering help and autocompletion. Coincidentally, the author was my officemate at Google about 13 years ago. (Hi, Mike!) Back then, I was working more on migrating away from shell than migrating toward it :) He cites these posts:
  • In Praise of the ./go Script - Part I
  • In Praise of the ./go Script - Part II
• adriancooney/Taskfile is a template (not a framework) which is almost identical to what I do with run.sh.
  • Introduction to the Taskfile format (medium.com)

March 2022 Update:

Github has a related pattern called Scripts to Rule Them All (2015):

Being able to get from git clone to an up-and-running project in a development environment is imperative for fast, reliable contributions.

• Sample repo: https://github.com/github/scripts-to-rule-them-all
• Real example in their linguist repo: https://github.com/github/linguist/tree/master/script

The $0-Dispatch Pattern Solves Three Important Problems

1. A shell script is a simple way of documenting project-specific dev tools. That is, if you have a text file that describes a bunch of shell commands (e.g. a "playbook"), you can turn it into a shell script:

   • Add # before the descriptions, turning them into comments.
   • Organize commands into functions so that each step is executable on its own.
   • This is precisely the run.sh pattern from the last post. It's nice to have executable documentation, even if it's only semi-automated.

2. The pattern also lets you expose shell functions to tools like xargs / find -exec, and sudo / chroot.

   • This style of composition is possible because shell functions are isomorphic to processes. (In Oil, they're called "procs" and defined with the proc keyword, since they're unlike functions in Python or JavaScript.)
   • It's also closely related to the first two posts in #shell-the-good-parts:
     • Shell Has a Forth-Like Quality. For example, sudo $0 myfunc "$@" is concatenative.
     • Pipelines Support Vectorized, Point-Free, and Imperative Style. I often use the pattern cat tasks.txt | my-filter | xargs -- $0 myfunc "$@" — in Oil's own tests and benchmarks, for example.
   • Oil will have "blocks" to avoid some of the awkwardness of $0.
   • Aside: this pattern solves any problem that the bash misfeature export -f was supposed to solve. This construct caused the infamous ShellShock bug.

3. The $0-dispatch pattern solves the ignored errexit problem.

   • Replace if myfunc with if $0 myfunc. (This advice is probably new/unique. Leave a comment if you want details before I publish this post.)
   • 2/26 Update: This comment explains the problem and solution in more detail. I also wrote demo/dollar0-errexit.sh to demonstrate it.

Related points:

• Benchmarks are naturally shell scripts because you're gluing together heterogeneous programs. For example, I compare Oil vs. other shells, or an old version vs. a new version of Oil.
• If you're not using more than one test framework, you're probably not testing enough. Oil uses many kinds of tests, and the results are published with each release.

Web Sites Are Naturally Made With Shell Scripts

An HTTP daemon is such a simple thing that a simple shell script will often suffice.

— Tim Berners-Lee, inventor of the World Wide Web

• http://info.cern.ch/hypertext/WWW/Provider/Overview.html: This page shows that the early web was "just" a shim over heterogeneous collections of files.
• http://info.cern.ch/hypertext/WWW/Provider/ShellScript.html: HTTP 1.0 was trivial to process with /bin/sh. Berners-Lee parses the GET verb and a URL with the shell's read builtin, and cats the file back to the network. (This practice ignores escaping, which I wrote about in 2017.)

The Unix philosophy of combining heterogeneous tools remains useful and powerful today. Building sites with tools rather than libraries leads to efficient, flexible, and loosely coupled code.

I built this website with classic Unix tools and my own tools written in Python. The main() is always in shell, not Python.

Some direct analogies between HTML and shell:

• HTML is to CSS/JavaScript/WebAssembly as Shell is to sed/awk/make.
• HTML files and shell scripts both talk about hierarchical paths. The Web adds the crucial concept of a hyperlink.
• Data URLs are like here docs.

The Alternative Shell Challenge

I've noticed that some alternative shells on the wiki don't have all the good parts of shell. This is often because they don't directly control the file descriptor table of a process, which is done with low-level system calls like dup2() and fcntl().

That is, they aren't Unix shells in the classic sense.

A couple years ago, I wrote a challenge for alternative shells (and Perl). How do you express this simple program?

f() {
  echo ---
  ls /
  echo ---
}

f > out.txt
f | wc -l

It's very related to the Git Log in HTML post, which adds proper escaping.

print-html-listing() {
  local dir="$1"
  
  echo '<html>'
  echo '  <body>'
  ls "$dir" | escape-html
  echo '  </body>'
  echo '</html>'
}

print-html-listing > listing.txt

I recently learned that Elvish does well on this challenge with a "virtual" file descriptor table. However, it doesn't have the POSIX semantics of "copying" the shell's memory with fork().

I'd like to do a survey of other shells. Let me know if you'd like to help test an alternative shell or scripting language.

What's Next?

This post is two levels deep on the "blog stack". After releasing Oil 0.8.pre1, I drafted an announcement:

1. Oil 0.8.pre1: Contributions and Metrics

But it needed context, so I started writing:

2. February Brain Dump: with links to Hacker News comments and Zulip threads.

This post summarized just one comment, mentioned in the first paragraph. The next two posts will pop the stack, summarizing more comments and describing the latest release.