Why Sponsor Oils? | blog | oilshell.org

Recent Progress on the Oil Language

After garbage collector progress in Oil 0.8.8, and addressing user feedback in the Oil 0.8.9, I started working on the Oil language again. For context, the language has only existed since 2019, despite the fact that the Oil project as a whole is 5 years old.

I ended up making rapid progress in several areas, and took notes on Zulip. This post expands on the thread Notes for Oil 0.9.0 (requires login).

I expect 0.9.0 to be one of the biggest releases ever!

Rather than fully explaining each feature, this post enumerates #blog-topics. Not all the work is done, but I want to broach these topics now so the code doesn't get too far ahead of the blog.

As always, feel free to ask questions in the comments.

Posts That Are Ready To Write

The features in this section are mostly done, but still need user feedback.

New Features

• Oil is a Headless Shell. I expect this enhancement to the #shell-runtime to be a huge part of the project. I first mentioned it in Notes on the HotOS Unix Shell Panel a few days ago. See my comments there.

• Oil Has Python-Like Multi-Line Strings, But Enhanced Like Julia.

  • I found a clean way to parse triple quoted strings (''' and """). I mentioned this possibility in November's Proposed Changes to Oil's Syntax.
  • The rule to strip leading whitespace is similar to that of the Julia language.
  • As a result, here docs are deprecated in the Oil language! I always found that syntax confusing, and multiline strings are more orthogonal -- with respect to whether $var interpolation is allowed, and how whitespace is stripped.

• Oil Has a Lightweight Module System.

  • We now have a module builtin. This feature is about organizing code into files, rather than "modularity" per se. The natural unit of composition in the Oil is still the proc (shell function).
  • We also have a runproc builtin to help with the "task file" pattern. (related to Shell Scripts Are Executable Documentation).
  • These mechanisms are very lightweight! Originally I thought we needed a new use mechanism, but Oil's modules are built on top of source. This is better because any new feature must compose with all the old features.
  • We don't even have namespaces, although they could be added if proven necessary. The goal is to prevent basic coding mistakes when organizing a large script into files.
  • TODO: we should also make sure the module mechanism can be statically analyzed by tools for single-file deployment.

Oil's Solutions To Real Problems

I use the #real-problems tag to describe Oil's solutions to problems found "in the wild" (via user feedback and production code).

• Relative Imports

  • As part of the module work above, I want to implement $_this_dir. For details, see issue 587, and the linked Hacker News complaint. (These complaints are absolutely true and valid, but also feel like "groundhog day".)

• Oil Fixes Shell's Error Handling. This post can start from my Hacker News comment demonstrating an old yet surprising "feature" of set -e / errexit.

  • I enumerated fixes to error handling in Four Features That Justify a New Unix Shell (October 2020). But that post lacked examples, which means that not everyone understood it.
  • Some readers on lobste.rs have found these claims offputting, so I want to a good job of explaining it clearly.
  • Related: The run builtin was renamed to try. It doesn't work like try/catch, but I think try suggests that the builtin is related to error handling, unlike run.

Real Problems From Chris Siebenmann

Chris Siebenmann prolifically documents shell issues on his blog, a rich source of #real-problems.

• Oil Fixes the SIGPIPE Problem
  • I replied to Unix shell pipelines have two usage patterns with the solution using the old run builtin.
  • After some thinking, and consulting with the authors of NGS and Elvish, I changed the mechanism to a shell option:
    • shopt --set sigpipe_status_ok, which is on in option group oil:basic.

He also wrote that The Bourne shell and Bash aren't the right languages for larger programs, and I agree. Oil addresses all of his complaints, though it's not all documented:

• Oil Has a Lightweight Module System (mentioned above)
  • He says that dkms (a program to manage Linux kernel modules) is a 4000-line shell script, which is hard to manage in a single file.
• Oil Fixes Shell's Error Handling (mentioned above)
  • He notes problems with shell and bash error handling, as many others have.
• Oil Procs Have Named Arguments
  • He doesn't like shell's argument passing with $1, etc., and I agree.
• Oil has Python-Like Expressions
  • He dislikes "outsourced functionality", i.e. reliance on test and expr.
  • Related docs Command vs Expression Mode, Oil vs. Python (rough draft)
• Oil Has Recursive Data Structures
  • He points out shell's lack of data structures, as many others have.
  • Oil fixes the behavior of bash's arrays and associative arrays, and allows them to be recursive. This requires a garbage collected heap (discussed at the end of the HotOS notes).
  • In Oil, values are tagged with types, not locations for values (cells). This is how Python and JavaScript work. Remember, they're "the dynamic languages that won".

Here's a 2008 post from Chris's blog saying essentially the same thing:

• The Bourne shell is not a programming language

Unfortunately I don't think there's a really good Unix programming language to replace the Bourne shell, which is one of the reasons that writing programs in the Bourne shell remains so tempting.

So that's why I say that these posts feel like "groundhog day", despite being valuable and correct. Instead of documenting shell's limitations for future generations, we should fix shell. That's not easy, as it has to be done in a compatible way.

But this is the purpose of the Oil project! Below I mention A Tour of Oil, which will show that Oil isn't encumbered by much legacy at all. (We extend bash's shopt mechanism to achieve this.)

More Blog Updates

I want to take another stab at Python vs. Shell question in the Oil FAQ. It's a persistent question, and came up again in the HotOS Slack channel.

This short response that found some agreement, and I will incorporate it into the FAQ:

I would say the biggest misconception is that it's either/or. Python and bash go well together. The basic calculus is to write 200 lines of shell calling 200 line Python script, rather than 1000 lines of Python, and the whole thing runs faster.

Work in Progress

The work in this section isn't done yet, but thinking about how to describe it is useful. I like documentation-driven development!

Educational Posts to Support String Safety

Shell is fundamentally string-based, which is error prone. Oil adds mechanisms to make this style correct. String safety is just as important than memory safety (which Rust has raised awareness of).

These mechanisms deserve a few pedagogical posts as background. Over years of professional programming, I've noticed that seemingly simple things like backslash escaping cause confusion in practice!

• Three Examples of Attacks: Shell, SQL, and HTML Injection (XSS)
  • Showing 3 concrete examples will be a good introduction to the concept of string safety.
  • I posted xkcd's 2007 Bobby Tables comic recently, which is a mnemonic for SQL injection. I suggested that we need The Ballad of Rimraf as a mnemonic for shell injection.
• How to Avoid Shell Injection Attacks. A practical guide for multiple languages.
  1. Don't invoke the shell! Pass an argv array, e.g. using Python's subprocess.call().
  2. Use a correct escaping function, i.e. the '\'' trick.
  3. Don't generate shell code. Use shell interpolation "at runtime", e.g. sh -c 'echo "$1"' dummy0 "$untrusted"
  4. Refer to the Julia blog posts that matklad has advocated.
• String Quoting Across Languages. It doesn't help that there are so many ways to quote strings! This could be a nice reference and comparison.
  • C-style with backslashes: Python, JavaScript, QSN, etc.
  • Shell: It has unusual quoting rules, which leads to an unusual '\'' solution. I learned about this from contributor Greg Price a few years ago!
  • Ninja: Double your dollar signs like $$, like Make.
  • SQL: Double your single quotes like ''.
  • HTML: & and so forth.
  • CSV: I think it has 2 methods of escaping. I'm confused.
• Oil's String Safety Features (not implemented yet)
  • $[x] and _ESCAPE. Mentioned in the HotOS notes.

The YAML Problem

I mentioned this problem in Comments on Build Systems and CI Services, and I've made some progress recently.

I also call it the Shell-in-{YAML,Docker,systemd,Chef,Vagrant,Python} problem, and mentioned it at the end of my BayLisa 2019 Notes.

• Naming is the hardest part :-)
  • bin/oven is a sandboxed interpreter that will evaluate Oil and print JSON (which valid YAML), Ninja, and more. It's like bin/oil, but it can't do any I/O. It can't use the kernel at all.
  • It used to be COWS -- Configuration in Oil Without Side effects -- but I decided I don't like that connotation. And I don't want a .cows file extension; the config files should still end with .oil.
• Mechanism: I implemented use dialect ninja, which requires the "shvar" _DIALECT=ninja. A shvar is a dynamically-scoped variable that the shell itself uses, akin to the boolean shopt mechanism. More on this below.
• TODO: We need APIs like block_to_dict(), block_source_code()

The declarative part of shell can be used for multiple purposes:

• For config files that embed shell: distro package definitions (e.g. Gentoo ebuilds), CI services, image definitions like Docker, service definitions like systemd, etc.
• Exporting shell code + annotations to compilers like PaSh and POSH. (This work is further in the future.)

Official Docs

I started docs on these features and need to finish them:

• doc/strings.md: Strings: Quotes, Interpolation, Escaping, and Buffers
• doc/modules.md: Oil Modules Separate a Program Into Files

The most important doc is:

• A Tour of Oil: Explain Oil as a clean slate language. I think many users don't understand the "upgrade path from bash" framing. Both views of the project are important.
  • Thanks to Raphael Megzari for starting and motivating this, as well as several other fixes to the Oil language I didn't mention!

Language Concepts

I notice that I'm reusing shell mechanisms to turn OSH into Oil, like shopt and source. This is good because it keeps the language smaller.

These generic concepts/mechanisms are an orthogonal axis to features like the module system and declarative configuration. I want to explain:

• shopt and shvar: Shell Options and "Stack-Oriented" Variables (similar to dynamic scope)
  • shopt: booleans like simple_word_eval, which are organized into groups like oil:basic
  • shvar: I recently added this builtin, and I like how it didn't add very much new to the execution model of shell!
    • Examples of shvars: IFS, _DIALECT, _ESCAPE
• Registers. I started calling silently-mutated variables "registers", and they share a PHP-like naming convention:
  • _status aka $?
  • _this_dir for relative imports (mentioned above)
  • _buffer for the new "builtin sub" feature
• The append builtin vs append() function
  • Again, Command vs. Expression Mode
  • Future work: Uniform function call syntax, like D.

What's Next?

This post helped me write my TODO list:

• Implement and document $_this_dir, which addresses #real-problems.
• Take another stab at the "Python vs. shell" FAQ.
• Write A Tour of Oil: the clean slate perspective.
• Implement config file evaluation with bin/oven. This is the missing "declarative part" of shell.

When drafting this post, many other #blog-topics came up, like those related to distributed systems, #software-architecture and dev tools. I've moved those to the next post.

With apologies for these dense notes, let me know if you have questions!