Why Sponsor Oils? | blog | oilshell.org

Notes on the HotOS Unix Shell Panel

On Thursday, I (virtually) attended the HotOS 2021 panel on Unix Shell, and took notes on Zulip. This post summarizes them, and elaborates on comments I made during the panel.

• Event: The Future of the Shell: Unix and Beyond

The event gathered a diverse crowd of researchers from operating systems, programming languages, and HCI. Bells Labs legend Doug McIlroy and bash maintainer Chet Ramey were also there!

Some of these notes may not make sense to people who weren't there. Feel free to ask questions in the comments.

Be warned: this post started out as fairly neutral notes, but as I edited it, I inserted several pronouncements about shell, and Oil. I invite disagreements, since will make the project better and help explain it :-)

What I Took Away

The panel attendees weren't the usual audience of this blog (programmers and practitioners from Hacker News, Reddit, lobste.rs), but they have many of the same issues with shell.

So I feel like the event validated the motivations for Oil, which has been in progress for over 5 years now (for better or worse)! I still need to write that "project retrospective", and do a better job of opening it up to more contributors.

There's a Lot of Interest in Unix Shell

Not only was the group diverse, but it was big! Many researchers wanted to talk about shell.

There were also many threads online about the paper that accompanied this project. It was at the top of Hacker News, Reddit, and lobste.rs multiple times!

• Round one:
  • https://news.ycombinator.com/item?id=27036572 (I recommend some shell reading, including a Usenix paper on Korn shell)
  • https://lobste.rs/s/kw3yn0/once_future_shell_pdf (I submitted this)
  • https://old.reddit.com/r/ProgrammingLanguages/comments/nfcij6/unix_shell_programming_the_next_50_years_hotos/ (I make some comments about shell parallelization, and Nix)
• Round two:
  • https://lobste.rs/s/mtatsi/unix_shell_programming_next_50_years (interesting discussion about "content type" metadata for pipes here)
  • https://news.ycombinator.com/item?id=27378444 (familiar complaints, and the message of the Oil project is getting through)

Shell Is Hard To Use

The difficulty of learning and using shell was the overwhelming feedback I heard. (This may have been because I spent time in the 3rd breakout room, which had several HCI researchers.)

This of course isn't news to me -- in fact, it's the main motivation for the Oil project :-) But it was good to hear this feedback again from a different group.

I have two responses to this:

(1) Shell should be more like the dynamic languages that "won" (Python, JavaScript, Ruby, ...). As on Hacker News, there was talk about Python as a better shell, which I view as a fallacy. It's not "either-or". However, it's absolutely true that users try to "import" their Python knowledge into shell, fail, and become sorely disappointed.

Remember that Oil has a dual nature: it's an upgrade path from bash, but it's also a "clean slate" shell language. It's not obvious that this is possible, and I need to explain it more, but so far everything is working. In the last 2 months, I've gotten even deeper into the Oil language, and I still feel this way.

(TODO for me: write about 10 posts on many new features; write A Tour of Oil.)

A few days before the panel, I updated the Oil home page to reflect the "clean slate" viewpoint, in addition to the legacy viewpoint:

Oil is for Python and JavaScript users who avoid shell!

(2) Oil's new headless mode can (optionally) divorce the shell from the terminal, enabling better user interfaces. Shell is fundamentally a language-oriented interface, but it doesn't have to be a terminal-oriented one. That's sort of a historical accident.

This work is very recent, and I haven't written much about it. I'm excited about it because it simultaneously reduces the scope of the Oil project, while allowing other people to build interesting things on top. More on this below.

Statements by Panelists

There were statements by four panelists, and a few caught my interest:

• Diomidis Spinellis (prolific shell user, author, researcher on dgsh) said that shell is failing as a "socio-technical system" because it lacks ownership. He also made comments on the pedagogy of Unix shell.
  • I'd like his feedback on the Oil project in general, and I may point to some claims about POSIX in the Oil FAQ.
• Arjun Guha (researcher on Puppet formalization) said that shell was the wrong tool for many jobs, in part because language composition is string-based.
  • My contrarian viewpoint is that string-based composition has advantages (e.g. compared with other forms of metaprogramming; this isn't the whole argument).
  • It also has disadvantages, which Oil will mitigate with upcoming string safety features. For example, $[x] will force escaping through a function specified by _ESCAPE, similar to JavaScript's template string literals. This is in contrast to ${x}, which does raw string interpolation.
• Chet Ramey reminded everyone that shell has many users, and there's a great potential for breakage.
  • Oil's implementation addresses this issue. I believe it's less prone to breakage due to its principled parsing, detailed code representation with algebraic data types, and extensive test suite. That is, OSH runs unmodified bash programs, but the style of its implementation is completely different. It's our upgrade path to a better language and runtime. More on this below.

Comments by Doug McIlroy

McIlroy wasn't one of the four panelists, but he made interesting comments:

• He mentioned the ; and & combinators for shell, and asked about syntax for new combinators for parallelism.
  • Oil adds Ruby-like blocks to shell, and they can be used for different kinds of parallelism. At the very least, we should have an each builtin that does what xargs -P does, except that it takes a (statically-parsed) block, not a string.
• He's interested in "distributable" computing (coined by ?)
  • Distributed computing is one of my longstanding interests with shell, but it's slightly further in the future. Container support (storage and deployment) is a prerequisite for the mechanisms I have in mind. Shell also needs a notion of data dependencies; see Comments on Build Systems and CI Services.
• He asked about pipelines which have feedback (circular or nonlinear pipelines).
  • This isn't something that's come up in my shell usage so far, but I'd be interested in the use cases for it.

Related: I played with Doug's 2014 code a few months ago, but didn't blog about it!

• Coroutine and prime number sieve (PDF)
  • I noted in a January e-mail that the sieve has a variable number of parts (depends on N), whereas a Unix pipeline has a static number of parts (unless you use eval). I'm not sure if this justifies new language features, but it's worth thinking about.
  • A related 2016 interview with McIlroy (PDF)

My Messages for Researchers

Oil's Headless Mode Should Be Useful for UI Research

The headless name is an analogy to a headless browser, like Chrome. Just like you can have a web browser without the UI, you can have a shell without the UI.

I want the UI to (optionally) live in a separate process from the shell itself, which enables it to be written in a different language against a stable IPC interface.

I mentioned headless mode during the panel, but I don't think the message will get through without a concrete demo. Thankfully Subhav Ramachandran has started one! He also deserves a lot of thanks for motivating this work in the first place.

I need to write properly about this later, but here are some preliminary docs:

• https://github.com/oilshell/oil/wiki/Headless-Mode
• https://www.oilshell.org/release/latest/doc/headless.html -- Start bin/osh --headless and send commands over a Unix domain socket.

And comments on what you can do with it:

• https://lobste.rs/s/s2trbn/oil_language_variable_declaration#c_ixznii -- If you want a mental picture you can imagine a shell UI that looks like a browser – it has a command line that’s like a URL bar, and then each command is run in its own terminal. That’s not the only thing you could build, but it’s one of them.
• https://news.ycombinator.com/item?id=27055926 -- The headless mode provides a simple interface and allows integration that can't be done with bash (or any other shell AFAIK). (Conversation with the authors/founders of https://withfig.com/)

A GUI around a shell can show state. I believe that will help users considerably. Right now experienced shell users just "imagine" what state they are in, with a tiny bit of help from the prompt.

Let me know if this sounds interesting! I need help.

Oil Is Implemented Like a Programming Language

I suggest that Oil's implementation is more amenable to research hacking than bash, or even dash. (Although dgsh is notably a bash fork!)

A concrete way to see this is with osh -n:

# print the syntax tree (in abbreviated form)
$ osh -n -c 'echo "hello $name"'
(C {<echo>} {(DQ <'hello '> ($  Id.VSub_DollarName '$name'))})

Reasons:

• It's written in Python, not C. PaSh is also written largely in Python, presumably for similar reasons (fast prototyping).
• It has a strongly typed code representation via Zephyr ASDL (and MyPy). In contrast, shells like bash and dash use a homogeneous AST node type, stringly-typed words, and "control codes".
• The code is much shorter than bash's, but implements much of the same language, and more. To solve the speed problem, we're semi-automatically translating it to C++, which has produced promising results.

How Can Oil Support PaSh- and POSH-Like Annotations?

This issue was discussed multiple times during the panel: it's not possible to apply traditional compiler analyses to shell because external commands like grep are opaque to it. If you don't know what side effects happen, then you don't know if parallelization is safe, etc.

These recent research projects both rely on command annotations to solve that problem:

• PaSh: Light-touch Data-Parallel Shell Processing
• POSH: A Data-Aware Shell

Building such parallelization and distribution directly into Oil probably isn't in scope, but the annotations are (similar to Python supporting multiple external type checkers with its type annotation syntax). I filed this issue:

• Issue 867: Lessons from PaSh and POSH

I also wonder if Oil should expose its internals for metaprogramming:

• Issue 492: Expose ASDL structures in Oil itself

I've been working on features to let Oil evaluate untrusted config files, and some of that work should be useful for annotations. Annotations on proc arguments can also be used for docstrings, completion hints, and even types / content types.

Let me know what you think! The PB&J paper in the next section seems related to PaSh and POSH.

Papers to Read

There were pointers to recent shell research I wasn't aware of. It would be nice to have a "shell bibliography" on a Wiki page or in a git repo!

• PB&J: Easy Automation of Data Science/Machine Learning Workflows https://ieeexplore.ieee.org/document/9378128 (paywalled)
  • I've used shell in these areas, e.g. when working on Deep Learning With Differential Privacy, and numerous data science projects at Google.
• Empirical investigation of Command Line Customization https://arxiv.org/abs/2012.10206

Odds and Ends

• I talked about shell parsing with David Holland. You can contrast the language server / IPC strategy with a "declarative" syntax specification (yacc or Treesitter, the latter of which deals with incomplete code). I think IPC is cleaner from a software engineering POV, and it also lets you be aware of runtime state like alias.
  • Concretely, Oil's headless mode uses Unix domain sockets, a form of IPC. (dgsh also uses Unix domain sockets, and the paper mentions that they are underused!)
  • Not mentioned: I dreamt of creating my own parser generator "extracted" from Oil's style of #parsing-shell (lexer modes, etc.), but this probably won't happen.
• I believe the poor state of shell parsers is a "bottleneck" for implementing friendly UIs, and I discussed this with Michael Greenberg and Chet Ramey.
  • Unlike zsh and fish, bash defers its completion to the separate bash-completion project, which actually tries to parse bash in bash. This is a very bad idea, and is predictably filled with bugs. It has a poor experience which hasn't changed much in 10+ years.
  • I wrote about this in posts tagged #interactive-shell.
    • Oil Uses Its Parser For History And Completion
    • Oil's Parser Doesn't Worry About Aliases and Prompts
• Many people mentioned systems with REPLs that are more friendly and functional than the Unix shell, like TOPS20, VMS, and more.
  • Again, I think that the architecture of the shell itself matters. I'd be very surprised if anyone built a rich and reliable interactive system on top of bash's architecture (and I'd say ble.sh supports this). The dynamic nature of the language is a problem, but the structure of bash's code is also a problem. I say this after five years of implementing something similar, in a different style!
  • I believe zsh also has some architecture/implementation issues. I discussed some of them with zsh dev Oliver Kiddle (a couple years ago).
• Very important: I don't say these things to insult or discredit the work of others. Rather, I'm suggesting that we've collectively learned things in the last 30 years which can produce a better shell.
  • I've also pointed out Oil's own implementation issues many times on this blog. (I believe they're not "systemic", and I monitor them with #metrics).

I think I want to double down on this analogy:

Oil is to bash as LLVM/Clang is to GCC.

The LLVM project was initiated in the early 2000's. After several years, it increased in scope to the Clang compiler (C, C++, Objective C). About 10 years later, it enabled Rust, Julia, and Zig.

Analogously, I hope that there will great things built on top of Oil, particularly the new headless mode. But I know it will take a long time! I don't have the bandwidth to do it, so I want open up the project more.

Discussion: Does Shell Have A Heap?

This was a short but useful exchange on language design and the philosophy of the shell. Paraphrasing:

• Chet Ramey: Shell doesn't have or need a heap.
• Michael Greenberg: The heap is the file system.
• Me: Idiomatic shell scripts do use the file system, but shell should be more like the dynamic languages that won. Python and JavaScript both have garbage-collected heaps.
  • I did go back and forth on this issue early in Oil's design. Two things pushed it over the edge: JSON support, and using Oil as a syntax for config files (to address the "YAML problem").
  • Something I didn't mention: Arjun dislikes language composition with strings. But if you want to do better, i.e. represent and manipulate shell code for metaprogramming, you need recursive data structures. And thus you need a (garbage-collected) heap. This relates to issue 492, mentioned above.
  • The command annotations mentioned above might be recursive!

Conclusion

Thanks to Michael Greenberg, Konstantinos Kallas, and Nikos Vasilakis for organizing. This was a great event with a lot of energy!

I want to follow up with 2 or 3 people mentioned above. It also caused me to resume a conversation with Stephen Kell on text vs. binary data. (That reminds me that I still need to write a post on this comment: text as a narrow waist.)

I imagine that some parts of this post didn't made sense to people who weren't there, so feel free to ask questions (and disagree) in the comments.