Why Sponsor Oils? | blog | oilshell.org

Summer Blog Backlog: Understanding and Using Shell

Last month, I published Recent Progress on the Oil Language, which enumerated #blog-topics on the Oil language. It outlined posts on Oil's improvements over shell, as well as educational posts on concepts like string safety.

I started this post with remaining topics that aren't about the Oil language, and it ended with a focus on understanding and using shell in general. I list drafts and other topics as well.

As you can see, the blog is backlogged by 10 to 20 posts. This has happened before, so I think it makes sense to "declare bankruptcy" every year :-)

Let's go through the topics, and then I'll figure out what I can realistically write. If I have time in the future, I can return to some of them . In the meantime, I can refer to this post in the future, so the blog remains coherently connected.

Spoiler

This post will meander through many ideas, so I'll state the conclusion up front. These are the two most important posts to write:

1. Five Years of Oil Milestones (And Some Wrong Turns). It's past time to take stock of the project! How did we get here, and what did we learn?
2. The Perlis-Thompson Principle. A #software-architecture principle that I've been hinting at in #comments. I've come up with a name, but not yet explained it. It has practical consequences: not just for Oil's design, but also for the design of languages, OSes, and distributed systems.
   • I elaborate on this post below, as it relates to many future posts.

Now let's see what else belongs on the Oil blog.

Drafts Written

These two posts are fairly close to done:

1. Oil Starts Fewer Processes than Other Shells. A worthwhile demonstration of how the #shell-runtime does basic optimizations. Surprisingly, all Bourne shells I tested do them! This draft is over a year old now, and I've already mentioned it in #blog-topics :-(
2. Shell History and Trivia. This post links to #comments, including ones on Ken Thompson's original paper on Unix shell. The paper popped up in a few places recently, which is great, because it will form part of the explanation for the Perlis-Thompson Principle.

Wiki: How to Understand Shell

Readers have praised this blog over the years, but there's still a big explanatory gap regarding Unix shell. I say this because I see repetitive arguments for and against shell erupt on many discussion threads, like announcements for Oil and other alternative shells.

As another anecdote, this post from last month links to The Bourne Shell Is Not a Programming Language. It sounds like it could have been written today, but it's from 2008.

So I've been thinking of ways to advocate shell. I want users to enjoy it. I've tried to do that with #shell-the-good-parts posts, but I haven't published enough of them.

Maybe I should start a Twitter account for pithy explanations of shell. This wiki page has ideas:

The following sections sketch selected ideas I like.

Slogans

Shell Is Now a Good Language. There are a number of users who avoid Oil specifically because it's a shell! This ironic because Oil is supposed to fix the things about shell that make you avoid it. :-)

Shell Is the Language of Heterogeneity and Diversity. Data analysis, building Linux distros, and distributed systems are inherently heterogeneous problems. This trend will only increase in the future. We'll never be able to express these problems under a single type system. (Feel free to argue with me.)

Oil Is the Shitty Language Rehabilitation Project. Related: this comment on the YAML Problem.

• Old sludge: Make embeds shell on every line, Awk embedded in shell, M4 generating shell (autotools), etc.
• New sludge. There are actually two distinct problems with YAML!
  1. Shell-in-{YAML,Docker,systemd,Ruby,Python}
  2. Shell-like programming languages embedded in YAML, like Github Actions.

Unix Programming Is Like Woodworking. Not everyone will get this lobste.rs comment, but I think it's worthwhile. Unix programmers and woodworkers make their own tools. Tools are made of wood; tools are made of text. You can make bespoke tools in IDEs, but in practice individual programmers don't do it.

Fallacies

Sometimes an inverted explanation of a concept is more memorable. That is, fallacies are a form of "negative knowledge".

Fallacy: Python Is A Good Shell Replacement. It's not either-or. I sketched a new answer in a post last month.

Fallacy: Unix Processes are Slow. They're often the only way to make use of your whole machine! Process-based concurrency is underrated.

Fallacy: Programming With Strings Is Slow. It's often much faster. The kernel natively deals with strings (byte streams), and the kernel is fast.

• Pointers are surprisingly big: I experienced this first hand with structured representations of textual source code like the lossless syntax tree.
• Garbage collection is expensive. Implementing a garbage collector improved my intuition for this.
• Textual/serialized representations don't have pointers and don't require garbage collection. You can perform a surprising number of operations directly on them. Or you can deserialize lazily.

Fallacy: You Can Extend Your Type System Across the Network. Large distributed systems can't be atomically upgraded. (Example: the entire Internet, although it's also true at much smaller scales.) This conflicts with type safety as a global property of a program.

• Tomorrow's post will mention this related comment on REPLs as Debuggers for Distributed Systems. Distributed systems are inherently dynamic. To see this, think of administrators / SREs as people who fix the bugs that escaped the type system. Their job revolves around a dynamic view of software.

Concepts

The Perlis-Thompson Principle. I figured out a good way to explain this crucial idea in software architecture: by coining a name that refers to Alan Perlis and Ken Thompson! (Incidentally, they're both early Turing Award winners.)

• Alan Perlis: It is better to have 100 functions operate on one data structure than 10 functions on 10 data structures.
• Ken Thompson: A program is generally exponentially complicated by the number of notions that it invents for itself ... [It] is my belief that you should base systems on a single notion.
• I claim that these two statements are reaching for the same fundamental truth. Let me know if you know of an existing name for this principle.
• It's also related to the idea of narrow waist. Here's a comment about text as a narrow waist (mentioned at the end of the HotOS Notes). The single notion/data structure often is the narrow waist, although there's subtlety I hope to tease out.
• Unix, Plan 9, and the World Wide Web all benefit from their use of the Perlis-Thompson Principle. Here's a thread about Go / Plan 9 and a comment on how fewer types lead to more composition.
• I'll go as far as to say that the degree to which systems adhere to the Perlis-Thompson Principle biggest force in practical software architecture. It determines how far you can scale, and the most scalable systems are the most important (Unix and the Web).
• It also has many practical implications for the Oil project:
  • With respect to to processes, coprocesses, procs, and functions
  • With respect to byte streams and the design for structured data.
• This post will take awhile to write because I want to give a solid definition, and answer expected objections. It's a design principle, not a hard theory or categorization.

Concept: String Hygiene (aka String Safety).

• You should see Shell injection, SQL injection, and HTML injection (XSS) as instances of the same pattern.
• The infamous ShellShock bash security hole of 2014 also relates to string hygiene in the bash interpreter itself.

Concept: Process-Based Concurrency (e.g. with fork(), exec(), wait()). As mentioned above, it's underrated. It's also the oldest form of concurrency, and we can contrast it with threads and async.

• Related: this discussion on straight-line synchronous code (threads and processes) vs. the asynchronous style

A thread is a process that allows you to write race conditionshttps://t.co/0QvA07mjS4

— Oil Shell Blog (@oilshellblog) March 24, 2021

Concepts: Policy vs. Mechanism; Control Plane vs. Data Plane. Last year, I mentioned that factoring into processes is an underrated design skill. These related ideas may help you determine where draw the line between processes.

• Guideline: Python and C programs shouldn't mention any concrete file system paths or port numbers. Such names are policy, not mechanism. Shell scripts should contain all of that knowledge, which you can also think of as wiring.

Wiki: How to Use Shell

Not only do I want to help users understand shell, I also want to show practical ways to use it. I started this wiki page:

As with the fallacies, the negative knowledge of anti-patterns is also useful.

Shell Programming Patterns

The $0 Dispatch Pattern. I sketched this in Four More Posts in "Shell: The Good Parts" (February 2020), but I haven't written a post that explains and demonstrates it. Nearly all my shell scripts are written like this. Example: the sample code in Shell Has a Forth-Like Quality.

• Brief explanation: Suppose all your shell code is in a function (rather than at the top level), and the script containing those functions ends with "$@". Then you can invoke functions with $0 myfunc. It's a recursive invocation of $0, dispatched on the function name.
• It's useful whenever you have an external command that invokes a command (somewhat related to #bernstein-chaining). Examples: sudo, xargs -P, find -exec.

The PPPT Pattern: Parameterized Parallel Process Table. This pattern helps you invoke parallel processes and collect their results.

• Another mnemonic for it is rows are processes; results are columns. Each row in a table contains the parameters for a process. Running the processes gives you new columns with results.
• It's frequently combined with the $0 Dispatch Pattern, as it's more flexible when the processes are shell functions, not external binaries.
• In 2013, I used this pattern for dumb parallelism for a cluster management task. Recently, I used Ninja to specify dependencies for the mycpp benchmarks and test cases, which makes it even more powerful.
• More examples: the "wild tests" that parse one million lines of shell on every release, as well as several other tests and benchmarks of shell and Oil.
• Oil will embrace this pattern with QTT: Quoted, Typed Tables.
• Trivial detail: The table is technically an (unordered) relation, because the processes complete asynchronously.

Anti-Patterns

Generating Code with Raw String Concatenation: Again, the Shell-in-{YAML,Docker,systemd,Chef,Vagrant,Python} Anti-Pattern is a primary example of this.

The Whitespace Joining Anti-Pattern. This came up recently in SSH quoting. Oil solves the analogous issue with echo and eval with shopt --set simple_echo simple_eval_builtin, which are on in option group oil:all.

Continued Tomorrow

This post got too big! There are still more #blog-topics to enumerate:

• More observations on distributed systems and the Perlis-Thompson Principle. I have problems with the state of the art in "the cloud". (Unfortunately, testing these theories with Oil is far in the future).
• Posts about the design of the Oil language. I've been saving comments related to Oil's design on the Zulip stream #blog-ideas.
• Odds and Ends.

In the meantime, let me know if there are particular posts that caught your interest. Or let me know if none of this made sense!

The wiki pages Slogans, Fallacies, and Concepts and Patterns and Anti-Patterns are rough, but I expect that they'll grow into a central part of the project in the future. It's hard to explain why and how to use shell!