The Long Slog Through the Shell (abridged)

2017-07-28

A few weeks ago, I wanted to describe all the work I had done on the shell runtime since mid-May. In other words, what accounts for the 200+ spec test cases that now pass?

mid-May: 573 total tests, 375 passing under OSH, 136 failing
early July: 752 total tests, 586 passing under OSH, 113 failing

So I wrote drafts of three posts that describe the implementation of more than a dozen shell features. But I haven't published them. Instead, I published a post about fixing the hardest bug ever, which happened at the end of the sprint.

Having released OSH, I'm more excited to write about the future of the project. So this post skips the details and makes the most important points from those three drafts.

(If you want to hear more about shell implementation details, leave a comment.)

OSH Is Achievable (but a lot of work)

More than one person told me it would be "impossible" to copy bash. They thought it was too hairy and too much work.

I can now say with confidence that this is untrue — at least when you consider bash as a programming language.

I implemented a large part of bash in 13K lines of Python code, along with thousands of lines of tests. Because I was able to quickly fix bugs and implement features, I have no doubt that I can finish a bash-compatible language.

It's a predictable process: the number of spec test failures goes down zero, while the test coverage goes up. It might take 6 or 12 months to finish, but it can be done.

As for the rest of bash, there are reasons why OSH is a good foundation for an interactive shell, but I haven't fully "de-risked" this part.

Even though OSH is achievable, I think the Oil language deserves some attention first. I'll explain this in the near future.

The Oil Execution Model

In order to correctly and automatically translate OSH to Oil, the two languages must have the same execution model, or "runtime semantics". Any differences should be accounted for by runtime options, e.g. a flag to turn off automatic word splitting.

By faithfully implementing bash semantics in OSH, I specified and solidified this execution model. It may be useful to think of these abstractions as part of a hypothetical shell VM:

core/state.py: Interpreter state.
- The Mem class is an interface to shell variables, local, global, and system-defined.
- The ExecOpts class holds global execution options. _ErrExit cleanly implements the subtle semantics of set -o errexit.
core/process.py: A library for launching processes and manipulating file descriptors.
- I've redesigned and polished the following abstractions: FdState, ChildStateChange, Thunk, Process, Pipeline, Job, and Waiter. They implement concurrent/asynchronous processes, and will help me implement Oil cleanly.

The Waiter abstraction is the most interesting from a computer science perspective, and it could be the subject of a future blog post. I used it to correctly implement pipelines, background jobs, and the wait builtin:

sleep 1 & to launch an asynchronous process.
$! to get its PID.
Three forms of the wait builtin:
- wait $mypid to wait for a specific process.
- wait -n to wait for the next process to finish.
- wait with no arguments to wait for all background jobs.
${PIPESTATUS[@]} is an array that has the exit status of every process in a pipeline.

In summary, releasing OSH and running real programs with it gave me confidence that it can be finished.

It also provides a tested execution model for Oil.

I have one more backward-looking post: a review of Roadmap #4. Then I'll write about the future of the project.

Again, I'm happy to answer questions about the shell runtime. There's a lot to cover, but a focused question may help get me started.