Why Sponsor Oils? | blog | oilshell.org

Success With the Interactive Shell

I'm now using OSH as my interactive shell! Three recent milestones enabled this:

1. Successfully running thousands of lines of bash completion scripts found in the wild. I wrote a blog post in October that explains why this task is both necessary and hard.
2. Running shell programs that modify the interactive prompt, like git-prompt.sh and Python's virtualenv.
3. Implementing basic history expansion. For example, !$ expands to the last word in the last command.

This was a long slog through many features and bugs. There's still more to do, but I'm confident that this "dogfooding" process will make the shell usable by others. It immediately revealed several issues, many of which I've already addressed.

This is the biggest milestone since last January, when I wrote that OSH could run large shell scripts that build Linux distros.

The Next Few Posts

This work will take more than one post to explain. Here's an outline:

(1) The Interactive Shell Needs a Principled Parser

Poking at bash's interactive features revealed several bugs that have the same underlying cause. In this post, I'll

• Demonstrate the bugs;
• Explain how OSH avoids them with its principled parser; and
• Show that OSH has 8 times less source code than bash. The reuse of the parser is a major reason this is possible (along with using Python, ASDL, and re2c). Remember that OSH can now run thousands of lines of diverse distro scripts and interactive programs, so the comparison is pretty fair.

(2) Interactive Shell Screencasts

I'll show OSH running the programs above, and a sketch for a better interactive UI.

(3) Release of OSH 0.6.pre14

As part of the dogfooding process, I used OSH to release itself. Recall that each /release/$VERSION/ page — with its many tests, metrics, and benchmarks — involves thousands of lines of shell. In particular, devtools/release.sh invokes many other shell scripts.

Version 0.6.pre13 is linked below, but the dogfooding process has already produced more changes, which will go in 0.6.pre14.

Caveat: the biggest open problem is that OSH is still too slow . Although this doesn't matter for many batch programs, it does matter for interactive scripts.

The code in bash-completion consists largely of messy and expensive string processing, and it's already too slow under bash. But it can be more than an order of magnitude slower under OSH — i.e. hitting TAB may take hundreds of milliseconds rather than tens.

I have some ideas to fix this, but they aren't guaranteed to succeed. OSH is still experimental in that respect.

I'll also discuss what I want to happen for the 0.6.0 release, which has been brewing for a long time.

More

I want to get back to coding after those three posts, but I also want to write:

(4) Project Roadmap. This big milestone, as well as feedback from the BayLISA talk, clarified what I should work on next. I'm really motivated to work on the interpreter performance problem.

(5) Blog Roadmap. There are even more things to publish, like:

• The next post in the Dev Log series.
• In January, I re-discovered a pervasive and old shell security issue, which is yet another reason that shell needs a principled parser.

OSH 0.6.pre13

In the meantime, please try OSH 0.6.pre13:

• https://www.oilshell.org/download/oil-0.6.pre13.tar.xz (gzip version)

I'm looking for experienced shell users to try OSH on real shell scripts — both interactive and batch — and report bugs!

This is important because changes to the OSH language will be more painful later. OSH can now run many real programs, so its architecture is becoming stable.

To build and run it, follow the instructions in INSTALL.txt.

Feel free to ask me questions in the comments. Or join oilshell.zulipchat.com and ask questions!

Appendix: What Took So Long?

Some readers may wonder why it took nearly 3 years to start using OSH myself every day.

This lobste.rs comment gives some color on that. Key points:

• The project was motivated by the use of Linux distros in distributed systems. I didn't care as much about the interactive shell. Of course, a bash replacement needs to do both jobs.
• Completion scripts are the hardest shell scripts to run, since they use every nook and cranny of bash.
• Yes, a significant motivation for Oil is other programmers "doing it wrong"!

This Hacker News comment explains why I took the long route to creating a new shell, with an analogy to C and C++. Replacing bash is a different task than creating a new shell.