Why Sponsor Oils? | blog | oilshell.org

Summer 2020 Blog Roadmap

I just released Oil 0.8.pre8, and it reached a significant milestone! The oil-native binary can now run Crestwave's brainfuck interpreter in bash, and it's even pretty fast.

So now is a good time to come up for air and write blog posts. This post is an outline of what I want to write. If the January Blog Roadmap is any indication, I'll publish a few posts, then go back to coding again.

Please ask questions in the comments section to help me prioritize the writing.

A Plan for Oil 0.8 and 0.9

The C++ translation process is creating a fast and correct shell, which is exciting. The next two releases will continue along the same path:

• Oil 0.8 will be a pure interpreter (without most I/O). In addition to brainfuck, it should run a Lisp interpreter and a JSON parser.
• Oil 0.9 will add shell I/O: processes, pipelines, and more. It should be able to run Python's configure script, big batch scripts from Linux distros, and Oil's own shell scripts.

Based on the last few months of progress, I believe this can be done by the end of the year. Recall that back in January, I said I wanted to get something concrete done in 2020. This is about what I expected.

Features Cut in 2020

I also want to reduce the scope of the project, and point out some problems.

• Oil 0.9 won't have an interactive shell. That requires a binding to GNU readline, which isn't hard, but it will take time.
• It won't have completion builtins, which require a nontrivial translation due to our use of Python's yield.

Making the Project Sustainable

I've been working on Oil for over 4 years! I considered it an experiment for most of that time:

• Can we write a compatible shell with a principled architecture? The Unix shell is never going away, but the bash codebase has reached a dead end.
• Can we use algorithms that are more clearly correct, rather than groveling through backslashes and braces one at a time in C?
• Can we turn this compatible shell into something more like Python or JavaScript? Let's make shell easier to learn and more powerful.

After the progress in 2019 and 2020, I'm confident that the answer is yes. There's no risk of failing to deliver 0.8 and 0.9 — it's just work.

And with enough people working on the project, I'd argue there's no risk to Oil 1.0 or Oil 2.0 either. However, even though 30+ people have contributed code to Oil, I'd estimate that 90% of the code has been touched only by me.

I want to change that. Here are some posts that address "project sustainability".

How Can Oil Be Useful Before It's Complete?

The lack of real Oil usage has been bugging me for a long time, and I've mentioned it when soliciting help. I don't have a solid answer to this, so I'm interested in your ideas.

Outline:

• How Can Oil Add Value Without an Interactive Shell?
  • Because the interactive shell will remain in Python beyond 2020, I want Oil to be useful in some other way in the short term. I think tracing and debugging of continuous builds is one possibility, and it's one that we can "dogfood".
• It's not surprising that the experience of using Oil is "raw". A gap needs to be bridged, and I can't do that alone. Users waiting around for a new shell will be disappointed.
• My fork of bash-completion has languished for 18 months. I still use it on every Oil release, but nobody else does!

How to Rewrite Oil in Nim, C++, D, or Rust (or C)

Last week, I got a mail from a reader Andrew who's rewriting Oil in the Nim language! This is great, as I believe in the power of parallel open source development.

I'd like to spread the knowledge of shell implicit in Oil, and independent implementations are a great way to do that.

Roughly speaking, there are 4 possible levels of reuse:

1. Reuse the list of syntactic ID elements. This is a hard-won list of 300 or so things a bash-compatible shell needs to handle!
2. Reuse the generated lexer, which is pure C and can be integrated into any language. This will save a huge amount of "groveling through backslashes and braces", i.e. what bash does with ~10,000 lines of hand-written C code.
3. Reuse the ASDL schemas, like frontend/syntax.asdl. In some sense, this is the OSH language definition.
4. Reuse the entire architecture of the Oil interpreter, which uses dependency inversion of state and I/O. (For example, it doesn't constantly save and restore global variables like bash does.)

Other hints:

• ls */*_def.py in the Oil repo shows Python code with "declarative" knowledge of shell
• ls */*_gen.py are the corresponding code generators. They produce Python and C++ code, but can be modified to produce Nim or Rust code, etc.
• The "Source Code" section at the bottom of /release/latest/ contains the links above

Addendum: Rob Landley and toybox should reuse some of this knowledge. He says he's been working on a bash-compatible shell on and off since 2006! And he has expressed doubt about finishing it, so sharing knowledge only makes sense. However I'm not sure if we can communicate on the same wavelength.

Accepting Donations for a Technical Writer

Last year, I was asked if I accept donations for the project. My answer was no, because Oil is still an experiment. (In other words, I wanted to be able to abandon the project at any moment without guilt.)

However, I now think the experiment is succeeding. It feels like the codebase is "just on the edge" of being a significant advance over bash, the most widely deployed shell in the world.

In contrast, the documentation is not close to being done. And I think attracting contributors to docs is harder than attracting contributors to the code.

So I'd like to solicit donations to hire a technical writer for Oil's documentation. I'm not sure when I will do this. I don't really want to be an administrator, but it seems necessary.

I'd appreciate advice from anyone who has raised enough money to pay someone. I imagine that this is around 6 months of full time work. (E-mail me or leave a comment.)

Some Kind of Teaching / Mentoring

I posted an offer to edit technical writing on lobste.rs. I can imagine something in this vein to on-board contributors to Oil, but I'm not sure what it would look like.

Though a major problem with this model is that it sets me up as some kind of authority, whereas I'd really like people to tell me what I'm doing wrong.

There have been multiple contributors, including Koichi Murase and Josh Nelson, who have fixed things that surprised me. And Crestwave has done a lot of difficult debugging on codebases I'm unfamiliar with.

In addition, our .travis.yml was contributed a few years ago, before I knew anything about Travis CI. It's gradually turned into a useful and functional continuous build, of course written in shell. (As of the last release, it runs the ble.sh test suite and the mycpp translation of oil-native).

Advocacy for Unix Shell and Oil

I did a good job of advocacy at the beginning of the project, but haven't had time for it lately. Here are some more ideas.

More Posts in #shell-the-good-parts

Headings for this post (yes, this is an outline for an outline!):

• Shell Is Executable Documentation.
• Shell Is a Language For Software Quality And Performance
  • Examples: Oil's own tests and benchmarks
• Shell Improves Your C Code
  • The C compiler is in the "standard library"
  • Experts at CppCon Know Shell
• Shell Lets You Use All of a Machine
  • xargs -P
• Shell Runs On Many Cloud Services And Lets you Abstract Them Away
  • Toil runs on Travis CI
  • It Uses Bernstein Chaining With SSH
• New Shell Patterns I've Used in Oil's Codebase
• Shell Lets You Reduce Code in Polyglot Systems
• Shell Is a Language for Describing Deployed Systems
• Appendix: Useful Shell Tools

The Unix Shell Should Evolve Like Perl 5

This is a post derived from these links:

• Announcing Perl 7 (lobste.rs)
• Getting Better, not Getting By by Sawyer X (YouTube video)

Summary and analogies:

• Python has from __future__ to introduce new features.
• Perl has use strict and more, but they don't change the defaults. A key point that Sawyer X makes is that when you're using Perl 5, by default you're using a language with 15+ year-old misfeatures.
• Ditto with shell. Bash has shopt, but by default you're using a language with 30-year-old misfeatures.

In contrast, Oil has shopt -s oil:basic and shopt -s oil:all.

Why Use Oil?

I need to explain Oil from scratch again. Recent progress on the implementation has made it more clear what Oil is and what it isn't. Draft on the wiki:

• Why Use Oil?

I want to condense those points and put them boldly and clearly on the Oil home page.

Oil Starts Fewer Processes Than Other Shells

The comics post back in April alluded to this result, but I never published/explained it. This is mainly because I wanted to gather data on real shell scripts in the wild. For now, you can see the benchmarks:

• syscall/by-code.txt
• syscall/by-input.txt

Summary: Across several dozen test cases, Oil starts fewer processes than every shell except the uncommon yash shell. I could have optimized it more, but I didn't want to game the benchmark.

For context, I want Oil to be faster than bash and other shells along three dimensions:

1. Parsing. This has been achieved, although changing the memory allocation strategy will affect the results.
2. Running programs that do computation without I/O, like autocompletion scripts.
3. Running programs that do a lot of I/O, like building Linux distros.

What's Next?

I outlined 8 or 9 posts here, but there are still two more with #blog-topics:

• Comments on Eggex and "regular languages", i.e. computer science theory.
• Comments on C++ and native code. And maybe on the connection between shared hosting and shell.

After I publish those, I'll write the plan for Oil 0.8 and 0.9.

Let me know what you're thinking in the comments!