Why Sponsor Oils? | blog | oilshell.org

Roadmap #5: Why Oil, and Why OSH?

I reviewed roadmap #4 over two months ago. I noted that the project completely changed directions: instead of rewriting OSH in C++ with custom code generators, I shipped the Python prototype of OSH!

Since then, I've made a second release, but I haven't written a new roadmap yet. One reason for this is that I've been changing my mind about what to work on.

Oil or OSH?

After the first release, I was burnt out by the long slog through a shell, so I decided to focus on Oil instead. I also promised that I would "re-sell" Oil, and focus on the message, which is:

Oil is your upgrade path from bash. It's the only language that shell / bash can be automatically converted to.

But after writing a few blog posts and drafting docs about the Oil language, I was ready to work on OSH again!

The meta-goal is to have some real users, whether they're for Oil or OSH. And right now OSH is closer to that goal, so I feel more motivated to work on it.

Goals

Running shell scripts found in the wild will focus me on the right goals. I expect to work on:

• Performance
• Adding and polishing error messages
• Unimplemented language features
• Unimplemented builtins (trap, etc.)

To be more specific:

1. Run Scripts Associated with Alpine Linux

(I explained this new direction on lobste.rs and /r/ProgrammingLanguages.)

I've been playing with Alpine Linux, and I like it. There's a 2500-line shell script that builds Alpine packages called abuild, and I believe it's a good motivating use case for OSH.

Possible progression:

1. Build packages with abuild running under OSH.

• Unusually, abuild is written in the busybox ash dialect, so this task is easier than running arbitrary bash.

1. Build packages with /bin/sh as OSH.
2. Build packages with /bin/bash as OSH (ambitious)

Then I could move onto runtime. e.g. booting an entire Alpine Linux system with /bin/sh as OSH.

Future blog topic: An Evaluation of Alpine Linux. Let me know if you don't understand why I chose it, or if you think another distro or shell project would be more appropriate for Oil.

2. Run Scripts That Use Different Shell Dialects

There is a similar progression with shell dialects:

1. I ran my own bash scripts with OSH, including most of the scripts in the Oil git repo.
2. I ran Python's configure script with OSH. Such autoconf-generated scripts use a subset of POSIX shell.
3. abuild is written in ash, which is a superset of POSIX shell.
4. After I get abuild running, I can run other people's bash scripts. debootstrap would be a big challenge.

3. The Oil Language, OPy, and OVM

The previous tasks are about grinding on OSH, but I'm also itching to work on the Oil language. It feels solid in my head, but unfortunately there's no implementation yet.

Because this work is further down the road, I'll point out some problems to solve here, without specifying solutions:

• The syntax of OSH is described in ~4200 lines of Python: a lexer, three recursive-descent parsers, and an expression parser. It would be nice to have a more compact description of the Oil language syntax.
• I need to unify the execution semantics of Oil and OSH. I don't want to have two entirely separate execution engines.
  • I also need to accomodate the parts of Oil that bash doesn't have, e.g. compound data structures with references, and garbage collection.
• OSH is an interpreter written in Python — that is, it's doubly-interpreted. I should probably fix that before writing Oil.
• OVM is still too big a subset of CPython.

I'm itching to work on the OPy compiler as well. I believe it will help with many of these tasks.

A Better Language for Linux Distributions

I want to work on Oil, but I think attracting OSH users is better for both my motivation and for the health of the project.

That means finding a "carrot". Right now I believe the carrot is:

OSH is a better shell dialect for writing Linux distributions. It has the bash features you already use, but it's stricter and easier to debug.

Linux distributions appear to be the projects that stretch the shell language the furthest. In other words, they need the most help. Recall that some of the first scripts I parsed were related to Linux distros:

• Aboriginal Linux
• debootstrap
• The features for the second release were largely motivated by a hairy bash script in Nix. See issue #26.

As mentioned, the carrot for Oil is that it's automatic conversion of bash to Oil, but unfortunately that goal is further off.

Also notice that I haven't mentioned anything about an interactive shell. Though I believe the OSH parser has a solid foundation for command completion, right now I'm picking my battles, and interactive features are also on the back burner.

Conclusion

This roadmap isn't as definitive as the previous ones, because at this point, what happens is determined by:

1. What issues Alpine Linux scripts uncover.
2. User feedback. Please try the latest OSH release and file bugs!

Let me know if these motivations make sense. And if you'd like to dip your toes in to the project without too much commitment, let me know as well.