Why Sponsor Oils? | blog | oilshell.org

Blog Roadmap for January 2021

A year ago, I wrote several posts to take stock of the project, starting with these two:

• Making Plans for Oil in 2020
• Blog Roadmap for January 2020

They still feel useful after year: I didn't write every post, but I ended up addressing almost every issue.

So it's time to do it again. What's happened recently in the Oil project, and what are our plans for 2021?

Summary for Casual Readers

I described the biggest accomplishment of the year in October, in Four Features That Justify a New Unix Shell. Summary: the OSH language is mature, with a compelling feature set, and the Oil language is taking a nice shape.

But the interpreter needs to be faster, and we're speeding it up by translating statically typed Python to C++. The translation relies on a runtime library that includes a garbage collector.

Unfortunately, you could say that writing and debugging the collector kicked my ass in December. I spent much of the month knee deep in segfaults, GDB plugins, and CLion.

I did manage to fix many bugs, and I addressed overlooked issues like vtable pointers in object headers. And I still believe the design is right. However, I grew tired of the work and took a break.

As a result, Oil didn't fully achieve its goals for 2020. But I'm still happy with the progress we made.

Flushing the Blog Queue

My "break" was writing some fun tools in PHP, Python, and JavaScript that will help with blogging. I'm going to take these tools for a spin before returning to work on the garbage collector.

Long-time readers know that I make many comments about the project on Hacker News, Reddit, and lobste.rs. Explaining the project ends up shaping it.

Some of these comments are naturally blog posts, so I keep track of them on the #blog-ideas stream on Zulip (and previously #oil-documentation).

But rather than write full blog posts, I'll link and summarize the comments. I feel like this style of dialogue could be more effective at communicating subtle ideas, and it takes less time. I divided them into these categories:

1. Comments About Parsing: Theory vs. Practice. I want to preserve a few popular comments under the #parsing tag.
2. Comments About Shell and Oil's Design. These comments provide color on the Oil project. There is also some shell history, philosophy, and trivia.
3. Comments About Languages, Distributed Systems, and Software Architecture. I hinted at the relation between shell in distributed systems at the end of this November post. Around that time, I also chatted with Michael Greenberg and the authors of PaSh about some related and ambitious topics.
   • Goal: Start a conversation, and encourage others to view the shell language as a basis for distributed systems. It's already crucial as a debugging tool for administrators, but it's naturally more than that. A better shell can statically and dynamically describe distributed processes, their configuration, and interconnection.
4. Comments About C++, mycpp, and Tea. Oil has an unusual implementation strategy, and I wrote several comments to justify it.

Writing these summaries should take a week or two. I hope that some of the "big ideas" will attract a new audience to the project.

Selected Recap of 2020

Let's review a few posts from 2020 before talking about 2021.

• Ambitions for a Unix Shell (January). I said that Oil would no longer reuse parts of the Python interpreter, and that we'll instead translate a smaller subset to C++. This scope reduction turned out to be a good idea!
• Oil 0.8.pre4 - The Biggest Shell Programs in the World (April). OSH is the most bash-compatible shell by a mile. I've taken pains to make adoption easy.
• Summer 2020 Blog Roadmap (July). I had some concerns about the sustainability of the project. Six months later, I feel that again! But I'm excited by the blog posts above, so let's take it one step at a time. Taking a break from coding usually gives me a new perspective.
• Changes to Shell Runtime Semantics (November). The 4th post in a series about OSH and Oil. This was a big achievement, and it closely matches my vision for the project from the beginning: Oil fixes bash's problems, and provides a smooth upgrade to a better language.
  • For better or worse, these milestones also encouraged me to increase the scope of the project again!

Good News

I can see that many people are downloading Oil, and the blog is more popular than ever. Each new post has many readers, and people continue to read what I wrote years ago.

But I can use more feedback! Let me know if OSH works on your shell programs, and let me know what you think of the Oil language. (Thanks to Danilo Spinella for recent comments.)

See Where To Send Feedback. There's also a new #oil-help stream on Zulip.

Plans for Oil in 2021

To be honest, I haven't figured out what 2021 will look like. After I flush the blog queue, I should write about project plans. Explaining the project to others often helps me figure out what to do.

Off the top of my head, here are the top priorities:

• Finish the garbage collector and C++ translation. Optimistically, I feel like this is 2 more months of work, but it's probably more like 6.
• Improve and polish the Oil language. The comments about distributed systems may end up driving this. We also have many opportunities dogfood Oil, since the project infrastructure consists of many shell scripts.
• Some efforts to fix the project "bottleneck" problem. Maybe I should release something minimal and solid, and switch to "cheerleading" mode. That is, I'll encourage others to build things on top of Oil rather than build them myself.

More Writing

• We need a 2021 Update to the 2018 FAQ: Why Create a New Shell?
• I haven't written The Ultimate Guide to errexit yet, but all the code has been there for a few months. This is more evidence of the project bottleneck.

Conclusion

This post outlined what I want to write about in January. I hope to get most of it done, but no promises. The first post in this series will be Comments About Parsing: Theory vs. Practice.

Appendix: Deferred Blog Topics

A few interesting side projects have made their appearance behind the scenes. I mention them here so you know they exist and can questions, but I don't have plans to write about them.

Unix-y Web Services I Wrote

Three tools that help with blogging and web publishing:

1. wwz. A FastCGI program to serve static files out of a .zip file. I wrote this in Python in 2017, and it's used extensively on every release page.
2. picdir. Context: I want to make the blog more visual, but that's difficult in Vim. I'm now using the StackEdit Markdown editor, which provides realtime preview.
   • So I just wrote picdir, a small PHP program that serves dynamically resized images. It will be useful for writing, and it also gave me experience with PHP.
3. hashdiv. JavaScript web scraping and an associated Flask server. It's immature, but will help with the next few blog posts.

Experiences With Various Languages

I tend to write code in useful languages, not simple or elegant ones. This is a rich source of ideas for language design, both positive and negative.

• C++. I've kept notes on Zulip about using C++ for Oil's implementation. For example, I've learned and used some "modern C++" like constexpr, but I've also used C-style macros to work around various limitations.
  • It makes more sense to write about this when Oil is actually running smoothly in C++, which still hasn't happened.
• R. I used R for web analytics and Oil benchmarks.
• PHP. I learned that PHP feels a lot like shell, and Oil should have some of these features! The Flask framework in Python appears to be somewhat influenced by PHP.
• JavaScript. Web scraping is naturally done in the browser with document.querySelector. It's a very Unix-y form of ad-hoc reuse.