Why Sponsor Oils? | blog | oilshell.org

March Recap / Link Dump

This post has a little of everything. It's tagged:

• #oil-release because I just made the 0.8.pre2 release. I also credit contributors to last month's 0.8.pre1 release.
• #zulip-links because it mostly summarizes Zulip threads, which are related to:
  • #oil-dev: I want to help contributors to Oil. These threads address questions like: How is the Oil codebase structured? How do I run tests? How can I help?
  • #oil-language: I'm still polishing the Oil language. Remember, I'm reducing the scope of the project in hopes of getting it "done" in 2020. But you can influence the design of the Oil language, and many people already have.
  • #mycpp: I haven't written much about the hacky but effective mycpp translator, but there's been interest in it. I kept a log of its development on Zulip.

The threads are grouped by topic, and roughly in reverse chronological order.

Feel free to ask questions on Zulip or in the comments!

Messages To Take Away

• As mentioned last month, I'm working on translation to C++ as the highest priority. Whatever else fits in 2020 is what will get done!
  • I still want to write Features Cut From Oil to make this clearer.
• But I'm responding to feedback, so nothing is set in stone. If you're interested in a feature, ask about it.
  • I take suggestions on the language design (preferably based on real code or #real-problems). For example, discussions with David A. Wheeler and several others shaped dashglob and Simple Word Evaluation.
  • I can use help with the dev tools, as well as feedback on them. For example, Travis A. Everett just added a shell.nix which I think will prove useful. I want Oil to be a self-sustaining project, even if I stop working on it, so the tools need to be in good shape.
  • I can use help implementing features, fixing bugs, and testing. There were four contributors to the 0.8.pre1 release.

Popular Posts and Discussions

There's been more interest in Oil lately, so I updated Popular Posts With Many Comments with recent threads on Hacker News, lobste.rs, and reddit.

I expect questions about Python and mycpp to be common, so I posted Implementation Language FAQ on Zulip. Here's the underlying question on lobste.rs. I'm drafting docs on the #oil-documentation channel, and feel free to chime in.

Releases

Oil 0.8.pre2

I released Oil 0.8.pre2 (changelog) a few days ago because bin/osh_eval.py now translates to C++, compiles, and runs. This is more than 16K lines of code, compared with less than 10K in the first oil-native demo.

Zulip thread: osh_eval compiles and links. And it also runs! I'm excited about this.

Oil 0.8.pre1

Last month's recap mentioned the 0.8.pre1 release (changelog), but I didn't credit the four contributors:

1. Aaron Sokoloski added type annotations to osh/word_eval.py (and osh/cmd_exec.py in an earlier release). These two files are the core of the shell evaluator, and the translation work I did builds directly on top of it. Oil's code must type check before it can be translated to C++.
2. timetoplatypus
   • Implemented the times builtin. This made two more Smoosh tests pass!
   • Added a failing spec test to expose a bug in popd. Note: I'll merge failing spec tests to show where Oil behaves differently than other shells. Even if we don't fix the bug immediately, it's useful. In this case I also fixed the popd bug.
3. Travis A. Everett started adding Nix support for Oil developers. Oil has slowly grown more dev dependencies (e.g. CommonMark), so I expect this to be useful to new developers.
4. lbonanomi implemented history -c and -d.

Check out the #help wanted label if you're looking for something to contribute!

Zulip: 0.8.pre1 Release

Discussions

Shell Features and Milestones

• Oil Runs a bash implementation of the chacha20 hash function.
  • This required implementing the "nameref" / declare -n feature, which was also used in one of Travis Everett's shell programs.
  • lobste.rs thread
• Here's a thread on implementing nameref, which was accompanied by shopt -s strict_nameref. When I implement shell features, I try to come up with cleaner but compatible semantics. I often hide surprising, ill-defined behavior behind a strict option.
  • namerefs overlap with the ${!ref} feature in confusing ways.
  • Circular namerefs were surprising! bash and [mksh]$(xref) both do this check, but they do it at different times.

Language Design

I'm removing more global options from Oil, which is consistent with what I wrote on the new wiki page Language Design Principles.

• Removed the do keyword and shopt -s parse_do. Instead of do f(x), you will be able to write f(x). This was a nice side effect of reducing the scope of the language: I no longer care about obj.method().
• setvar brainstorming. Added setlocal, which is a synonym for set that doesn't require shopt -s parse_set.
• The write builtin is a better echo. Removed shopt -s simple_echo.

Other:

• The CSTR serialization format is a tiny serialization format for strings. Mental shortcut: it specifies how ls --escaped and stat display filenames with special characters. Those tools do it informally and inconsistently.
• Brainstorming expression syntax for various features, which relates to the language design principles above. See issue 616. We need names for:
  • All the obscure string operators in ${}.
  • A command sub which doesn't drop trailing newlines.

mycpp: The Good, the Bad, and the Ugly

I keep track of the translation process in several threads.

• Ongoing mycpp Rewrites. I have to make small changes to the Python code so that mycpp can translate it to C++. For example, this thread documents ASDL usage idioms (which involve downcasting). Rewriting Python reflection with textual code generation is another big theme.
• Unfortunate mycpp Rewrites lists places where we mangled the code to make it translate. For example, Python tuples can be unpacked but lists can't, so we have to write it "longhand". There haven't been too many of these.
• Good mycpp Rewrites. Sometimes the translation process improves the code.
• mycpp Language Subset roughly defines the subset of statically-typed Python that the translator understands.
• mycpp Transformations is similar; it lists some non-obvious transformations performed. In most cases, mycpp is more like an "AST printer" than a compiler, but there are a few execptions. One of the more notable transformations is compiling % format strings to C++ functions (rather than printf, which is different).
• Translation Bugs That Survived the Compiler. Generating strongly typed C++ has proven to be pretty appealing, but it's not perfect.
• Back to Work on Translation. One of the longest threads, with messages going back to November 2019.

Tips On Oil's Source Code

• Oil is made of ideas; it's not a pile of code in a particular language. This is why it's feasible to translate: not just to C++, but other languages as well.
• Oil continues to use dependency inversion in a pretty rigid way.
  • This style unexpectedly made the code easier to translate. I can write a "slice" of the interpreter like bin/osh_eval.py and translate it on its own.
  • The new core/vm.py file expresses circular dependencies, which is unusual but works well.
• Naming convention for metaprogramming. Oil's source code is converging on a pattern. The command ls */*_def.py */*_gen.py shows build-time definitions and code generators. We compile high level descriptions to low-level C++ code.

From Contributors

• About command history in Oil.
• About interactive shell features. I want Oil to have good interactive features, but I can't handle it myself. Oil could have a plugin API, but that's also a lot of work.
• Using Nix for several purposes:
  • macOS + Nix + Oil
  • nix ci

Help Wanted

• What's Left to Type Annotate. Run types/run.sh need-typechecking to see what's left. Just ~7K lines of Python.
• Meaty task: pgen2 code generator for C. This is similar to what CPython already has.
  • It's a completely independent task that would save a lot of dev time and enable more Oil features. It already has good tests.
  • It requires some knowledge of parsing. Ask me about it if you're interested!

What's Next?

• Spec Test issues, like #167. Travis Everett brought these up with his work on Nix.
• Writing docs. Thanks to GammaFn for language feedback that reminded me what's next:
  • Command vs. Expression Mode. Clarify a fundamental concept that was mentioned in October.
  • Assignment Statements in Oil. I continue to tweak the semantics of assignment with respect to scope and declaration-before-use. There are many keywords, but I believe they're all justified, and you can use a simple subset of them for shell-like semantics.
  • Oil Language FAQ. For example, what's the difference between $(basename $x) and Oil's $f(x)? Where are sigils needed?
• More translation: osh/cmd_exec.py and core/process.py are the "meatiest bits". Getting those to work will be a big milestone toward a working shell.

If anything doesn't make sense, ask us about it on Zulip! Oil has many small subprojects that I hope you'll find interesting.