Why Sponsor Oils? | blog | oilshell.org

Dev Log #8: Shell Protocol Designs

I didn't write much during the fall, so this is the second of four posts to bring readers up to date on the project's status:

1. Dev Log #7: Hollowing Out the Python Interpreter
2. Dev Log #8: This post is about ideas I've had to improve the shell ecosystem.
3. Dev Log #9: Oil Subprojects (OPy, OVM, etc.)
4. Dev Log #10: Coding Experiments and Research.

I'm writing this for present and future Oil developers. If you have questions, please leave a comment!

Shell-Agnostic Autocompletion

After trying to run bash completion scripts with OSH, I realized that most alternative shells have the same problem: creating a new corpus of completion scripts.

It would be better if we could share this task rather than doing it N times — once for each new shell.

I started a discussion with several shell developers, including Qi Xiao of Elvish and zsh developer Oliver Kiddle. I was impressed that Qi started porting git-completion.bash to Elvish, which will teach us a lot.

This is work-in-progress, but here are some wiki pages that came out of it:

• Shell Autocompletion. What problem are we solving?
  • I summarize what I learned about existing completion scripts, e.g. some differences between bash and zsh.
• Shellac Protocol Proposal. A sketch of a protocol for shell completion, in the spirit of the Language Server Protocol. It still needs to be prototyped.
• Parsing Models Cheatsheet. I would rather use principled parsing techniques for autocompletion rather than heuristics, although this is something of an open problem because all existing shells use heuristics.
  • Parsing Shell Quotes Requires Parsing the Whole Language

Please leave a comment if you've written a shell completion script in any language!

Running Bash Completion Scripts

The other approach I'm taking is to run existing completion scripts with OSH. Here's a wiki page for people who want to help:

• Debugging Completion Scripts

Issues:

• Issue #208: Run the bash_completion project
• Issue #175: Run the git-completion.bash

Coprocess Protocol

In the Shellac protocol proposal, I wrote that sending a request to a coprocess would result in lower latency compared to invoking a binary for every TAB completion.

In this usage, a coprocess as a single-threaded server that runs main() in a while (1) loop. (bash has had support for coprocesses since version 4, but they're limited and rarely used.)

A few weeks later, I had a productive conversation with Avery Pennarun, author of redo, about the problem of process startup time in builds.

This turned into a prototype of a coprocess protocol:

• oilshell/shell-protocols/coprocess

The idea is that is should be easy to turn existing command-line tools written in any language into coprocesses.

Please try the demo described in the README, and let me know if it makes sense. Can you think of a tool you use that starts slowly, and can you imagine retrofitting it with this protocol?

The Problem of VMs that Start Slowly

I'd like to write a post about small programs being burdened by large runtimes. If you write a 100-line Python or Java program, the runtime does a lot of computation and I/O before you even get to main(). This inhibits Unix-style reuse.

• Here's my analysis of Python's startup time, to which mhneu responded, You've characterized the problems well.
• Here's a post on Microsoft's CLR, which powers C#: The 68 things the CLR does before executing a single line of your code

This problem seems to be getting worse, not better.

Here are some more rough documents on the potential solution:

• Coprocess Protocol Proposal
• Coprocess Protocol V2

There's still more to do, including:

• Handling concurrent coprocesses.
• Improving efficiency in some cases with descriptor passing over Unix sockets.
• Proving that the protocol is viable by porting more command line tools to it. The tools will operate in coprocess mode only when an environment variable is set, so it will be a backward-compatible change.
• Eventually build it into OSH and Oil.

Summary

I'm excited by the coprocess protocol! It's easy to implement and solves a concrete problem that's only getting worse.

I'm also excited by the possibility of a completion protocol, but I recognize that it's a long-term goal with many dependencies. On the bright side, every author of an alternative shell will hit this problem, and our interests are aligned in solving it.

These two protocols aim to improve the wider shell ecosystem. The next post will discuss projects that are part of Oil proper: OPy, OVM, and more.