Why Sponsor Oils? | blog | oilshell.org
This blog is again falling behind the code, so I'm writing "backlog" posts to catch up. For example, I wrote two popular posts in this style during the summer:
The purpose of these posts is to maintain continuity and skip the detail. They mainly have bullet points and links, including #zulip-links and #comments.
If you want to read more about a topic, let me know in the comments!
This first backlog post sketches recent progress and releases, and is more detailed. For context, the last release was Oil 0.9.4 - User Feedback, less than 2 weeks ago.
I made a release on Sunday that resolved a major design question in the syntax of the Oil language.
This change was motivated by a wart in A Tour of the Oil
Language, and by
the desire to filter tables with expressions like ls --qtt | where (size > 10).
For example:
$ const obj = {name: 'bob', age: 42}
# pass typed arg 'obj' to 'json' builtin
$ json write (obj)
{
"name": "bob",
"age": 42
}
Everything in parens is parsed in expression mode, so you can also write it inline:
$ json write ({name: 'bob', age: 42})
Perhaps surprisingly, this syntax is backward compatible with shell, so it's
available in both bin/osh and bin/oil.
It was tricky to parse this without conflicting with shell functions, but I found a clean way to do it: by adding another lexer mode for lookahead.
And I now consider Oil's syntax "complete"! It's tricky to design a clean language that's also an upgrade from shell, but our parsing model has made it possible. The subtitle for this release should be The Triumph of Lexer Modes!
(While Oil's syntax is complete, we still need to rewrite the evaluator to avoiding "metacircularity", so it can be translated to C++ without the Python interpreter.)
Recall that Oil also enhances commands to take blocks:
cd /tmp {
echo $PWD
}
Commands can now take both typed args and blocks:
when (size > 10) {
echo $name
} # these three lines are a single command!
That is, this hypothetical when command takes 2 typed arguments:
size > 10echo $nameBut the block syntax is now just syntactic sugar for a trailing typed argument. It could also be written (awkwardly) like this:
const myexpr = ^[size > 10] # unevaluated expression
const myblock = ^(echo $name) # unevaluated block
when (myexpr, myblock)
I like this rule because it simplifies the language semantics, while allowing a rich and familiar syntax.
There's still more work to do in this direction, like implement parameter binding for typed arguments, and documentation.
I expect this syntax to be a flexible but readable foundation for many kinds of
DSLs. The when example is inspired by Awk.
It also should enable dplyr-like functionality:
ls --qtt | where (size < 10)
The plan for a TSV upgrade called QTT (quoted, typed tables) was mentioned in the Summer Blog Backlog.
I also brainstormed more use cases for typed arguments on the issue tracker:
eval (myblock) # extension to eval
assert (status === 2) # test framework
--long.test/spec-cpp regression, so we now have 1114 tests passing
in oil-native. This is an all-time high and well over half the OSH
tests. (On the other hand, no Oil tests pass in C++ because the
pgen2 parser doesn't have a runtime.)Back in October, I implemented extended globs because Raphael Megzari noticed their use in Nix's shell scripts.
I drafted a release announcement, but it grew two more posts intended for the Nix audience. I didn't finish them, and instead skipped to the Oil 0.9.4 announcement. Here's a sketch of these posts.
I used the implementation of extended globs to illustrate three design principles of OSH, the compatible shell language. Each principle has a subtitle / slogan.
The canonical example was:
echo hi > @(*.cc|*.h)
Even though the word on the right looks like an extended glob, bash and mksh treat it as a string, writing a file with literally that name. OSH disallows this, which brings all 3 principles above into play.
OSH wants you to write
echo hi > '@(*.cc|*.h)'
if that's what you mean, and this obviously falls in the common subset of OSH and bash.
I finished this post and got feedback from #oil-dev on Zulip. But I
decided it wasn't a great post.
While those 3 principles are absolutely true and useful to explain, there are other ones that explain less obvious design decisions.
So I updated the Language Design Principles wiki page, adding these:
Together, these two principles explain why OSH tags values with types, not
locations like bash does. That is, declare -i and declare -A
don't work like they do in bash. (But as usual, there is a large common
subset.)
The bash model means that the assignment statement will occasionally
fail, printing errors to stderr, but letting your program continue! I
consider this unacceptable.
Implementing extended globs wasn't easy. I had put it off for years, and there was at least one failed attempt when I resumed the work in October.
But I came up with some creative and compatible solutions that illustrate how OSH is implemented, and how the implementation affects the design.
As the Zen of Python says:
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
So this post would describe the development process I use:
The second section describes the tools and techniques, which are largely metalanguages:
The point of this post is that this implementation strategy has been stable for five years, and we continue to feel the benefits!
For example, I doubt I could have implemented extended globs or typed arguments without these "guard rails". Those features were both tricky late additions, and the metalanguages helped me made sense of them.
These thoughts aren't as well-formed than the ones above.
There are always too many things to do. In this thread, I dumped all the tasks that came to mind and tried to prioritize them.
Summary:
I'm always thinking about ways to get the project done faster. The next backlog post will address this topic, but this link is a dump of my thoughts. Summary:
match statement
operating on a typed AST. I'm eagerly awaiting MyPy support
for the match statement!I like this idea in theory. But I also just enjoy working on Oil without "administrative duties". Usually after I write a few blog posts, I want to get back to coding.
Let me know if you know of any candidates for these jobs, at any price!