Why Sponsor Oils? | blog | oilshell.org
I released Oil 0.11.0 last month, with the biggest collection of features in years. The top feature was Hay, an extension to Oil that lets you use Ruby-like blocks to create internal DSLs. This is what I've wanted Oil to be all along — a language to build a simpler cloud!
It got good feedback on lobste.rs. So I wanted to circulate it more widely, and help people try it. But I immediately switched gears to translating Oil to garbage collected C++, working closely with our new "hired contributor" Jesse.
That was another solid month of work, contained in the subsequent Oil 0.12.0 release. So we have two big releases to talk about. This is an announcement for the first, but the second is also worth writing about later.
Oil version 0.12.0 - Source tarballs and documentation.
All this work has meant that blogging has fallen by the wayside. So here's a quick recap of where we are.
First, the project's scope has increased and outgrown me. I wrote about this in March: Oil Is Being Implemented "Middle Out" .
As of April, we have an NLnet grant to pay for specialized help, particularly in Europe.
And we now have one paid contributor (Jesse), but he is not European! And I've talked with several others who are interested, skilled, and available.
As a result, the C++ code is in better shape, and easier to work with. I greatly improved the build, translation, test, and CI infrastructure.
I've also reached out to authors of other alternative shells to "join forces".
But we need your help. NLnet prefers to pay engineers in Europe, which is understandable, because it's ultimately funded by European taxpayers! But we have interested programmers from the United States, Canada, Australia, Russia, and Brazil.
This means that we don't have money to pay them, and even without the geographical restriction, the grant wouldn't be enough.
The other source of funding we have is Github sponsors. So please sponsor Oil if you're excited about the project.
If you can't for some reason, consider telling your friends about Oil!
The progress I described means that this money will be converted into working, open-source code. That wasn't obvious three months ago, but it's clearly true now.
We have 5 sponsors at the moment, who I really appreciate, but it's not enough!
As of this week, the Oil project is $5000 in debt to me personally. Although it's not the top funding priority, I hope to recoup this somehow — Oil is supposed to be an open source project, not a charity project funded by my personal savings :-)
(Note that because it's open source, most contributors aren't paid. The funding is for specialized skills, i.e. Compiler Engineer Job — analogous to how both Python and Rust were funded for long periods. I've learned the hard way that designing and implementing a language requires sustained effort among many people.)
Now on to the features in this release.
This feature is novel and deserves good documentation. Here is a start:
To repeat the slogans:
Again, this gets at the core of what I want the Oil language to be! I also wrote this wiki page to help users understand the space of possible solutions:
The lobste.rs thread I mentioned also has substantive comments and viewpoints.
Please play with it and send feedback on Zulip or Github.
But that's not all!
(1) Oil now has a for loop variant for typed data.
enumerate(), keys(), values(), and
items().
(2) Dictionaries now retain the insertion order of pairs, as recent versions of Python do. This makes JSON serialization more predictable.
(3) For readability, the new fopen builtin lets you attach redirects to the front of a block:
fopen >out.txt {
echo one
echo two
}
I added a section in Oil vs. Shell Idioms about this.
(4) The shopt builtin now respect groups like oil:all.
shopt --set strict:all {
my-func # run with more errors
}
other-func # fewer errors
shopt --set oil:upgradeOil's slogan is that "it's our upgrade path from bash". Prior to this
release, the way to
upgrade
was with the oil:basic option group. It lets you use new features, but
doesn't require you to fix strict errors and remove discouraged idioms. (If
you want that use, strict:all or run bin/oil.)
Thanks to feedback from ca2013, I renamed oil:basic to oil:upgrade.
So now shopt --set oil:upgrade will be one of our primary memes, which
makes a lot more sense.
I'm surprised that this "naming mistake" persisted for so long! Again, if you're interested in a new shell language, please try Oil and send feedback. I know the documentation needs improvement, and we're making some strides there. (Again, sponsorship helps.)
I also wrote this doc with the help of ca2013:
Surprisingly little breaks! I feel good about this. Not only is OSH the most bash-compatible shell by a mile — it won't be that hard to upgrade to Oil either.
You'll get better error handling; you won't have to quote everything; you'll get Python-like expressions on typed data, as well as Ruby-like blocks. But everything else will mostly work.
(Related comment: Evolution or Clean Slate? Choose both.)
More changes:
(5) I changed the recommended syntax for the case statement.
My thought was that when using the Oil language, you should balance parentheses and quote constant strings:
case $name {
# left paren is optional in shell, but required in Oil
(*.py)
echo 'Python'
;;
('README') # quotes required to avoid confusion with variable
echo 'README'
;;
}
The reasoning is that parens introduce expressions in Oil, like
if (x > 0). But case statements already use parens to means something
different, and I don't want to confuse (x) and ('x'), regardless of
context.
On second thought, I think we can do even better, removing the two meanings of
(), with something like this:
case $name {
| *.py >
echo Python
| README >
echo README
}
This should make it easier to implement syntax highlighting for Oil, and I think it's nice to get rid of shell's unusual ;; terminator.
Thanks to Kazuma Arino for starting a VSCode Extension for Oil with syntax highlighting!
I'm reluctant to change syntax this late in the game, but shell's case syntax
will still work with bin/osh, and I don't want to live with an inconsistent
case statement in bin/oil. So a future release should implement this new
idea. (As always, feedback is welcome.)
(6) I worked out many tricky design issues regarding =. This had been in
flux for awhile.
name=val is disallowed in Oil.
PYTHONPATH=. foo.py are now allowed. That is, you don't
need an env prefix.
ca2013 for feedback on this. I added a new Language Design
Principle:
Don't break the interactive shell, or short examples printed in books.Package grep {
version = '3.1' # no var or const
maintainers = ['alice', 'bob'] # typed data on RHS
}
(7) The proc keyword is no longer present in bin/osh. It must be enabled
with the parse_proc option, which is part of bin/oil.
shopt --set.proc, you know you're running Oil! For example, you're using
Oil error
handling,
not shell error handling. The body of a proc no longer has two different
interpretations.(8) parse_dparen is off in Oil and disallows the bash construct (( .
(9) Bug fix: data is allowed as a variable name. Thanks to Kelly Brazil for reporting this.
We're making it easier to contribute to Oil:
(1) The master branch is now always green (passes a large test suite).
How is this implemented?
soil-staging branch, not master.master if the tests pass.So this implements the NRSROSE rule (merge after testing, not before). We do it with a simple shell script on Github Actions, which I'm pretty happy with!
(2) Thanks for Geoffrey Huntley, we now have Gitpod support.
bin/oil immediately. Then
hack on some Python!(3) Reducing friction
Future blog posts should cover two parts of our infrastructure, which directly relate to the new Hay feature:
Related: Comments About Build Systems and CI Services (April 2021).
In addition to the Hay and upgrade breakage docs, there are new wiki pages to help contributors:
I'm happy with the new features in this release, and excited about having help. This has enabled rapid progress on translating Oil to C++. (Teaser: the C++ translation is passing ~1680 of 1789 spec tests, up from 1495 in May.)
But even though Hay is a long time coming, it feels like just the beginning! I think we need more help on the Oil language, not just with C++ translation.
As one example, I had a nice chat last month with Alex of Shell++, an impressive alternative shell (which was started in 2016, simultaneously with Oil!)
I wanted to make the case that Oil is a good "Shell++", and he agreed! Oil's strategy of a smooth upgrade path is the best way to "replace" bash.
I also learned several things from transcribing Shell++ example code to Oil. It revealed holes in the Oil language, and vice versa.
So I'd like to invite the authors of other alternative shells to collaborate on Oil. We had some conversations way back in 2017 and 2019, but we should renew them. (We can use Zulip or another tool.)
Oil is more mature now, and I believe it has a solid foundation for more growth.
To pick some concrete examples, I think that both the ABS language and the Koi language are extremely similar to Oil and Shell++, and separate efforts are something of a wasted opportunity!
Reminder: If you like the changes in this release, and the direction of the project, please sponsor Oil! We have real people to pay now.
As always, let me know what you think about any of this in the comments.
Again, thanks to ca2013, Kazuma Arino, and Kelly Brazil for feedback and help.
Thanks to Timothy Sample and Danilo Spinella for great discussions on Hay.
Thanks to Jesse Hughes and Chris Watkins for help on Oil 0.12.0, which I want to write about next.
Here is more detail on the work above.
| #1149 | Disable proc keyword when no options are set |
| #1140 | allow temp bindings even in bin/oil (PYTHONPATH=. foo.py) |
| #1139 | Better error message: typed args vs. parse_equals |
| #1137 | Rename oil:basic -> oil:upgrade |
| #1135 | for loop enhancements: over typed data, and with index / key / value |
| #1134 | shopt --unset parse_bare_word for case and for |
| #1132 | Disallow shell style assignment in Oil |
| #1131 | 'data' should be allowed in expressions (soft keyword or remove it altogether) |
| #1114 | Require left paren in case clauses - case $x in ( foo ) |
| #1104 | flaky test/stateful cases |
| #1099 | shopt block syntax should accept oil:all |
| #1021 | Dicts should retain insertion order (add spec tests) |
| #1016 | (( bash construct conflicts with Oil on occasion, may need to be ( ( |
| #863 | fopen builtin |
| #745 | continuous build: merge to master on green |
These are commits from other contributors. You can also view the full changelog.
| ec7ccb32 | Geoffrey Huntley | Enable software development from any device via https://gitpod.io (#1156) |
| c40c8950 | Justin Hong | [build] Fix macOS build from source failing at make step (#1162) |
These metrics help me keep track of the project. Let's compare this release with the last one announced, version 0.10.0.
The spec test suites for both OSH and Oil continue to expand and turn green.
The significant expansion in Oil is due to Hay!
Despite the new features, the code is still compact. Significant lines:
Physical lines:
The oil-native parser got a bit slower:
Hm, this is the first change in quite awhile. It may be due to the for loop and case statement changes.
This test of the slow Python build uses wall time and is thus very noisy:
configureconfigureAs mentioned, we've made good progress in the last month:
This nice reduction is due a code cleanup:
We now match constant strings like "parse_proc" with a plain C++ function, rather than using re2c.
Binary size:
This is also a nice reduction, due in part to the same change. We also changed compiler and linker flags to remove unused symbols. As mentioned, I want to write about the new Ninja build.