Why Sponsor Oils? | blog | oilshell.org

Release of OSH 0.7.pre1

This is the latest version of OSH, a bash-compatible shell:

• https://www.oilshell.org/download/oil-0.7.pre1.tar.xz (gzip version)

Please try it on your shell scripts and report bugs! To build and run it, follow the instructions in INSTALL.txt. Please report bugs on Github.

If you're new to the project, see Why Create a New Shell? and the recent FAQ.

Changes

This section continues where I left off in What's Happened Since February?, published last month. That post summarized versions 0.6.pre16 through 0.6.pre22.

This section summarizes the last 3 releases, including 0.6.0.

I didn't write an announcement for the 0.6.0 release because the AMA on /r/linux last month generated a lot of feedback. Thanks for the support!

Contributors: let me know if I missed something notable.

0.6.pre23 on June 26th (changelog)

Highlights:

• Implemented basic job control. The typical flow of vim, Ctrl-Z, and fg now works.
• Marek Suppa implemented test -o to check if an option is set.
• Implemented the -b -c -p -S flags to test.
• Fixed bugs reported by Quentin L'Hours:
  • Word are now joined correctly when $IFS is empty and when it's unset.
  • Literal words like echo are no longer split.

I also triaged spec test failures. See the metrics at the end of this post.

0.6.0 on July 1st (changelog)

OSH now has a manual, and a list of known differences with other shells. From now on, it will be published with every release.

• OSH Manual (latest version)
• Known Differences (latest version)

Changes related to strict options:

• Moved them under shopt, rather than set -o
  • e.g. shopt -s strict-array turns an option on, and -u turns it off.
• Added shopt -s all:strict to turn on all strict options at once.
  • Goal: OSH helps you write better shell scripts, like a ShellCheck that also has knowledge of what happens at runtime.
• Turned on strict-arith by default.
• Documented strict options in the manual.

Other:

• printf: Implement hex and octal formatting.
• Fixed bugs reported by Quentin L'Hours:
  • &> and &>> no longer take a file descriptor on the left.
  • Boundary cases for ${var%*} and ${var#*}.

0.7.pre1 on July 17th (changelog)

Highlights:

• Overhauled and implemented the rest of array and associative array semantics, including literals like A=(['key']=value ['k2']=v2).
  • These semantics still need to be documented, but I'm happy with how it turned out. It relates to the issue I raised in Parsing Bash is Undecidable almost 3 years ago!
• Implemented the hash builtin, along with $PATH caching. (I didn't see any notable speedups as a result.)
• Implemented test -O and -G.
• Joshua Nelson
  • Made the prompt width calculation Unicode-aware.
  • Changed source to look up entries in $PATH.
  • Run files without a shebang with #!/bin/sh.
  • Added location information for ~, which improves error messages.
  • Started property-based testing using Hypothesis.
  • Tested OSH on FreeBSD. Fixed an incompatibility in the configure script, and in our libc binding.
  • Initial implementation of $PROMPT_COMMAND, which many bash users use to customize their interactive experience.
• Wilke Schwiedop committed an initial implementation of find and xargs. This isn't released yet, but I expect it to play an important role in the Oil language, which will subsume the functionality of these two tools. Why?
  • find is a recursive expression language, like many other shell sublanguages.
  • xargs starts processes and splits strings, just like a shell.
• Fixed bugs reported by Quentin L'Hours:
  • exit behavior across sourced modules.
  • The default status for exit and return.
  • Waiting for singleton pipelines like ! true.
• Matt Singletary and Stephen Gregoratto revised the man page. (Still needs to be released and installed.)
• Some work on the Oil language. I decided to start with the expression language. Details forthcoming.

What's Next?

There's still plenty more to do. Here are some highlights of major project areas:

(1) OSH language. After recent changes and enhancements to arrays, printf, and brace expansion, it feels like it's "converging". I expect the upcoming overhaul of assignment builtins to be one of the last major changes.

(2) Oil language. I have a concrete design for the language, but I only recently started implementing it. I also started a design-docs directory. It will be a lot of work, but we're finally getting to the main point of the project!

(3) Performance. Still needs a lot of work. Python lets us work quickly and safely, but the code needs to be much faster. I have an idea for a blog post: How to Rewrite Oil in C++, Rust, or D. If you're interested in that, leave a comment.

(4) Documentation. We now have a short manual, but the quick reference and help builtin need a lot of work.

(5) The Interactive Shell. I want to make the shell more immediately appealing. People use shell interactively more than they write programs with it.

• Display completion descriptions from various sources, as zsh and fish do.
• Color in error messages (optional).

How to Help

It's good that Oil is getting more contributions, because I can't do this all by myself. I can use more help!

• I suggested some ways to help in the recent FAQ: How to Help
• I continue to keep issue labels up to date:
  • #good-first-issue and #help-wanted
  • #open-problem: Apply your unique expertise here!
  • #should-run-this: Shell programs we should run. Try running your own programs too.
• I updated the README.md, which gives an overview of the repo. It looks big, but the parts are modular and well tested.
• Oil is a plain Python program, with plain C extensions. That means it's quick to modify and test. The main thing to get used to is the use of multiple code generators, which makes the build system complicated. This is documented on the Contributing wiki page.

Please ask questions on oilshell.zulipchat.com.

I'm making more fine-grained announcements there as well. For example, I just wrote a wiki page documenting internal data structures and algorithms:

• OSH Word Evaluation Algorithm

Conclusion

The metrics below show that everything is pointing in the right direction.

Oil is making solid progress, but there's still a lot more to do!

Appendix: Metrics for the 0.6.0 Release

The last time I reviewed metrics was in the February release of OSH 0.6.pre15. Let's compare it with version 0.6.0 rather than 0.7.pre1, since I like round numbers.

New features and bug fixes made more than 114 new spec test cases pass. I also marked failures as known differences, which led to the failure count dropping:

• spec tests for 0.6.pre15: 1260 tests, 1116 passing, 104 failing
• spec tests for 0.6.0: 1393 tests, 1230 passing, 78 failing

We have ~1400 new significant lines of code:

• cloc for 0.6.pre15: 11,041 lines of Python and C, ~200 lines of ASDL (excluding testdata).
• cloc for 0.6.0: 12,447 lines of Python and C, 239 lines of ASDL

And ~2800 new lines of physical code:

• src for 0.6.pre15: 20,553 lines of Python
• src for 0.6.0: 23,343 lines of Python

Native Code and Bytecode Metrics

The amount of native code was reduced:

• nativedeps for 0.6.pre15: 133,291 lines
• nativedeps for 0.6.0: 130,403 lines

This is largely due to forking and modifying posixmodule.c. I did this to backport PEP 475 to Python 2, which fixed our EINTR issues.

The binary size stayed almost the same:

• ovm-build for 0.6.pre15: 1,042,936 bytes of native code (under GCC)
• ovm-build for 0.6.0: 1,048,888 bytes of native code (under GCC)

The bytecode size increased significantly, out of proportion to the increase in lines of code:

• src-bin-ratio-with-opy for 0.6.pre15: 798,036 bytes
• src-bin-ratio-with-opy for 0.6.0: 963,446 bytes

It appears this is partly due to bloat in the code generated by our ASDL implementation. I'll keep an eye on this.

I hope the MyPy / C++ translation will make these metrics obsolete, but I'll continue to keep track of them until that's done.