Why Sponsor Oils? | blog | oilshell.org

Release of OSH 0.6.pre2

This is the latest version of OSH, a bash-compatible shell:

• https://www.oilshell.org/download/oil-0.6.pre2.tar.xz (gzip version)

Please try it on your shell scripts and report bugs! To build and run it, follow the instructions in INSTALL.txt.

If you're new to the project, see Why Create a New Shell?. OSH can run unmodified shell scripts that are thousands of lines long, as described in the announcement for OSH 0.4.

Although I released version 0.6.pre1 just a few days ago, I'm making another release because Unicode support in basically done, and the binary is significantly smaller. I continue to chip away the problem of OSH being too big and too slow.

The release process involves an array of tests and benchmarks, which usually catch bugs. This release was no exception.

The following sections summarize the raw changelog.

Visible Changes

OSH now respects UTF-8 in two places:

1. The string length operator ${#s} counts code points, not bytes.
2. String slicing like ${s:1:3} also counts code points, not bytes.

In addition to implementing #2, I cleaned up related error conditions:

• UTF-8 decode errors are fatal under set -o strict-word-eval, an OSH-specific mode. (TODO: Write a blog post about the bad behavior of bash and zsh in this situation, under #shell-the-bad-parts)
• String slices with negative arguments like ${s: -1 : -3} are disallowed. Another candidate for #shell-the-bad-parts: the second argument to slicing is a length if 0 or positive, but a position if negative!
• set -o strict-arith was re-enabled. In this mode, an expression like $((s+5)) will fail if s is an invalid string like abc, rather than coercing to 0 and returning 5. (OSH is also the only shell that has this!)

NOTE: The motivation for implementing UTF-8 ourselves was to remove Python's unicodeobject.c from the build, which is over 10K lines of code. This was done in OSH 0.5.

Under the Hood

• ASDL schemas are no longer included in the binary. We compile them offline into Python code and reflection metadata (now stored with pickle, an tiny VM that can serialize graphs with sharing).
• Removed the dependency on platform module, which was just a wrapper around os.uname() and sys.version.
• As a result of those two changes, we no longer need the re module. So files like _sre.c and sre_parse.py from CPython have been removed. We now depend on ~10K fewer lines of code! Detailed measurements are below.
• Removed bytearrayobject.c. (Late in the release process, I discovered that this broke oheap encoding, which was an experiment not visible to users.)
• Removed unicodeobject.h (a straggler) and future.c from the build.

Selected Metrics

The tests and metrics published with each release quantify the changes above:

• spec tests for 0.6.pre1: 1038 tests, 873 passing, 117 failing
• spec tests for 0.6.pre2: 1045 tests, 880 passing, 115 failing

Lines of native code went down because I removed more parts of CPython:

• nativedeps for 0.6.pre1: 151,656 lines
• nativedeps for 0.6.pre2: 140,302 lines

Lines of Python code also went down because I removed parts of CPython as described above:

• pydeps for 0.6.pre1: 31,740 lines
• pydeps for 0.6.pre2: 27,701 lines

This is reflected in the smaller binary, which is faster to build:

• ovm-build for 0.6.pre1: 1,497,688 bytes of native code (under GCC)
• ovm-build for 0.6.pre2: 1,366,040 bytes of native code (under GCC)

The bytecode size went down as well:

• pyc-bytes for 0.6.pre1: 1,024,357 bytes
• pyc-bytes for 0.6.pre2: 906,010 bytes