OSH Parser Performance

We time $sh -n $file for various files under various shells, and repeat then run under cachegrind for stable metrics.

Source code: oil/benchmarks/osh-parser.sh

Summary

Instructions Per Line (via cachegrind)

Lower numbers are generally better, but each shell recognizes a different language, and Oil uses a more thorough parsing algorithm. In thousands of "I refs".

shell label total lines thousand irefs per line
bash 118,863 15.4
dash 111,674 1.6
mksh 111,674 5.7
osh-native 118,863 26.0

(zsh isn't measured because zsh -n unexpectedly forks.)

Average Parsing Rate, Measured on Two Machines (lines/ms)

Shell startup time is included in the elapsed time measurements, but long files are chosen to minimize its effect.

shell label total lines host broome host lenny
dash 111,674 1,458 2,660
mksh 111,674 583 981
bash 118,863 344 505
zsh 118,863 104 160
osh-native 118,863 160 138
osh-cpython 118,863 7 6
osh-ovm 118,863 7 5

Per-File Measurements

Instructions Per Line (in thousands)

bash dash mksh osh-native num lines filename
15.7 1.4 4.4 10.0 1,679 t4014-format-patch.sh
16.7 2.1 8.4 39.1 1,733 functions
23.7 NA NA 31.4 2,029 configure-helper.sh
17.9 2.2 8.9 42.3 2,512 abuild
13.6 1.8 5.9 27.5 2,698 Build.sh
13.7 1.0 3.3 6.4 3,123 t9300-fast-import.sh
32.0 NA NA 41.4 5,160 test-cmd-util.sh
18.2 1.8 6.1 27.9 9,661 ltmain.sh
17.4 1.9 6.4 29.2 20,489 configure
12.9 1.5 5.4 23.8 69,779 configure-coreutils

Elasped Time (milliseconds)

host label bash dash mksh zsh osh-ovm osh-cpython osh-native osh to bash ratio num lines filename
broome 9 2 4 99 155 161 7 0.8 1,679 t4014-format-patch.sh
broome 10 3 7 11 410 404 24 2.3 1,733 functions
broome 15 NA NA 23 462 430 23 1.6 2,029 configure-helper.sh
broome 15 4 11 16 615 564 32 2.1 2,512 abuild
broome 13 3 8 77 460 418 27 2.1 2,698 Build.sh
broome 13 2 5 114 175 140 10 0.7 3,123 t9300-fast-import.sh
broome 31 NA NA 27 1,343 1,182 50 1.6 5,160 test-cmd-util.sh
broome 35 9 25 41 1,593 1,411 76 2.2 9,661 ltmain.sh
broome 62 17 39 123 3,324 2,961 126 2.0 20,489 configure
broome 143 37 92 609 9,464 8,647 368 2.6 69,779 configure-coreutils
lenny 3 1 2 105 126 124 10 2.9 1,679 t4014-format-patch.sh
lenny 4 1 3 14 434 434 29 7.0 1,733 functions
lenny 6 NA NA 26 402 409 53 8.7 2,029 configure-helper.sh
lenny 7 3 5 9 665 775 101 14.4 2,512 abuild
lenny 6 2 4 79 442 401 44 7.7 2,698 Build.sh
lenny 7 1 2 120 142 147 19 2.9 3,123 t9300-fast-import.sh
lenny 21 NA NA 11 1,425 1,485 55 2.6 5,160 test-cmd-util.sh
lenny 23 4 11 17 2,013 1,420 57 2.4 9,661 ltmain.sh
lenny 46 10 22 55 4,604 3,580 121 2.6 20,489 configure
lenny 112 22 65 304 13,171 9,965 374 3.3 69,779 configure-coreutils

Parsing Rate (lines/ms)

host label bash dash mksh zsh osh-ovm osh-cpython osh-native num lines filename
broome 195 911 398 17 11 10 232 1,679 t4014-format-patch.sh
broome 167 647 248 162 4 4 73 1,733 functions
broome 136 NA NA 87 4 5 87 2,029 configure-helper.sh
broome 168 709 228 159 4 4 78 2,512 abuild
broome 214 848 342 35 6 6 100 2,698 Build.sh
broome 242 1,404 585 27 18 22 323 3,123 t9300-fast-import.sh
broome 165 NA NA 189 4 4 102 5,160 test-cmd-util.sh
broome 274 1,033 388 234 6 7 126 9,661 ltmain.sh
broome 330 1,188 530 167 6 7 163 20,489 configure
broome 489 1,911 754 115 7 8 190 69,779 configure-coreutils
lenny 493 2,123 795 16 13 14 172 1,679 t4014-format-patch.sh
lenny 420 1,580 531 125 4 4 60 1,733 functions
lenny 338 NA NA 77 5 5 39 2,029 configure-helper.sh
lenny 361 926 533 269 4 3 25 2,512 abuild
lenny 476 1,790 742 34 6 7 61 2,698 Build.sh
lenny 466 3,041 1,514 26 22 21 163 3,123 t9300-fast-import.sh
lenny 243 NA NA 480 4 3 93 5,160 test-cmd-util.sh
lenny 413 2,645 891 559 5 7 170 9,661 ltmain.sh
lenny 447 2,152 942 370 4 6 170 20,489 configure
lenny 624 3,220 1,067 229 5 7 186 69,779 configure-coreutils

Memory Usage (Max Resident Set Size in MB)

Again, Oil uses a different algorithm (and language) than POSIX shells. It builds an AST in memory rather than just validating the code line-by-line.

host label bash dash mksh zsh osh-ovm osh-cpython osh-native num lines filename
broome 3.4 1.7 1.9 4.1 14.7 18.4 7.2 1,679 t4014-format-patch.sh
broome 3.4 1.7 1.9 4.0 17.6 21.2 8.8 1,733 functions
broome 3.5 NA NA 3.9 18.1 21.9 9.5 2,029 configure-helper.sh
broome 3.5 1.8 2.0 4.1 19.7 23.3 11.1 2,512 abuild
broome 3.5 1.7 2.0 4.1 18.4 22.3 9.7 2,698 Build.sh
broome 3.5 1.6 1.9 4.1 14.9 18.4 7.3 3,123 t9300-fast-import.sh
broome 3.5 NA NA 4.1 28.4 32.5 17.6 5,160 test-cmd-util.sh
broome 4.4 2.2 3.0 4.5 30.9 35.5 21.4 9,661 ltmain.sh
broome 4.0 2.5 2.7 4.3 51.3 58.2 34.2 20,489 configure
broome 3.6 1.8 2.3 4.1 116.8 130.2 91.7 69,779 configure-coreutils
lenny 3.4 1.8 1.9 4.1 14.8 18.2 7.1 1,679 t4014-format-patch.sh
lenny 3.5 1.7 1.7 4.0 17.7 21.3 9.2 1,733 functions
lenny 3.5 NA NA 4.1 18.1 21.7 9.7 2,029 configure-helper.sh
lenny 3.5 1.7 2.0 4.1 19.7 23.4 11.0 2,512 abuild
lenny 3.4 1.7 2.0 4.1 18.2 22.1 9.5 2,698 Build.sh
lenny 3.5 1.6 1.9 3.9 15.0 18.5 7.4 3,123 t9300-fast-import.sh
lenny 3.6 NA NA 4.1 28.4 32.7 17.7 5,160 test-cmd-util.sh
lenny 4.5 2.2 3.1 4.5 30.9 35.7 21.6 9,661 ltmain.sh
lenny 4.0 2.4 2.7 4.4 51.3 58.2 34.2 20,489 configure
lenny 3.7 1.9 2.2 4.2 116.9 130.4 91.7 69,779 configure-coreutils

Shell and Host Details

shell label shell id
bash bash-9af8f89f
dash dash-9ff48631
mksh mksh-e3755400
zsh zsh-a785003a
osh-cpython osh-9d3a7f12
osh-ovm osh-39c84017
osh-native osh-7a6ca51a
host label host id
broome broome-7a69aabc
lenny lenny-f88955fe

Raw Data

filename
lenny.2023-03-07__20-58-16.cachegrind.tsv
broome.2023-03-07__22-05-08.times.csv
lenny.2023-03-07__20-58-16.times.csv