source | all docs for version 0.8.0 | all versions | oilshell.org

Warning: Work in progress! Leave feedback on Zulip or Github if you'd like this doc to be updated.

Notes on Oil's Architecture

This doc is for contributors or users who want to understand the Oil codebase. These internal details are subject to change.

Links

• Contributing (wiki) helps you change the code for the first time.
• README describes how the code is organized.
• Interpreter State describes the interpreter's user-facing data structures.
• Parser Architecture
• OSH Word Evaluation Algorithm (wiki) describes shell's complex word evaluation. Oil uses Simple Word Evaluation instead.

Source Code

Build Dependencies

• Essential: libc
• Optional: GNU readline (TODO: other line editing libraries).
• Only in the OVM build (as of March 2020): yajl

Borrowed Code

• ASDL front end from CPython (heavily refactored)
• frontend/tdop.py: Adapted from tinypy, but almost no original code remains
• pgen2
• All of OPy (will be obsolete)
  • compiler2 from stdlib
  • byterun
• Build Dependency: MyPy

Metaprogramming / Generated Code

• Try ls */*_def.py */*_gen.py
  • The def.py files are abstract definitions. They're not translated by mycpp.
  • The gen.py files generate source code in Python and C++ from these definitions.
  • For example, we define the core Id type and the lexing rules abstractly.
  • TODO: Details on each def / gen pair.
• See build/dev.sh and build/codegen.sh

Other Cross-Cutting Observations

Where $IFS is Used

• Splitting of unquoted substitutions
• The read builtin
• To split words in compgen -W (bash only)

Shell Function Callbacks

• Completion hooks registered by complete -F ls_complete_func ls
• bash has a command_not_found hook; OSH doesn't yet

Where Unicode is Respected

See the doc on Unicode.

Parse-time and Runtime Pairs

In OSH:

• echo -e '\x00\n' and echo $'\x00\n' (OSH shares lexer rules between them)
• test / [ and [[ (OSH shares the parser and evaluator)
• Static vs. Dynamic Assignment. local x=$y vs. s='x=$y'; local $s.
  • All shells have both notions!

Other Pairs:

• expr and $(( )) (expr not in shell)
• Later:
  • printf can have a static variant like ${myfloat %.3f}
  • find and our own language (although this may be done with blocks)

State Machines

• $IFS splitting in osh/split.py
• compadjust needs to split partial argv by user-defined delimiters, e.g. :=

The point of a state machine is to make sure all cases are handled!

Other Topics

• Dependency Injection