http://www.oilshell.org/blog/2021/11/release-0.9.4.html blog | oilshell.org Oil 0.9.4 - User Feedback 2021-11-20 This is the latest version of Oil, a Unix shell that's our upgrade path from bash: Oil version 0.9.4 - Source tarballs and documentation. To build and run it, follow the instructions in INSTALL.txt. The wiki describes How To Test OSH and Where To Send Feedback. If you're new to the project, see Why Create a New Shell? and posts tagged #FAQ. Table of Contents What's Happened Recently? Summary of Changes Prompt Enhancements The read builtin Bug: Pipelines Aren't Blocked on Background Processes The Oil Language Closed Issues What's Next? Links Appendix: Metrics for the 0.9.4 Release Spec Tests Source Code Size Benchmarks Native Code Metrics What's Happened Recently? Many things have happened in the project, but as usual the blog is behind! Highlights: * We had the first "oil-dev walkthrough" over videoconference yesterday. It surfaced a lot of great feedback and context! + Notes on Zulip: #oil-dev > oil-dev walkthrough. + I want to have more events like this. If you're interested in attending, let me know on Zulip! * It was motivated in part by a proposal to use Oil in Nix, and the resulting discussion. I also didn't forget about the "big ideas" I wrote about this summer. I plan to return to those in some form. However, this is a "workmanlike" release, to get back in the rhythm of things after traveling and moving. Summary of Changes For the last couple weeks, I focused on fixing user-reported bugs and addressing feature requests. Thanks to Miles Alan and bb010g for particularly thorough testing. Prompt Enhancements To customize your bash prompt, you use a cryptic language in the $PS1 variable. This release implements more parts of that language: * \s and \v evaluate to the shell and version -- e.g. osh and 0.9.4 * \A and \D{} for substituting the current time -- 24 hour format and strftime() format, respectively We also ensure that $PS1 is set in interactive shells. (An interactive shell is one that's started without a script argument, and where stdin is connected to a terminal.) The read builtin * If the read() syscall is interrupted, run trap handlers and retry. (bash and zsh agree on this behavior; it's not clear what POSIX says.) * Set the terminal mode appropriately for read -d vs. read. This fixes a bug where the backspace key didn't work interactively. Bug: Pipelines Aren't Blocked on Background Processes Miles Alan reported an interesting process concurrency bug. It dates back to late 2018 when pipelines started to use the more sophisticated shopt -s lastpipe semantics. By default, OSH runs the last element of a pipeline in the main shell process. Among other things, this means that the variable binding in echo foo | read myvar persists. The bug is too subtle to explain here, but it reminds of this canonical example of why process-based concurrency is tricky: sleep 1 & # start background process # The shell has to wait() for 2 processes in this pipeline my-command | wc -l But the sleep process might finish before both, in between them, or after both. In other words, process done notifications via waitpid(-1) are asynchronous, and serialized, so shells need data structures to keep track of it all. This is an important part of the #shell-runtime I haven't written about. The Oil Language The feedback above is solidly in the OSH portion of the project, but we use it to improve the Oil language. Some thoughts: * Oil still needs a prompt hook, but we don't want to use the $PS1 language. It interleaves parsing and evaluation in a confusing and potentially unsafe way. + Instead, Oil will likely use the builtin sub mechanism with procs. If you're interested in this, let me know on issue 498 ! * The read changes reminded me that the read invokes the read() syscall for every byte! This is very unbuffered I/O! + So I've defined Oil's read --line and read --all to use buffered I/O instead. I think this is more appropriate for batch programs, and is consistent with Python and Perl. * The pipeline fix above applies to both OSH and Oil. In both cases, pipelines behave like zsh: they behave as if the bash shopt -s lastpipe option is set. + Recall that OSH and Oil are variants of the same interpreter, and live in the same binary. In particular, they share the runtime that executes pipelines. We speak of them separately, but there's a smooth upgrade path from one to another. Closed Issues # Pipeline launched from main process waits for a background 1002 process # Unhandled termios exception 1001 # read builtin: backspace interpreted as ^? 1000 #996 osh throws an `OverflowError` & dumps a stack trace on `(return 2147483648)` or `(exit 2147483648)` #986 Builtin read not interruptible via signal #982 Set PS1 in interactive shells #962 Support \A in PS1 #748 Support \D{} in PS1 and PS0 What's Next? I want to write these blog posts next: Oil 0.9.3 - Extended Globs for Nix. Motivated by Nix, I implemented one of the "last" OSH features. This is a good opportunity to restate and explain the principles behind the design and implementation of OSH. I hope to answer many questions on the RFC thread with concrete examples and general principles. Oil Winter Blog Backlog. As mentioned, the blog is backed up, and I want to catch up. This might overflow into a few parts, like the Summer Blog Backlog did (part 1 and part 2). Links * In case I don't write about it soon enough on the blog, here's a Hacker News comment that summarizes the state of the project. * There is also more "content" on the #blog-ideas Zulip channel. Feel free to reply there and let me know what you want to read! Reminder: Please try Oil and send feedback! The best feedback results from trying OSH and Oil on real programs. I just got some great feedback from Arturo Martin-de-Nicolas, which surfaced at least one bug, and reminded me of a gap in the Oil language (issue #1011). * #oil-dev > Feedback on Port to OSH Appendix: Metrics for the 0.9.4 Release These metrics help me keep track of the project. Let's compare this release with version 0.8.12, which I announced in July. Spec Tests The spec test suites for both OSH and Oil continue to expand and turn green. The 0.9.3 work on extended globs resulted in whole suite of tests, because they can appear in many contexts. * OSH spec tests for 0.8.12: 1905 tests, 1693 passing, 83 failing * OSH spec tests for 0.9.4: 1958 tests, 1738 passing, 84 failing --------------------------------------------------------------------- * Oil spec tests for 0.8.12: 372 tests, 340 passing, 32 failing * Oil spec tests for 0.9.4: 396 tests, 366 passing, 30 failing Source Code Size We have ~250 significant lines of code: * cloc for 0.8.12: 18,753 lines of Python and C, ~337 lines of ASDL * cloc for 0.9.4: 19,085 lines of Python and C, 334 lines of ASDL And ~600 new lines of physical code: * src for 0.8.12: 35,606 in OSH and common libraries, 4,814 in Oil * src for 0.9.4: 36,224 in OSH and common libraries, 4,867 in Oil Benchmarks I started using new benchmark machines in version 0.9.2, so let's use that as our baseline. It looks like there was no change. * Parser Performance for 0.9.2: 8.6 thousand irefs per line * Parser Performance for 0.9.4: 8.6 thousand irefs per line Runtime: * Runtime Performance for 0.9.2: 87.8 and 88.1 seconds running CPython's configure * Runtime Performance for 0.9.4: 88.6 and 88.2 seconds running CPython's configure Native Code Metrics I've been maintaining the C++ translation behind the scenes, trying to make sure we don't dig ourselves into a hole. This comparison looks OK: * OSH C++ spec tests for 0.8.12: 1696 tests, 1171 pass in Python, 929 pass in C++ * OSH C++ spec tests for 0.9.4: 1688 tests, 1555 pass in Python, 937 pass in C++. But I noticed a regression versus version 0.9.3, released last month: * OSH C++ spec tests for 0.9.3: 1679 tests, 1547 pass in Python, 1109 pass in C++. Looking over the results, I see many 255 exit codes that are supposed to be 0. I believe this is due a change that started truncating exit codes, which fixed a user-reported crash. I suspect it didn't translate to C++ correctly, e.g. because Python has arbitrary size integers and C++ has fixed size integers. We have ~1000 more lines of native code: * oil-cpp for 0.8.12: 91,081 lines of C++ * oil-cpp for 0.9.4: 92,115 lines of C++ But curiously the binary is slightly smaller: * ovm-build for 0.8.12: 1,368,920 bytes of native code (under GCC) * ovm-build for 0.9.4: 1,350,544 bytes of native code (under GCC) * Discuss This Post on Reddit * Get notified about new posts via @oilshellblog on Twitter * Read Posts Tagged: oil-release shell-runtime * Back to the Blog Index