https://surfingcomplexity.blog/2024/03/26/the-problem-with-invariants-is-that-they-change-over-time/ Skip to content Surfing Complexity Lorin Hochstein's ramblings about software, complex systems, and incidents. The problem with invariants is that they change over time Lorin Hochstein incidents, software, systems March 26, 2024March 26, 2024 1 Minute Cliff L. Biffle blogged a great write-up of a debugging odyssey at Oxide with the title Who killed the network switch? Here's the bit that jumped out at me: At the time that code was written, it was correct, but it embodied the assumption that any loaned memory would fit into one region. That assumption became obsolete the moment that Matt implemented task packing, but we didn't notice. This code, which was still simple and easy to read, was now also wrong. This type of assumption is an example of an invariant, a property of the system that is supposed to be guaranteed to not change over time. Invariants play an important role in formal methods (for example, see the section Writing an invariant in Hillel Wayne's Learn TLA+ site). Now, consider the following: * Our systems change over time. In particular, we will always make modifications to support new functionality that we could not have foreseen earlier in the lifecycle of the system. * Our code often rests on a number of invariants, properties that are currently true of our system and that we assume will always be true. * These invariants are implicit: the assumptions themselves are not explicitly represented in the source code. That means there's no easy way to, say, mechanically extract them via static analysis. * A change can happen that violates an assumed invariant can be arbitrary far away from code that depends on the invariant to function properly. What this means is that these kinds of failure modes are inevitable. If you've been in this business long enough, you've almost certainly run into an incident where one of the contributors was an implicit invariant that was violated by a new change. If you're system lives long enough, it's going to change. And one of those changes is eventually going to invalidate an assumption that somebody made long ago, which was a reasonable assumption to make at the time. Implicit invariants are, by definition, impossible to enforce explicitly. They are time bombs. And they are everywhere. Share this: * Twitter * Facebook * Like Loading... [f2641f12e8] Published by Lorin Hochstein View all posts by Lorin Hochstein Published March 26, 2024March 26, 2024 Post navigation Previous Post When there's no gemba to go to 5 thoughts on "The problem with invariants is that they change over time" 1. [f31a1] Bart Smaalders says: March 26, 2024 at 10:41 pm Any code that is maintained long enough will have this issue in spades. When I worked at Sun, we often found assumptions in kernel code about things that had been true ten (or twenty!) years earlier, but now seemed laughable. Software gardening is critical; we just have to revist our (or our predecessors) assumptions every so often. Every part of the software "garden" needs spading at some point. Reply 2. [66f0d] Verisimilitude says: March 28, 2024 at 1:16 pm I disagree. Our systems change over time. In particular, we will always make modifications to support new functionality that we could not have foreseen earlier in the lifecycle of the system. Lorin Hochstein The article hinges on this incorrect assumption. To provide a counterexample, I'll mention that mere modularity defeats this assumption. Functionality can be fully specified, in some limited context, and never thereafter changed. An implementation of a data structure would qualify as an example; its internal representations, affecting such things as optimizations, can change, but the interface to this functionality doesn't change. It follows that systems should comprise smaller modules, so that ideally only the top-level, representing the particular program itself, needs to change. These invariants are implicit: the assumptions themselves are not explicitly represented in the source code. That means there's no easy way to, say, mechanically extract them via static analysis. Lorin Hochstein I use the Ada programming language, which makes these invariants explicit. SPARK goes even further with Ada. However, this quote is an argument in support of higher level programming systems, which have fewer and fewer assumptions. It's entirely possible to specify something at so high a level that it need never change, and a system which processes mathematical equations would be an example. Writing every program at a level where worthless details pop up repeatedly and constantly is a recipe for disaster, yes. A change can happen that violates an assumed invariant can be arbitrary far away from code that depends on the invariant to function properly. Lorin Hochstein This quote is an argument in support of small modules. What this means is that these kinds of failure modes are inevitable. Lorin Hochstein Very cute, but this is wrong for the reasons I've explained. Software can be perfect. If software be kept small, rewriting it becomes an alternation from changing it. Implicit invariants are, by definition, impossible to enforce explicitly. They are time bombs. And they are everywhere. Lorin Hochstein When I write code in Common Lisp, and especially APL, I don't really suffer from implicit invariants, although some do remain. In addition to every method I've mentioned so far, these implicit invariants should be documented where they can't be made explicit otherwise. Reply 3. [fff26] Charlie Todd says: April 22, 2024 at 12:12 pm If invariants are known at the time of development, unit tests should be written to verify their continued value. Autoconf setups are full of these checks and are important for invariants not under the developers control, like return values in essential library functions that change between platforms. Behavioral testing suites are designed to validate that externally facing interfaces haven't spontaneously started behaving differently. Linus Torvalds has noted that outsiders don't depend just on your API (ABI), but also the behavior of that interface. Behavioral dependencies also include timing, like when a call starts taking too long, they might start hitting timeouts. The end user treated the time it takes your code to do an operation as an invariant, although many developers are wise to this assumption nowadays. Unfortunately, if useful, perfect software could be written, we'd have a lot more of it. Too many leaky abstractions. Reply 4. Pingback: The problem with invariants is that they change over time - CIBERSEGURANCA 5. Pingback: chorasimilarity Leave a comment Cancel reply [ ] [ ] [ ] [ ] [ ] [ ] [ ] D[ ] Search for: [ ] [Search] Elsewhere on the Internet * Website: lorinhochstein.org * Mastodon: @norootcause@hachyderm.io * Twitter: @norootcause Recent Posts * The problem with invariants is that they change over time * When there's no gemba to go to * What if everybody did everything right? * Book Review: Trust in Numbers * Negotiability is non-negotiable Archives * March 2024 * February 2024 * January 2024 * December 2023 * September 2023 * August 2023 * June 2023 * April 2023 * March 2023 * February 2023 * January 2023 * December 2022 * November 2022 * October 2022 * September 2022 * August 2022 * July 2022 * June 2022 * May 2022 * April 2022 * January 2022 * December 2021 * November 2021 * October 2021 * September 2021 * August 2021 * July 2021 * June 2021 * May 2021 * April 2021 * February 2021 * January 2021 * December 2020 * September 2020 * August 2020 * July 2020 * June 2020 * May 2020 * April 2020 * March 2020 * February 2020 * January 2020 * November 2019 * October 2019 * September 2019 * August 2019 * July 2019 * June 2019 * May 2019 * March 2019 * January 2019 * December 2018 * October 2017 * June 2017 * May 2017 * February 2017 * January 2017 * July 2016 * June 2016 * October 2015 * August 2015 * June 2015 * April 2015 * February 2015 * November 2014 * October 2014 * September 2014 * August 2014 * July 2014 * June 2014 * May 2014 * April 2014 * March 2014 * November 2013 * September 2013 * August 2013 * June 2013 * May 2013 * March 2013 * February 2013 * December 2012 * November 2012 * October 2012 * August 2012 * July 2012 * April 2012 * March 2012 * February 2012 * January 2012 * December 2011 Categories * books * cognitive-systems-engineering * education * engineering * incidents * journalism * netflix * openstack * research * resilience * software * systems * Uncategorized * work Meta * Register * Log in * Entries feed * Comments feed * WordPress.com Blog at WordPress.com. * Comment * Reblog * Subscribe Subscribed + [surfer] Surfing Complexity Join 155 other subscribers [ ] Sign me up + Already have a WordPress.com account? Log in now. * + [surfer] Surfing Complexity + Customize + Subscribe Subscribed + Sign up + Log in + Copy shortlink + Report this content + View post in Reader + Manage subscriptions + Collapse this bar %d [b]