[HN Gopher] A Higgs-Bugson in the Linux Kernel
___________________________________________________________________
A Higgs-Bugson in the Linux Kernel
Author : Ne02ptzero
Score : 202 points
Date : 2025-07-02 18:34 UTC (1 days ago)
(HTM) web link (blog.janestreet.com)
(TXT) w3m dump (blog.janestreet.com)
| protocolture wrote:
| I love the term "Higgs Bugson". Its much better than what I
| usually do which is just call a system haunted.
| EarlKing wrote:
| Haunted? Hell, it's positively possessed.
| GTP wrote:
| I was used to the more common Heisenbug, but I find Higgs-
| Bugson more funny.
| burnt-resistor wrote:
| There is no magic at any time. All behavior lives somewhere.
| hglee wrote:
| https://lists.openwall.net/linux-kernel/2025/03/19/1374
| penguin_booze wrote:
| I wish developers--new and old alike--pay attention to the
| commit messages that goes into the kernel. Granted, it takes a
| subject matter expert to really understand what's being said,
| but the general format and layout of commit messages is
| instructive. Commit messages helps the reader/reviewer get
| their bearings; they also help to build the case from the
| bottom up.
|
| The fact that the development team is globally distributed both
| necessitates this kind of knowledge serialization and preserves
| it for posterity. It's completely different from tapping a
| colleague sitting next you on the shoulder, and saying "psst,
| can you approve this quick? It's just a bunch of fixes".
| sedatk wrote:
| > A higgs-bugson is a bug that is reported in practice but
| difficult to reproduce
|
| This was the first time I heard of "higgs-bugson". The term
| sounded so forced that I had to know how it differed from
| Heisenbug. In short, it doesn't[1].
|
| Then why did it even exist?
|
| The term somehow made it to the "Heisenbug"'s Wikipedia page[1],
| so I checked the sources. There were two and both end up at the
| same site: Jeff Atwood's blog post[2] quoting some StackOverflow
| answers to a poll-like question ("what's a programming term you
| coined?") because he wanted to remove lighthearted content from
| the site as he thought it clashed with SO's mission of educating
| people and advancing their skills[3].
|
| There was a proposal on Meta StackExchange about undeleting that
| question with the answers, but it was refused by Jeff Atwood
| again because it invited "made up stuff"[4] among other reasons.
|
| So, Wikipedia in the end, has this term in Heisenbug page because
| someone just blurted out something in 2010, it was copy-pasted to
| a blog, and then got scooped up by some news outlet. There are no
| other sources. Kagi doesn't find any instances of the term before
| it was coined on StackOverflow in 2010. For all we know,
| "gingerbreadboy" from England invented it.
|
| The irony is that the term somehow made it to the literature
| -hence the blog post here- because someone was just having fun at
| StackOverflow. It obviously either sounded good, or just clicked
| that others started using it. StackOverflow deleted the content
| that actually made a small part of computer science history
| because it wasn't "serious".
|
| In other words, StackOverflow cut off one of its history-making
| parts because it had an incomplete and simplistic view of useful.
| I think it might be possible to draw a line from their
| understanding of communities and societal dynamics to the
| downfall of StackOverflow after the emergence of AI[5].
|
| [1] https://en.wikipedia.org/wiki/Heisenbug
|
| [2] https://blog.codinghorror.com/new-programming-jargon/
|
| [3] https://stackoverflow.blog/2010/01/04/stack-overflow-
| where-w...
|
| [4] https://meta.stackexchange.com/questions/122164/can-we-un-
| de...
|
| [5] https://blog.pragmaticengineer.com/stack-overflow-is-
| almost-...
| chris_wot wrote:
| Yeah, stack overflow is dying, we all know it.
| zahlman wrote:
| > because he wanted to remove lighthearted content from the
| site as he thought it clashed with SO's mission of educating
| people and advancing their skills[3].
|
| No; he wanted to remove _discussion and socialization_ ,
| because it clashed with SO's mission of _presenting useful
| information without parsing through others ' discussion_.
|
| https://meta.stackexchange.com/questions/2950
|
| https://meta.stackexchange.com/questions/19665
|
| https://meta.stackexchange.com/questions/92107
|
| https://meta.stackexchange.com/questions/131009
|
| > In other words, StackOverflow cut off one of its history-
| making parts because it had an incomplete and simplistic view
| of useful.
|
| How does this in any way demonstrate that the view of
| usefulness was "incomplete" or "simplistic"?
|
| How is the deleted content "useful"?
|
| > I think it might be possible to draw a line from their
| understanding of communities and societal dynamics to the
| downfall of StackOverflow after the emergence of AI[5].
|
| What downfall?
|
| Before you point at any of the incoming-question-rate
| statistics: why should they be interpreted as representing a
| "downfall"? That is, why is it actually bad if fewer questions
| are asked?
|
| Before you answer that, keep in mind that Stack Overflow
| already has _more than three times_ as many publicly visible
| questions about programming as Wikipedia has articles about
| _literally anything notable_.
| robertlagrant wrote:
| > why should they be interpreted as representing a
| "downfall"?
|
| I agree, but also SO has certainly gone through ups and
| downs. It does feel as though it's now in a terminal "down"
| having invested its limited resources in things lots of the
| dedicated members didn't seem to want, instead of basic
| improvements to moderation and to chat features.
| dh2022 wrote:
| I think Heisenbug refers to a bug that stops repro'ing during
| debugging (the act of observing the system changes the system
| behavior). This bug was different: it was very rare and
| debugging it didn't make it go away.
| bux93 wrote:
| Like they say, "Stop trying to make fetch happen!"
| anonymousiam wrote:
| "The normal timeout logic can take care of retransmission in the
| unlikely case that one is needed."
|
| NFS can be run over TCP or UDP. Does the retransmission occur
| when using UDP?
| ninjha wrote:
| Yes! The retransmission logic in Linux NFS is independent of
| transport (see the `retrans` option in `mount.nfs`).
|
| Weirdly enough this also means that if you're running with TCP
| you can have retransmits at the NFS/SunRPC level _and_ at the
| TCP level depending on your configuration.
| konsalexee wrote:
| TIL higgs-bugson and Heisenbug
| snvzz wrote:
| With millions of LoCs, it is no surprise there are bugs.
|
| Worse yet, the kernel runs in supervisor mode.
|
| This kernel design is bankrupt. There's much better available,
| such as seL4+Genode.
| eddythompson80 wrote:
| seL4+Genode is equally as bankrupt. I run my code in the SMM
| anyway.
| eqvinox wrote:
| Please try keeping your snide comments to issues they actually
| apply to. This is a logic bug, with the kernel missing a piece
| of abnormality handling. You can get the exact same bug in a
| microkernel (or, FWIW, a memory safe, e.g. Rust)
| implementation; neither of those concepts help here.
| lotharcable wrote:
| > This kernel design is bankrupt. There's much better
| available, such as seL4+Genode.
|
| I am sure that the tech community would love to read the
| details of your great success in deploying microkernels for
| large variety of production workloads.
| burnt-resistor wrote:
| seL4 exhibited great advances in software engineering processes
| and advances in correctness, zero-copy microkernel IPC
| performance, and capabilities-based security, but these need
| explanation, adaptation, and evangelism to real-world use-cases
| like Linux.
|
| Microkernels have severe limitations when it comes to
| transactional boundaries of calling multiple subsystems and
| rolling back on failure.
|
| Linux has too much inertia to reinvent itself instantly or
| completely into XYZ.
|
| What would add more value would be gradual conversion to Rust
| and adding formal verification to C and Rust like specifying
| invariants in comments/metadata like frama-c and/or flux.
|
| PS: Religious judgement opinion wars are rarely constructive.
| Havoc wrote:
| Didn't know jane street did tech writeups
| Balinares wrote:
| It's a great recruitment device. It takes a certain kind of
| nerd to salivate over the glorious technical depths that such a
| write-up goes into, and for the kind of company who values this
| flavor of nerd, this is a great way to attract their attention.
| burnt-resistor wrote:
| OCaml and OxCaml https://blog.janestreet.com/introducing-
| oxcaml/
| vrnvu wrote:
| I'd like to highlight this:
|
| >NFS with Kerberos
|
| secure, simple, battle tested. no crazy architecture
|
| works so well a bug showed up in the kernel :-)
| eqvinox wrote:
| > works so well a bug showed up in the kernel :-)
|
| What _exactly_ are you trying to highlight here? Most code has
| bugs. This one is someone forgetting to stick to actual
| behavior described in 1997, it 's a mistake, mistakes happen.
| Which one of "secure", "simple", "battle tested" and "no crazy
| architecture" do you think this disproves?
|
| Or do you think CIFS or Ceph have no bugs?
| gyesxnuibh wrote:
| I think they're saying typically the kernel one of the last
| places you'd expect the bug, so it shows that it is battle
| tested?
|
| I don't think they're being snarky.
| eqvinox wrote:
| I didn't really read it as snarky, I just straight up don't
| understand what they mean (and maybe why that smiley is
| there?)
| vrnvu wrote:
| By "no crazy architecture" I meant it avoids the modern
| trend of building monstrous data platforms on top of data
| meshes, event buses, and layers of cloud abstractions.
| The kind I sometimes see, hence the smiley :-)
| burnt-resistor wrote:
| The Linux kernel needs to adopt better testing methodologies
| because they're almost entirely reliant on meatcloud CI than
| provably-correct code with invariant contracts.
| gnfargbl wrote:
| Calling this a "Higgs-Bugson" doesn't make a lot of sense.
| There's nothing uncertain or difficult to reproduce about the
| Higgs.
|
| The reason that it took so long to find was that the cross-
| section of production is very low, the decay signatures are hard
| to separate from the background, the specific energy scale it
| existed at was not well-defined, and building the LHC was (to put
| it mildly) difficult and expensive.
|
| Roughly, if you'll forgive a bad analogy from a long-lapsed
| physicist, it was the equivalent of trying to find a very weak
| glow from a specific type of bug hiding at an unknown location in
| a huge field of corn. Except that your vision was very bad, so
| you had to invent a new type of previously-unimaginably excellent
| eyeglasses to see the thing. Also before you could even start
| looking you had to expend a painful amount of time and money
| building a flashlight so incredibly huge that it needed new types
| of cryogenic cooling inventing, just to stop it from melting when
| you switched it on.
|
| If you had a software bug that you were almost certain was there,
| but you needed half of the world's GPU clusters for three years
| to locate and prove it, then _that_ would be a Higgs-Bugson.
| lisper wrote:
| A bug that shows up in production but goes away when you try to
| debug it is usually called a Heisenbug.
| alexpotato wrote:
| Regarding NFS, I've always loved this quote from the CTO at a
| hedge fund I once worked at:
|
| "NFS is lot like heroin:
|
| at first, it seems amazing.
|
| But then it ruins your life"
|
| (This is a place that did almost EVERYTHING via NFS including
| different applications communicating via shared files on NFS
| mounts. It also had the weird setup of using BOTH Linux AND
| Windows permissions on NFS mounts shared between user desktops
| [windows] an servers [linux])
| stavros wrote:
| The problem I have with reviews like these is that they're
| expressed in absolute terms. Yes, NFS might ruin my life, but
| if it ruins my life less than every other alternative, it's
| still a win.
| eqvinox wrote:
| I'd go as far as saying most networked concurrent file access
| will ruin your life one way or another, because it's just a
| _hard problem_ , and it's trying to solve it at a very odd
| layer; a "classic" fs can't really take advantage of higher
| layer transactional or other known constraints in order to
| make things work better...
| fragmede wrote:
| Google Docs solved the problem at the right layer then.
| johncolanduoni wrote:
| Continuing the analogy, many people eventually discover that
| they used NFS because they didn't understand their underlying
| problem clearly.
| burnt-resistor wrote:
| The devil is in the details, the devil you know is
| preferable, and there's yet no perfectly angelic systems or
| code (because of the widespread allergy to formal methods and
| job security).. which will lead to less evil, but still
| imperfect systems.
| jxjnskkzxxhx wrote:
| Content marketing for Jane street.
| mikeyg wrote:
| anyone else feel like the linux kernel release quality has come
| down a bit here in the 2020s? i feel like it hasn't been this bad
| since the mid 90s. anecdotally in the past couple years, i've
| experienced a data corruption bug in xfs, wonky wifi
| firmware/kernel regressions, graphics artifacts and hard crashes
| in amdgpu. my experience with mainline releases before 2020 has
| been that they're rock solid. i'd doubt myself before i doubted
| the kernel. i say all this with a deep appreciation for everyone
| and the work they're doing... my intuition says that the
| complexity of it all is reaching a tipping point that is finally
| overwhelming the ages old release engineering processes.
| nycerrrrrrrrrr wrote:
| Curious why they're using NFSv3 instead of v4?
| emmelaich wrote:
| Kerberos is 'fun'. I had to manage a system which used Kerberos
| to provide authentication between a Rubik's cube of various
| Windows flavours with various crypto standards, Linux machines of
| various versions and Java versions and apps of various maturity.
| It was an ever present source of weird behaviour and I had to
| bury myself in the innards of all these systems.
|
| I know, not directly related to the article. Just needed to vent
| bitterly.
| alienbaby wrote:
| I always thought this was rather called a 'Heisenbug'?
| 12_throw_away wrote:
| I _think_ the distinction the author is making is that:
|
| - a Heisenbug is stochastic and potentially non-local, but
|
| - a Higgs-Bugson is a bug that is known to exist (in prod) but
| is extremely hard to observe in the lab (during dev/testing).
|
| I can see it being a useful distinction. A bug that I can't
| even reproduce _at all_ sits in a different ring of hell from,
| say, a memory corruption bug that makes the program crash on
| random unrelated lines of code.
| rurban wrote:
| NFS is usually only used in mixed linux/windows environments. The
| easiest fix is to avoid NFS and esp. Windows. NFS alone is
| nightmare enough, Windows is just insanity.
___________________________________________________________________
(page generated 2025-07-03 23:01 UTC)