[HN Gopher] Kernel bugs hide for 2 years on average. Some hide f...
___________________________________________________________________
Kernel bugs hide for 2 years on average. Some hide for 20
Author : kmavm
Score : 272 points
Date : 2026-01-08 02:18 UTC (20 hours ago)
(HTM) web link (pebblebed.com)
(TXT) w3m dump (pebblebed.com)
| sedatk wrote:
| Firefox bugs stay in the open for that long.
| steveklabnik wrote:
| One of my favorite Firefox bugs was some I don't quite remember
| the details of, but went something like this:
|
| "There's a crash while using this config file." Something more
| complex than that, but ultimately a crash of some kind.
|
| Years later, like 20 years later, the bug was closed. You see,
| they re-wrote the config parser in Rust, and now this is
| fixed."
|
| That's cool but it's not the part I remember. The part I always
| think about is, imagine responding to the bug right after it
| was opened with "sorry, we need to go off and write our own
| programming language before this bug is fixed. Don't worry,
| we'll be back, it's just gonna take some time."
|
| Nobody would believe you. But yet, it's what happened.
| nurettin wrote:
| To be fair, any rewrite could have fixed it, didn't have to
| wait for Rust.
| Xss3 wrote:
| But that take ruins all the intrigue of their comment...
| But youre spot on. They fabricated a story.
| Yossarrian22 wrote:
| No, Graydon Hoare took one look at the config code, went
| "fuck this" and decided to create a new language instead.
| steveklabnik wrote:
| I didn't say otherwise. Rust is not the point here.
| mmooss wrote:
| All software has long-lived bugs. None are bug-free, at any
| point in their existance, so it's almost inevitable. Have you
| seen Windows' bug tracker?
|
| The anti-Firefox mob really is striving to take shots at it.
|
| The point of the article isn't a criticism of Linux, but an
| analysis that leads to more productive code review.
| snvzz wrote:
| Millions of lines of code, all running in supervisor mode.
|
| One bug is all it takes to compromise the entire system.
|
| The monolithic UNIX kernel was a good design in the 60s; Today,
| we should know better[0][1].
|
| 0. https://sel4.systems/
|
| 1. https://genode.org/
| windowssuperfi wrote:
| Yeah cause windows is amazing Or maybe macos? Ignore their
| freebsd parts of course.
| DowsingSpoon wrote:
| Yes. As far as kernels go, NT was pretty damn good.
|
| So is Mach, by the way, if you can afford the microkernel
| performance overhead.
| dundarious wrote:
| If you include all the drivers too (which surely makes the
| comparison more accurate), is that still the case?
| nobodyandproud wrote:
| Windows NT 3.x was a true microkernel. Microsoft ruined
| it but the design was quite good and the driver question
| was irrelevant, until they sidestepped HAL.
|
| The Linux kernel was and is a monstrosity.
| WalterGR wrote:
| What do you meant by them sidestepping the HAL?
| avadodin wrote:
| I think the biggest one is that the whole GDI library was
| moved into the Kernel in 3.5x because the performance was
| terrible at the time.
|
| I don't think they ever intended to keep all drivers
| strictly userland, though. Just the service side.
| nobodyandproud wrote:
| Mind you I don't have access to Microsoft code, so this
| is all indirect, and a lot of this knowledge was when I
| was fledgling developer.
|
| The Windows NT code was engineered to be portable across
| many different architectures--not just X86--so it has a
| hardware abstraction layer. The kernel only ever
| communicated to the device-driver implementation through
| this abstraction layer; so the kernel code itself was
| isolated.
|
| That doesn't mean the device drivers were running in
| user-land privilege, but it does mean that the kernel
| code is quite stable and easy to reason about.
|
| When Microsoft decided to compromise on this design, I
| remember senior engineers--when I first started my career
| --being abuzz about it for Windows NT 4.0 (or apparently
| earlier?).
| pjmlp wrote:
| This is outdated since Windows Vista, and even more so in
| Windows 11.
| nobodyandproud wrote:
| Windows Vista isn't Windows NT 3.x. In the internal
| versioning, it's not even 4.0.
| pjmlp wrote:
| Indeed, it is something better, Windows NT 6.0.
|
| And it is irrelevant anyway, given that this comment was
| written from 10.0.26100.
| nobodyandproud wrote:
| Oh, I see.
|
| You're saying they improved the design. I know they added
| user-privilege device driver support for USB (etc).; did
| they revert the display compromise/mess as well?
| johncolanduoni wrote:
| Mach is not a very good microkernel at all, because the
| overhead is much higher than necessary. The L4 family's IPC
| design is substantially more efficient, and that's why
| they're used in actual systems. Fuchsia/Zircon have
| improved on the model further.
|
| Someone will of course bring up XNU, but the microkernel
| aspect of it died when they smashed the FreeBSD kernel into
| the codebase. DriverKit has brought some userspace drivers
| back, but they use shared memory for all the heavy lifting.
| saagarjha wrote:
| > Mach is not a very good microkernel at all, because the
| overhead is much higher than necessary. The L4 family's
| IPC design is substantially more efficient, and that's
| why they're used in actual systems.
|
| As opposed to Mach, which is not used in any actual
| systems
| johncolanduoni wrote:
| I mentioned XNU below. It doesn't really count as a
| microkernel if you, you know, don't actually use the
| microkernel part. At least for the 30 years between the
| FreeBSD collision and the introduction of DriverKit,
| which does most of its IPC through shared memory (because
| the mach ports are not efficient enough, I would assume).
| heavyset_go wrote:
| XNU monolith-ized itself over time, even over some
| microkernel-esque boundaries.
| cosmic_cheese wrote:
| Apple at least has been making a concerted effort to kick
| more macOS/iOS functionality out into userland in the past
| several years.
| pjmlp wrote:
| Just like Windows since Vista.
| lifetimerubyist wrote:
| NT is actually a pretty good kernel. NTFS and the userland is
| what is shit.
| speed_spread wrote:
| NTFS is a beast of a filesystem and has been nothing but
| solid for 25+ years. The performance grievances ignore the
| warranties that NTFS offers vs many antiquated POSIX
| filesystems.
| johncolanduoni wrote:
| What warranties? I assume you're comparing it to ext4 and
| not e.g. ZFS?
| speed_spread wrote:
| Here are a few
|
| - mandatory byte-range locks enforced by the kernel
|
| - explicit sharing modes
|
| - guarantees around write ordering and durability
|
| - safe delete-on-close
|
| - first-class cache coherency contracts for networked
| access
|
| POSIX aims for portability while NTFS/Win32 aims for
| explicit contracts and enforced behavior. For apps
| assuming POSIX semantics (e.g. git) NTFS feels rigid and
| weird. Coming the other way from NTFS, POSIX looks
| "optimistic" if not downright sloppy.
|
| Of course ZFS et al. are more theoretically more robust
| than EXT4 but are still limited by the lowest common
| denominator POSIX API. Maybe you can detect that you're
| dealing with a ZFS backed volume and use extra ioctls to
| improve things but its still a messy business.
| pjmlp wrote:
| Also to point out that outside UNIX, surviving mainframe
| and micros, the filesystems are closer to NTFS than UNIX
| world, in regards to what is enforced.
|
| There is also the note that some of them are more
| database like, than classical filesystems.
|
| Ah, and modern Windows also has Resilient File System
| (ReFS), which is used on Dev Drives.
| johncolanduoni wrote:
| These are pretty much all about mandatory locking. Which
| giveth and taketh away in my experience. I've had
| substantially fewer weird file handling bugs in my Linux
| code than my Windows code. POSIX is very loosey-goosey in
| general, but Linux's VFS system + ext4 has a much
| stronger model than the general POSIX guarantees.
|
| `FILE_FLAG_DELETE_ON_CLOSE`'s equivalent on Posix is just
| rm. Windows doesn't let you open new handles to
| `FILE_FLAG_DELETE_ON_CLOSE`ed files anyway, so it's
| effectively the same. The inode will get deleted when the
| last file description is removed.
|
| NFS is a disaster though, I'll give you that one. Though
| mandatory locks on SMB shares hanging is also very
| aggravating in its own right.
| edoceo wrote:
| Userland peaked in Windows 2000
| IcePic wrote:
| I think NTFS get a bit of crap from the OS above it adding
| limitations. If you read up on what NTFS allows, it is far
| better than what Windows and the explorer allows you to do
| with it.
| bawolff wrote:
| Year of HURD on the desktop?
| calvinmorrison wrote:
| Highly unrealistic rewrite disease
| tlb wrote:
| My conclusion is that microkernels offer some protection from
| random reboots, but not much against hacking
|
| Say the USB system runs in its own isolated process. Great, but
| if someone pwns the USB process they can change disk contents,
| intercept and inject keystrokes, etc. You can usually leverage
| that into a whole system compromise.
|
| Same with most subsystems: GPU, network, file system process
| compromises are all easily leveraged to pwn the whole system.
| __bjoernd wrote:
| How are SEL4 and Genode going for you in your day-to-day
| compute usage?
| josefx wrote:
| Of course by now processor manufacturers decided that blowing
| holes into the CPUs security model to make it go faster was the
| way to go. So your micro kernel is stuck on a hardware security
| model that looks like swiss cheese and smells like
| Surstromming.
| esseph wrote:
| Imagine if no one outside a select circle ever got to examine the
| code.
| immibis wrote:
| Everything is open source if you're skilled with Ghidra.
|
| We call AI models "open source" if you can download the binary
| and not the source. Why not programs?
| KK7NIL wrote:
| > We call AI models "open source" if you can download the
| binary and not the source.
|
| Who's "we"? There's been quite a lot of pushback on this
| naming scheme from the OSS community, with many preferring
| the term "open weights".
| serf wrote:
| >We call AI models "open source" if you can download the
| binary and not the source. Why not programs?
|
| the weights of a model aren't equivalent to the binary output
| of source code, no matter how you try to stretch the
| metaphor.
|
| >why not
|
| because we aren't beholden to change all definitions and
| concepts because some guy at some corp said so.
| immibis wrote:
| Unless that corp is OSI, right?
| heavyset_go wrote:
| Binaries and AI models can be inscrutable. They're meant to
| be interpreted by machines.
|
| We want human readable, comprehensible, reproducible and
| maintainable sources at minimum when we say open source.
| dspillett wrote:
| North Korea is called a "democratic people's republic". Just
| because one thing that really isn't <whatever> is called
| <whatever> by the people in coontrol of it, doesn't mean that
| it is or that incorrectly calling other things <whatever> is
| correct.
| eulgro wrote:
| From the stats we see that most bugs effectively come from the
| limitations of the language.
|
| Impressive results on the model, I'm surprised they improved it
| with very simple heuristics. Hopefully this tool will be made
| available to the kernel developers and integrated to the
| workflow.
| NewsaHackO wrote:
| It may be just my system, but the times look like hyperlinks but
| aren't for some reason. It is especially disappointing that the
| commit hashes don't link to the actual commit in the kernel repo.
| Telaneo wrote:
| They're <strong> tags with color:#79635c on hover in the CSS. A
| really weird style choice for sure, but semantically they
| aren't meant to be links at all.
| NewsaHackO wrote:
| I know, I am saying they _should_ be links, as it is what one
| would expect from an article like this.
| Fiveplus wrote:
| Before the "rewrite it in Rust" comments take over the thread:
|
| It is worth noting that the class of bugs described here (logic
| errors in highly concurrent state machines, incorrect hardware
| assumptions) wouldn't necessarily be caught by the borrow
| checker. Rust is fantastic for memory safety, but it will not
| stop you from misunderstanding the spec of a network card or
| writing a race condition in unsafe logic that interacts with DMA.
|
| That said, if we eliminated the 70% of bugs that are memory
| safety issues, the SNR ratio for finding these deep logic bugs
| would improve dramatically. We spend so much time tracing
| segfaults that we miss the subtle corruption bugs.
| anon-3988 wrote:
| > It is worth noting that the class of bugs described here
| (logic errors in highly concurrent state machines, incorrect
| hardware assumptions) wouldn't necessarily be caught by the
| borrow checker. Rust is fantastic for memory safety, but it
| will not stop you from misunderstanding the spec of a network
| card or writing a race condition in unsafe logic that interacts
| with DMA.
|
| Rust is not just about memory safety. It also have algebraic
| data types, RAII, among other things, which will greatly help
| in catching this kind of silly logic bugs.
| JuniperMesos wrote:
| Yeah, Rust gives you much better tools to write highly
| concurrent state machines than C does, and most of those
| tools are in the type system and not the borrow checker per
| se. This is exactly what the Typestate pattern
| (https://docs.rust-embedded.org/book/static-
| guarantees/typest...) is good at modeling.
| aw1621107 wrote:
| > It is worth noting that the class of bugs described here
| (logic errors in highly concurrent state machines, incorrect
| hardware assumptions)
|
| While the bugs you describe are indeed things that aren't
| directly addressed by Rust's borrow checker, I think the
| article covers more ground than your comment implies.
|
| For example, a significant portion (most?) of the article is
| simply analyzing the gathered data, like grouping bugs by
| subsystem: Subsystem Bug Count Avg
| Lifetime drivers/can 446 4.2 years
| networking/sctp 279 4.0 years networking/ipv4
| 1,661 3.6 years usb 2,505 3.5
| years tty 1,033 3.5 years
| netfilter 1,181 2.9 years networking
| 6,079 2.9 years memory 2,459 1.8
| years gpu 5,212 1.4 years bpf
| 959 1.1 years
|
| Or by type: Bug Type Count Avg
| Lifetime Median race-condition 1,188 5.1 years
| 2.6 years integer-overflow 298 3.9 years 2.2
| years use-after-free 2,963 3.2 years 1.4 years
| memory-leak 2,846 3.1 years 1.4 years buffer-
| overflow 399 3.1 years 1.5 years refcount
| 2,209 2.8 years 1.3 years null-deref 4,931
| 2.2 years 0.7 years deadlock 1,683 2.2
| years 0.8 years
|
| And the section describing common patterns for long-lived bugs
| (10+ years) lists the following:
|
| > 1. Reference counting errors
|
| > 2. Missing NULL checks after dereference
|
| > 3. Integer overflow in size calculations
|
| > 4. Race conditions in state machines
|
| All of which cover more ground than listed in your comment.
|
| Furthermore, the 19-year-old bug case study is a refcounting
| error not related to highly concurrent state machines or
| hardware assumptions.
| johncolanduoni wrote:
| It depends what they mean by some of these: are the state
| machine race conditions logic races (which Rust won't
| trivially solve) or data races? If they are data races, are
| they the kind of ones that Rust will catch (missing
| atomics/synchronization) or the ones it won't (bad atomic
| orderings, etc.).
|
| It's also worth noting that Rust doesn't prevent integer
| overflow, and it doesn't panic on it by default in release
| builds. Instead, the safety model assumes you'll catch the
| overflowed number when you use it to index something (a
| constant source of bugs in unsafe code).
|
| I'm bullish about Rust in the kernel, but it will not solve
| all of the kinds of race conditions you see in that kind of
| context.
| aw1621107 wrote:
| > are the state machine race conditions logic races (which
| Rust won't trivially solve) or data races? If they are data
| races, are they the kind of ones that Rust will catch
| (missing atomics/synchronization) or the ones it won't (bad
| atomic orderings, etc.).
|
| The example given looks like a generalized example:
| spin_lock(&lock); if (state == READY) {
| spin_unlock(&lock); // window here where
| another thread can change state do_operation();
| // assumes state is still READY }
|
| So I don't think you can draw strong conclusions from it.
|
| > I'm bullish about Rust in the kernel, but it will not
| solve all of the kinds of race conditions you see in that
| kind of context.
|
| Sure, all I'm trying to say is that "the class of bugs
| described here" covers more than what was listed in the
| parentheses.
| rjzzleep wrote:
| I'd argue, that while null ref and those classes of bugs
| may decrease, logic errors will increase. Rust is not an
| extraordinary readable language in my opinion, especially
| in the kernel where the kernel has its own data
| structures. IMHO Apple did it right in their kernel
| stack, they have a restricted subset of C++ that you can
| write drivers with.
|
| Which is also why in my opinion Zig is much more
| suitable, because it actually addresses the readability
| aspect without bring huge complexity with it.
| oguz-ismail2 wrote:
| > Zig is much more suitable, because it actually
| addresses the readability aspect
|
| How? It doesn't look very different from Rust. In terms
| of readability Swift does stand out among LLVM frontends,
| don't know if it is or can be used for systems
| programming though.
| Someone wrote:
| Apple claims Swift can be used for systems programming,
| and is (partly) eating its own dogfood by using it in
| FoundationDB
| (https://news.ycombinator.com/item?id=38444876) and by
| providing examples of embedded projects
| (https://www.swift.org/get-started/embedded/)
|
| I think they are right in that claim, but in making it
| so, at least some of the code loses some of the
| readability of Swift. For truly low-level code, you'll
| want to give up on classes, may not want to have copy-on-
| write collections, and may need to add quite a few some
| annotations.
| galangalalgol wrote:
| Swift is very slow relative to rust or c though. You can
| also cause seg faults in swift with a few lines. I Don't
| find any of these languages particularly difficult to
| read, so I'm not sure why this is listed as a
| discriminator between them.
| saagarjha wrote:
| But those segfaults will either be memory memory safe or
| your lines will contain "unsafe" or "unchecked"
| somewhere.
| rowanG077 wrote:
| I would expect the opposite. C requires you to deal with
| extreme design complexity in large systems because the
| language offers nothing to help.
| aw1621107 wrote:
| > I'd argue, that while null ref and those classes of
| bugs may decrease, logic errors will increase.
|
| To some extent that argument only makes sense; if you can
| find a way to greatly reduce the incidence of non-logic
| bugs while not addressing other bugs then of course logic
| bugs would make up a greater proportion of what remains.
|
| I think it's also worth considering the fact that while
| Rust doesn't guarantee that it'll catch all logic bugs,
| it (like other languages with more "advanced" type
| systems) gives you tools to construct systems that can
| catch certain kinds of logic bugs. For example, you can
| write lock types in a way that guarantees at compile time
| that you'll take locks in the correct order, avoiding
| deadlocks [0]. Another example is the typestate pattern
| [1], which can encode state machine transitions in the
| type system to ensure that invalid transitions and/or
| operations on invalid states are caught at compile time.
|
| These, in turn, can lead to higher-order benefits as
| offloading some checks to the compiler means you can
| devote more attention to things the compiler can't check
| (though to be fair this does seem to be more variable
| among different programmers).
|
| > Rust is not an extraordinary readable language in my
| opinion, especially in the kernel where the kernel has
| its own data structures.
|
| The above notwithstanding, I'd imagine it's possible to
| think up scenarios where Rust would make some logic bugs
| more visible and others less so; only time will tell
| which prevails in the Linux kernel, though based on what
| we know now I don't think there's strong support for the
| notion that logic bugs in Rust are a substantially more
| common than they have been in C, let alone because of
| readability issues.
|
| Of course there's the fact that readability is very much
| a personal thing and is a multidimensional metric to boot
| (e.g., a property that makes code readable in one context
| may simultaneously make code less readable in another). I
| don't think there would be a universal answer here.
|
| [0]: https://lwn.net/Articles/995814/
|
| [1]: https://cliffle.com/blog/rust-typestate/
| bcrosby95 wrote:
| I would argue logic errors would decrease because you
| aren't spending as much time worrying about and fixing
| null ref and other errors.
| Tarucho wrote:
| can you prove that?
| viraptor wrote:
| Maybe increase as a ratio, but not absolute. There are
| various benefits of Rust that affect other classes of
| issues: fancy enums, better errors, ability to control
| overflow behaviour and others. But for actual experience,
| check out what the kernel code developer has to say:
| https://xcancel.com/linaasahi/status/1577667445719912450
| staticassertion wrote:
| Rust is a lot more explicit. I suspect logic bugs will be
| much less common. It's far easier to model complexity in
| Rust.
| jiggawatts wrote:
| The default Mutex struct in Rust makes it _impossible_ to
| modify the data it protects without holding the lock.
|
| _" Each mutex has a type parameter which represents the
| data that it is protecting. The data can only be accessed
| through the RAII guards returned from lock and try_lock,
| which guarantees that the data is only ever accessed when
| the mutex is locked."_
|
| Even if used with more complex operations, the RAII
| approach means that the example you provided is much less
| likely to happen.
| materielle wrote:
| I don't think that the parent comment is saying _all_ of
| the bugs would have been prevented by using Rust.
|
| But in the listed categories, I'm equally skeptical that
| _none_ of them would have benefited from Rust even a bit.
| johncolanduoni wrote:
| That's not my point - just that "state machine races" is
| a too-broad category to say much about how Rust would or
| wouldn't help.
| apaprocki wrote:
| > Furthermore, the 19-year-old bug case study is a
| refcounting error
|
| It always surprised me how the top-of-the line analyzers,
| whether commercial or OSS, never really implemented C-style
| reference count checking. Maybe someone out there has written
| something that works well, but I haven't seen it.
| RealityVoid wrote:
| Why doesn't it surprise me that the CAN bus driver bugs have
| the longest average lifetime?
| johncolanduoni wrote:
| This is I think an under-appreciated aspect, both for
| detractors and boosters. I take a lot more "risks" with Rust,
| in terms of not thinking deeply about "normal" memory safety
| and prioritizing structuring my code to make the logic more
| obviously correct. In C++, modeling things so that the memory
| safety is super-straightforward is paramount - you'll almost
| never see me store a std::string_view anywhere for example. In
| Rust I just put &str wherever I please, if I make a mistake
| I'll know when I compile.
| the8472 wrote:
| The concurrent state machine example looks like a locking
| error? If the assumption is that it shouldn't change in the
| meantime, doesn't it mean the lock should continue to be held?
| In that case rust locks can help, because they can embed the
| data, which means you can't even touch it if it's not held.
| paulddraper wrote:
| Rust would prevent a number of bugs, as it can model state
| machine guarantees as well.
|
| Rewriting it all in Rust is extremely expensive, so it won't be
| done (soon).
| wiz21c wrote:
| Expensive because of: 1/ a re-write is never easy 2/ rust is
| specifically tough (because it catches error and forces you
| to think about it for real, because it makes some contruct
| (linked list) really hard to implement) for kernel/close to
| kernel code ?
| IshKebab wrote:
| Both I'd say. Rust imposes more constraints on the
| structure of code than most languages. The borrow checker
| really likes ownership _trees_ whereas most languages allow
| any ownership graph no matter how spaghetti it is.
|
| As far as I know that's why Microsoft rewrote Typescript in
| Go instead of Rust.
| IshKebab wrote:
| Rust has other features that help prevent logic errors. It's
| not _just_ C plus a borrow checker.
| keybored wrote:
| No other top-level comments have since mentioned Rust[1] and
| TFA mentions neither Rust nor topics like memory safety. It's
| just plain bugs.
|
| The Rust phantom zealotry is unfortunately real.
|
| [1] Aha, but the chilling effect of dismissing RIR comments
| before they are even posted...
| staticassertion wrote:
| Yes, I saw this last night and was confused because only one
| comment mentioned Rust, and it was deleted I think. I nearly
| replied "you're about to prompt 1,000 rust replies with this"
| and here's what I woke up to lol
| kubb wrote:
| It's hilarious that you feel the need to preemptively take
| control of the narrative in anticipation of the Rust people
| that you fear so much.
|
| Is this an irrational fear, I wonder? Reminds me of methods
| used in the political discourse.
| irishcoffee wrote:
| > It's hilarious that you feel the need to preemptively take
| control of the narrative in anticipation of the Rust people
| that you fear so much.
|
| > Is this an irrational fear, I wonder? Reminds me of methods
| used in the political discourse.
|
| In a sad sort of way, I think its hilarious that hn users
| have been so completely conditioned to expect rust evangelism
| any time a topic like this comes up that they wanted to get
| ahead of it.
|
| Not sure who it says more about, but it sure does say a whole
| lot.
| kubb wrote:
| I don't think evangelism is necessary anymore. Rust
| adoption is now a matter of time.
| Bridged7756 wrote:
| People who make that kind of remarks should be called out and
| shunned. The Rust community is tired of discrimination and
| being the butt of jokes. All the other inferior languages
| prey on its minority status, despite Rust being able to solve
| all their problems. I take offense to these remarks, I don't
| want my kids to grow up as Rustaceans in such a caustic
| society.
| ramon156 wrote:
| You're fighting air
| mgaunard wrote:
| I don't think 70% of bugs are memory safety issues.
|
| In my experience it's closer to 5%.
| redeeman wrote:
| probably quite a bit less than 5%, however, they tend to be
| quite serious when they happen
| mgaunard wrote:
| Only serious if you care about protecting from malicious
| actors running code on the same host.
| redeeman wrote:
| you dont? I would imagine people that runs for example a
| browser would have quite an interest in that
| cogman10 wrote:
| I believe this is where that fact comes from [1]
|
| Basically, 70% of high severity bugs are memory safety.
|
| [1] https://www.chromium.org/Home/chromium-security/memory-
| safet...
| mgaunard wrote:
| Right, which is a measure which is heavily biased towards
| memory safety bugs.
| saagarjha wrote:
| High severity _security_ issues.
| tester756 wrote:
| That's the figure that Microsoft and Google found in their
| code bases.
| IshKebab wrote:
| 70% of _security vulnerabilities_ are due to memory safety.
| Not all bugs.
| stonemetal12 wrote:
| Using the data provided, memory safety issues (use-after-
| free, memory-leak, buffer-overflow, null-deref) account for
| 67% of their bugs. If we include refcount It is just over
| 80%.
| pjc50 wrote:
| > race condition in unsafe logic that interacts with DMA
|
| It's worth noting that if you write memory safe code but mis-
| program a DMA transfer, or trigger a bug in a PCIe device, it's
| possible for the hardware to give you memory-safety problems by
| splatting invalid data over a region that's supposed to contain
| something else.
| lynx97 wrote:
| Thanks for raising this. It feels like evangelists paint a
| picture of Rust basically being magic which squashes all bugs.
| My personal experience is rather different. When I gave Rust a
| whirl a few years ago, I happened to play with mio for some
| reason I can't remember yet. Had some basic PoC code which
| didn't work as expected. So while not being a Rust expert, I am
| still too much fan of the scratch your own itch philosophy, so
| I started to read the mio source code. And after 5 minutes, I
| found the logic bug. Submitted a PR and moved on. But what
| stayed with me was this insight that if someone like _me_ can
| casually find and fix a Rust library bug, propaganda is
| probably doing more work then expected. The Rust craze feels a
| bit like Java. Just because a language baby-sits the developer
| doesn 't automatically mean better quality. At the end of the
| day, the dev needs to juggle the development process. Sure,
| tools are useful, but overstating safety is likely a route
| better avoided.
| john01dav wrote:
| Rust has more features than just the borrow checker. For
| example, it has a a more featured type system than C or C++,
| which a good developer can use to detect some logic mistakes at
| compile time. This doesn't eliminate bugs, but it can catch
| some very early.
| wordisside wrote:
| But unsafe Rust, which is generally more often used in low-
| level code, is more difficult than C and C++.
|
| https://rust-unofficial.github.io/too-many-lists/
|
| https://chadaustin.me/2024/10/intrusive-linked-list-in-rust/
| aw1621107 wrote:
| > But unsafe Rust, which is generally more often used in
| low-level code, is more difficult than C and C++.
|
| I think "is" is a bit too strong. "Can be", sure, but I'm
| rather skeptical that _all_ uses of unsafe Rust will be
| more difficult than writing equivalent C /C++ code.
| marcosdumay wrote:
| Eh... Removing concurrence bugs is one of the main selling
| points for Rust. And algebraic types are a really boost for
| situations where you have lots of assumptions.
| silver_sun wrote:
| Their section on "Dataset limitations" says that the study "Only
| captures bugs with Fixes: tags (~28% of fix commits)."
|
| Just worth noting that it is a significant extrapolation from
| only "28%" of fix commits to assume that the average is 2 years.
| tremon wrote:
| Why? A sample size of 28% is positively _huge_ compared to what
| most statistical studies have to work with. The accuracy of an
| extrapolation is mostly determined by underlying sampling bias,
| not the amount of data. If you have any basis to suggest that
| capturing "only bugs with fixes tags" creates a skewed sample,
| that would be grounds to distrust the extrapolation, but simply
| claiming "it's only 28%" does not make it worth noting.
| dpc_01234 wrote:
| Might be obviously, but there is definitely a lot of biases in
| the data here. It's unavoidable. E.g. many bugs will not be
| detected, but they will be removed when the code is rewritten. So
| code that is refactored more often will have lower age of fixed
| bugs. Components/subsystems that are heavily used will detect
| bugs faster. Some subsystems by their very nature can tolerate
| bugs more, while some by necessity will need to be more correct
| (like bpf).
| a3w wrote:
| The kernel this speaks of is probably linux. Does windows have
| a similar round time?
| pixl97 wrote:
| I mean, yea.
|
| Here is a device driver bug that was around 11 years.
|
| https://www.bitdefender.com/en-
| us/blog/hotforsecurity/google...
| ValdikSS wrote:
| grsecurity project has fixed many security bugs but did not
| contribute back, as they're profiting from selling the patchset.
|
| It's not uncommon for the bugs they found to be rediscovered 6-7
| years later.
|
| https://xcancel.com/spendergrsec
| __bjoernd wrote:
| > as they're profiting from selling the patchset
|
| Profiting from selling their patchset is not the whole story,
| though. grsec was public and free for a long time and there
| were many effects at play preventing the kernel from adopting
| it.
| woliveirajr wrote:
| But the patchset should use the same license as the original
| code, shouldn't?
| ValdikSS wrote:
| It is: https://grsecurity.net/faq
| staticassertion wrote:
| This implies (or states, hard to say) that they don't upstream
| specifically in order to profit. That is nonsense.
|
| 1. Tons of bugs are reported upstream by grsecurity
| historically.
|
| 2. _Tons_ of critical security mitigations in the kernel were
| outright invented by that team. ASLR, SMAP, SMEP, NX, etc.
|
| 3. They were completely FOSS until very recently.
|
| 4. They have always maintained that they are entirely willing
| to upstream patches but that it's a lot of work and would
| require funding. Upstream has always been extremely hostile
| towards attempts to take small pieces of Grsecurity and
| upstream them.
| zkmon wrote:
| A bug is a piece of code that doesn't agree with requirements or
| architecture. The misalignment can not be attributed to code
| alone.
| jackfranklyn wrote:
| The state machine race pattern resonates beyond kernel work. I've
| seen similar bugs hide for years in application code -
| transaction state edge cases that only trigger under specific
| sequences of user actions that nobody tests for.
|
| The median lifetimes are fascinating. Race conditions at 5.1
| years vs null-deref at 2.2 years makes intuitive sense - the
| former needs specific timing to manifest, while the latter will
| crash obviously once you hit the code path. The ones that need
| rare conditions to trigger are the ones that survive longest.
| MarleTangible wrote:
| One of the iOS 26 Core Audio bug (CVE-2025-31200) is about
| synchronizing two different arrays with each other and the
| assumption mistakes that were made trusting dimensional
| information which could be coming from the user.
|
| https://youtu.be/nTO3TRBW00E
| pixl97 wrote:
| >hide for years in application code
|
| Yea, it's pretty common. We had a customer years ago that was
| having a rare and random application crash under load. Never
| could figure out where it was from. Quite some time later a
| batch load interface was added to the app and with the rate
| things were input with it the crash could be triggered
| reliably.
|
| It's something else that's added/changed in the application
| that eventually makes the bug stand out.
| sureglymop wrote:
| Only tangentially related but maybe someone here can help me.
|
| I have a server which has many peripherals and multiple GPUs.
| Now, I can use vfio and vfio-pcio to memory map and access their
| registers in user space. My question is, how could I start with
| kernel driver development? And I specifically mean the dev setup.
|
| Would it be a good idea to use vfio with or without a vm to write
| and test drivers? How to best debug, reload and test changing
| some code of an existing driver?
| MORPHOICES wrote:
| Deep bugs, particularly in kernels, can go unnoticed for years,
| according to analyses I keep seeing. Decades at times. ~
|
| That seems frightening at first. However, the more I consider it,
| the more it seems... predictable.
|
| The mental model that I find useful:
|
| Users discover surface bugs.
|
| Deep bugs only appear in infrequent combinations.
|
| For some bugs to show up, new context is required.
|
| I've observed a few patterns:
|
| Undefined behavior-related bugs are permanently hidden.
|
| Logic errors are less important than uncommon hardware or timing
| conditions.
|
| Long before they can be exploited, security flaws frequently
| exist.
|
| I'm curious what other people think of this:
|
| Do persistent bugs indicate stability or failure?
|
| What typically leads to their discovery?
|
| To what extent do you trust "well-tested" code?
| fsflover wrote:
| > To what extent do you trust "well-tested" code?
|
| I don't, which is why I use Qubes OS providing security through
| compartmentalization.
| hun3 wrote:
| Then the question becomes: to what extent do you trust Xen
| and Qubes RPC?
| fsflover wrote:
| I do have to somewhat trust Xen, but Qubes' isolation
| relies on hardware virtualization (VT-d), which
| statistically has much less security issues than Xen
| itself. Most Xen advisories do not affect Qubes:
| https://www.qubes-os.org/security/xsa/
| saagarjha wrote:
| > Undefined behavior-related bugs are permanently hidden.
|
| No they are often found and fixed.
| burnt-resistor wrote:
| Speaking of nasty kernel bugs although on another platform,
| there's a nasty one in either Microsoft's Win 11 nwifi.sys
| handling of deadlock conditions or Qualcomm's QCNCM865
| FastConnect 7800 WCN785x driver that panics because of a watchdog
| failure in nwifi!MP6SendNBLInternal+0x4b guarded by a deadlocked
| ndis!NdisAcquireRWLockRead+0x8b. It "BSODs" the system rather
| than doing something sane like dropping a packet or
| retransmitting.
|
| Am I the only unreasonable maniac who wants a very long-term
| stable, seL4-like capability-based, ubiquitous, formally-verified
| mkernel that rarely/never crashes completely* because drivers are
| just partially-elevated programs sprinkled with transaction
| guards and rollback code for critical multiple resource access
| coordination patterns? (I miss hacking on MINIX 2.)
|
| * And never need to reboot or interrupt server/user desktop
| activities because the core mkernel basically never changes since
| it's tiny and proven correct.
| giamma wrote:
| Is the intention of the author to use the number of years bugs
| stay "hidden" as a metric of the quality of the kernel codebase
| or of the performance of the maintainers? I am asking because at
| some point the articles says "We're getting faster".
|
| IMHO a fact that a bug hides for years can also be indication
| that such bug had low severity/low priority and therefore that
| the overall quality is very good. Unless the time represents how
| long it takes to reproduce and resolve a known bug, but in such
| case I would not say that "bug hides" in the kernel.
| staticassertion wrote:
| > IMHO a fact that a bug hides for years can also be indication
| that such bug had low severity/low priority and therefore that
| the overall quality is very good.
|
| It doesn't seem to indicate that. It indicates the bug just
| isn't in tested code or isn't reached often. It could still be
| a very severe bug.
|
| The issue with longer lived bugs is that someone could have
| been leveraging it for longer.
| galangalalgol wrote:
| Worst case is that it doesn't even cause correctness issues
| in normal use, only when misused in a way that is unlikely to
| happen unintentionally.
| staticassertion wrote:
| I guess because I work in security the "unintentionally"
| doesn't matter much to me.
| SAI_Peregrinus wrote:
| But it matters for detection time, because there's a lot
| more "normal" use of any given piece of code than
| intentional attempts to break it. If a bug can't be
| triggered unintentionally it'll never get detected
| through normal use, which can lead to it staying hidden
| for longer.
| staticassertion wrote:
| That's not really contested? The statement was that
| longer detection time indicates lower severity.
| cogman10 wrote:
| > IMHO a fact that a bug hides for years can also be indication
| that such bug had low severity/low priority
|
| Not really true. A lot of very severe bugs have lurked for
| years and even decades. Heartbleed comes to mind.
|
| The reason these bugs often lurk for so long is because they
| very often don't cause a panic, which is why they can be really
| tricky to find.
|
| For example, use after free bugs are really dangerous. However,
| in most code, it's a pretty safe bet that nothing dangerous
| happens when use after free is triggered. Especially if the
| pointer is used shortly after the free and dies shortly after
| it. In many cases, the erroneous read or write doesn't break
| something.
|
| The same is true of the race condition problems (which are some
| of the longest lived bugs). In a lot of cases, you won't know
| you have a race condition because in many cases the contention
| on the lock is low so the race isn't exposed. And even when it
| is, it can be very tricky to reproduce as the race isn't likely
| to be done the same way twice.
| turtletontine wrote:
| > ...lurked for years and even decades. Heartbleed comes to
| mind.
|
| I don't know much about Heartbleed, but Wikipedia says:
|
| > Heartbleed is a security bug... It was introduced into the
| software in 2012 and publicly disclosed in April 2014.
|
| Two years doesn't sound like "years or even decades" to me?
| But again, I don't know much about Heartbleed so I may be
| missing something. It does say it was also patched in 2014,
| not just discovered then.
| cogman10 wrote:
| This may just be me misremembering, but as I recall, the
| bug of Heartbleed was ultimately a very complex macro
| system which supported multiple very old architectures. The
| bug, IIRC, was the interaction between that old macro
| system and the new code which is what made it hard to
| recognize as a bug.
|
| Part of the resolution to the problem was I believe they
| ended up removing a fair number of unsupported platforms.
| It also ended up spawning alternatives to openssl like
| boring ssl which tried to remove as much as possible to
| guard against this very bug.
| mrguyorama wrote:
| Maybe you are thinking of ShellShock
|
| https://en.wikipedia.org/wiki/Shellshock_(software_bug)
|
| The bug was introduced into the code in 1989, and only
| found and exploited in 2014.
| gjfr wrote:
| Interesting! We did a similar analysis on Content Security Policy
| bugs in Chrome and Firefox some time ago, where the average bug-
| to-report time was around 3 years and 1 year, respectively.
| https://www.usenix.org/conference/usenixsecurity23/presentat...
|
| Our bug dataset was way smaller, though, as we had to pinpoint
| all bug introductions unfortunately. It's nice to see the Linux
| project uses proper "Fixes: " tags.
| staticassertion wrote:
| > It's nice to see the Linux project uses proper "Fixes: "
| tags.
|
| Sort of. They often don't.
| jmyeet wrote:
| The lesson here is that people have an unrealistic view of how
| complex it is to write correct and safe multithreaded code on
| multi-core, multi-thread, assymmetric core, out-of-order
| processors. This is no shade to kernel developers. Rather, I
| direct this at people who seem to you can just create a thread
| pool in C++ and solve all your concurrency problems.
|
| One criticism of Rust (and, no, I'm not saying "rewrite it in
| Rust", to be clear) is that the borrow checker can be hard to use
| whereas many C++ engineers (in particular, for some reason) seem
| to argue that it's easier to write in C++. I have two things to
| say about that:
|
| 1. It's not easier in C++. Nothing is. C++ simply allows you to
| make mistakes without telling you. GEtting things correct in C++
| is just as difficult as any other language if not more so due to
| the language complexity; and
|
| 2. The Rust borrow checker isn't hard or difficult to use. What
| you're doing is hard and difficult to do correctly.
|
| This is I favor cooperative multitasking and using battle-tested
| concurrency abstractions whenever possible. For example the
| cooperative async-await of Hack and the model of a single thread
| responding to a request then discarding everything in PHP/Hack is
| virtually ideal (IMHO) for serving Web traffic.
|
| I remember reading about Google's work on various C++ tooling
| including valgrind and that they exposed concurrency bugs in
| their own code that had lain dormant for up to a decade. That's
| Google with thousands of engineers and some very talented
| engineers at that.
| wordisside wrote:
| > It's not easier in C++. Nothing is.
|
| The implementations of sort in Rust are filled with unsafe.[0]
|
| Another example is that of doubly linked lists.[1] It is
| possible to implement a doubly linked list correctly in C++
| without much trouble. In Rust, it can be significantly more
| challenging.
|
| In C++, pointers are allowed to alias if their types, roughly
| said, are compatible. In Rust, there are stricter rules, and
| getting those rules wrong in an unsafe block, or code outside
| unsafe blocks that code inside unsafe blocks rely on, will
| result in breakage of memory safety.
|
| This has been discussed by others.[2]
|
| Based on that, do you agree that there are algorithms and data
| structures that are significantly easier to implement
| efficiently and correctly in C++ than in Rust? And thus that
| you are being completely wrong in your claim?
|
| [0] https://github.com/rust-
| lang/rust/blob/main/library/core/src...
|
| [1] https://rust-unofficial.github.io/too-many-lists/
|
| [2] https://chadaustin.me/2024/10/intrusive-linked-list-in-
| rust/
| aw1621107 wrote:
| > The implementations of sort in Rust are filled with unsafe.
|
| Strictly speaking, the mere presence of `unsafe` says nothing
| on its own about whether "it" is easier in C++. Not only does
| `unsafe` on its own say nothing about the "difficulty" of the
| code it contains, but that is just one factor of one side of
| a comparison - very much insufficient for a complete
| conclusion.
|
| Furthermore, "just" writing a sorting algorithm is pretty
| straightforwards both in Rust and C++; it's the more
| interesting properties that tend to make for equally
| interesting implementations, and one would need to procure
| Rust and C++ implementations with equivalent properties,
| preferably from the same author(s), for a proper comparison.
|
| Past research has shown that Rust's current sorting
| algorithms have different properties than C++ implementations
| from the time (e.g., the "X safety" results in [0]), so if
| nothing substantial has changed since then there's going to
| be some work to do for a proper comparison.
|
| Edit: forgot to add the reference [0]:
| https://github.com/Voultapher/sort-research-
| rs/blob/main/wri...
| wordisside wrote:
| Also, the Linux kernel developers turned off strict aliasing in
| the C compilers they use, because they found strict aliasing
| too difficult. Yet unsafe Rust has more difficult rules than
| even strict aliasing in C and C++.
| aw1621107 wrote:
| > Also, the Linux kernel developers turned off strict
| aliasing in the C compilers they use, because they found
| strict aliasing too difficult.
|
| I'm not sure "they found strict aliasing too difficult" is an
| entirely correct characterization? From this rather
| (in)famous email from Linus [0]: The fact
| is, using a union to do type punning is the traditional AND
| STANDARD way to do type punning in gcc. In fact, it is the
| *documented* way to do it for gcc, when you are a f*cking
| moron and use "-fstrict-aliasing" and need to undo
| the braindamage that that piece of garbage C standard
| imposes. [snip] This is why we use
| -fwrapv, -fno-strict-aliasing etc. The standard
| simply is not *important*, when it is in direct conflict with
| reality and reliable code generation.
| The *fact* is that gcc documents type punning through unions
| as the "right way". You may disagree with that, but
| putting some theoretical standards language over the
| *explicit* and long-time documentation of the main
| compiler we use is pure and utter bullshit.
|
| [0]: https://lkml.org/lkml/2018/6/5/769
| marcosdumay wrote:
| > The Rust borrow checker isn't hard or difficult to use. What
| you're doing is hard and difficult to do correctly.
|
| There are entire classes of structures that no, aren't hard to
| do properly, but the borrow checker makes artificially hard due
| to design limitations that are known to be sub-optimal.
|
| No, two-directional linked lists and partially editable data
| structures aren't inherently hard. It's a Rust limitation that
| a piece of code can't take enough ownership of them to edit
| they safely.
| ryukoposting wrote:
| This is fascinating stuff, especially the per-subsystem data.
| I've worked with CAN in several different professional and
| amateur settings, I'm not surprised to see it near the bottom of
| this list. That's not a dig against the kernel or the folks who
| work on it... more of a heavy sigh about the state of the
| industries that use CAN.
|
| On a related note, I'm seeing a correlation between "level of
| hoopla" and a "level of attention/maintenance." While it's hard
| to distinguish that correlation from "level of use," the fact
| that CAN is so far down the list suggests to me that hoopla
| matters; it's everywhere but nobody talks about it. If a kernel
| bug takes down someone's datacenter, _boy_ are we gonna hear
| about it. But if a kernel bug makes a DeviceNet widget freak out
| in a factory somewhere? Probably not going to make the front page
| of HN, let alone CNN.
| pixl97 wrote:
| There is a general rule on bugs is that the more devices they
| are on, the more apt they are to trigger.
|
| A CAN with 10,000 machines total and relatively fixed
| applications is either going to trigger the bug right off the
| bat and then work around it, or trigger the bug so rarely it
| won't be recognized as a kernel issue.
|
| General purpose systems running millions and millions of units
| with different workloads are an evolutionary breeding ground
| for finding bugs and exploits.
| eab- wrote:
| I'd find this article a bit more compelling if it was used to
| find current introduced bugs, instead of just using a holdout set
| michaelcampbell wrote:
| Thank goodness for reader mode. The transparent background where
| the text is with the wiggly line background is... challenging.
| GaryBluto wrote:
| What's with the odd scribbles in the background?
| kmavm wrote:
| It's an easter egg on the website that usually goes unnoticed.
| It's our first time on the front page of HN, so it's a little
| overutilized right now. Capital-C clears it.
| redleader55 wrote:
| I don't think the problem is the kernel. Kernel bugs stay hidden
| because no one runs recent Kernels.
|
| My Pixel 8 runs kernel a stable minor from 6.1, which was
| released more than 4 years ago. Yes, fixes get backported to it,
| but the new features in 6.2->6.19 stay unused on that hardware.
| All the major distros suffer from the same problem, most people
| are not running them in production
|
| Most hyperscalers are running old kernel versions on which they
| do backports. If you go Linux conferences you hear folks from big
| companies mentioning 4.xx, 3.xx kernels, in 2025.
| calebm wrote:
| It's interesting to consider that the same phenomenon may also
| hold true for humanity's psychological software.
___________________________________________________________________
(page generated 2026-01-08 23:01 UTC)