[HN Gopher] Six Great Features with the Upcoming Linux 6.6 Kernel
       ___________________________________________________________________
        
       Six Great Features with the Upcoming Linux 6.6 Kernel
        
       Author : danielklnstein
       Score  : 45 points
       Date   : 2023-10-28 13:42 UTC (9 hours ago)
        
 (HTM) web link (www.phoronix.com)
 (TXT) w3m dump (www.phoronix.com)
        
       | ahmedfromtunis wrote:
       | Off topic, but I had this question for years now: why does the
       | Unix kernel (try to) contain the drivers to every and each
       | device?
       | 
       | Wouldn't it be cleaner, easier and simpler to keep those separate
       | and add them later as "modules"?
       | 
       | Isn't this how we're supposed to write other types of software?
       | Why is this case different?
        
         | woodruffw wrote:
         | > Wouldn't it be cleaner, easier and simpler to keep those
         | separate and add them later as "modules"?
         | 
         | That is exactly what Linux does. Part of building a Linux
         | distribution is determining which modules to ship by default to
         | your users. To the best of my knowledge, the overwhelming
         | majority of drivers are not installed by default, and the
         | average user will never need them.
        
           | MaxBarraclough wrote:
           | I think that by 'module' they were thinking of user-space
           | drivers rather than kernel modules.
        
             | IshKebab wrote:
             | No, I think they were imagining kernel drivers but
             | distributed separately and interacting with the kernel
             | through a stable ABI.
        
         | chownie wrote:
         | There's upsides and downsides to both approaches. I think that
         | Linux kernel maintainers continue this approach because this
         | forces driver maintainers to meet a minimum level of effort to
         | get their work merged.
        
           | joecool1029 wrote:
           | Theoretically the monolithic approach is supposed to be more
           | performant. Abstracting away everything into modules can be
           | more stable, but also usually incurs some overhead. But this
           | is holy war territory and I'm trying to ELI5 best as I can.
        
           | rwmj wrote:
           | One serious downside (for end users) of having a stable
           | driver ABI would be that manufacturers would release a binary
           | blob driver for their hardware which just about works, but no
           | bug could ever be fixed. Manufacturers wouldn't have any
           | incentive to fix bugs and kernel developers couldn't do it
           | either.
           | 
           | The situation of GPL'd drivers in a kernel with an unstable
           | ABI wasn't really planned, but for users it's probably the
           | best outcome.
        
             | nuancebydefault wrote:
             | I always thought that ABI was the interface between CPU and
             | executable code, ie the spec of the assembly language for a
             | CPU architecture: aarch64, x86 etc. I now would guess it is
             | also a term used for the interface between libraries
             | (kernel/driver) as long as they are binary (without the
             | source code or headers) ?
        
               | gdprrrr wrote:
               | It's also used for programming languages like Rust and
               | Haskell. The Rust compiler not having a stable ABI means
               | the way the compiler stores information about function
               | signatures, debug symbols etc in libraries may change
               | between compiler versions, so you have to recompile all
               | libraries.
        
         | joecool1029 wrote:
         | You're literally describing something like macOS/iOS which is
         | based around a microkernel (XNU, based on Mach). See also QNX
         | for a non-unix OS that does this.
        
           | speed_spread wrote:
           | Err.. loading drivers dynamically according to a stable ABI
           | doesn't make an OS a microkernel. If the drivers all share
           | the same address space that's a pretty conventional OS. For a
           | microkernel you need to have separate processes for each
           | driver. It's more robust but there's more IPC overhead.
        
         | danielklnstein wrote:
         | As another comment explains - the vast majority of drivers are
         | not compiled as part of a standard Linux distribution.
         | 
         | If you're asking why so many drivers are included in Linux's
         | mainline repository and not maintained off-tree - the major
         | advantage of being upstreamed is that changes to the kernel
         | have to take your driver into account as well. e.g. anyone who
         | changes any APIs will have to update your driver as well.
         | 
         | Moreover, being upstreamed certifies that your driver has gone
         | through a non-trivial review process to be merged into the
         | mainline and is an indication of quality/stability.
        
         | adrian_b wrote:
         | Because the Linux kernel does not have a stable API for
         | modules.
         | 
         | The modules included in the kernel are updated whenever the API
         | changes.
         | 
         | For the external modules, every time when a new kernel version
         | is released, they may become broken and there is a lot of work
         | to identify how to update the modules, unless one watches the
         | kernel mail lists every day, because there is also no
         | documentation about how to migrate the old modules to the new
         | kernel.
         | 
         | The reasons for module breakage may be just the movement of
         | some definitions from one header to another, which break
         | compiling, but most frequently some structures gain new members
         | or lose old members, or some functions gain new arguments or
         | lose old arguments.
         | 
         | When there are new members or arguments it is hard to discover
         | what values should be put in them, while when members or
         | arguments are deleted it is hard to discover whether their
         | absence must be compensated somehow, e.g. by inserting
         | invocations to other functions.
         | 
         | In the worst case everything can be solved by reading the
         | kernel source, but that takes a lot of time and those who
         | maintain out-of-kernel modules usually do not do this as a
         | full-time job, so they do not have time to scan every day the
         | kernel mail lists, to see if anyone has plans to make changes
         | that will break their modules.
        
           | seanwilson wrote:
           | How does this manage to scale? With even a modest number of
           | drivers, you'd think the maintenance burden would be
           | unreasonably high in terms of needing to have someone
           | available who understands how a set of drivers work, knows
           | how to test them, and keeps them up-to-date.
        
             | adrian_b wrote:
             | That is why everybody attempts to push their modules into
             | the kernel source.
             | 
             | Those who cannot do that because their modules are not open
             | source, like NVIDIA, have a lot of work to do for module
             | maintenance.
             | 
             | Nevertheless, there are also open-source modules that
             | cannot be pushed into the kernel because they are for
             | uncommon hardware with few users, and those also require a
             | lot of work for maintenance, so they are frequently
             | abandoned and no longer brought up-to-date, to be
             | compatible with the latest kernels.
        
               | seanwilson wrote:
               | > That is why everybody attempts to push their modules
               | into the kernel source.
               | 
               | > Those who cannot do that because their modules are not
               | open source, like NVIDIA, have a lot of work to do for
               | module maintenance.
               | 
               | Say someone gets a driver for a WiFi USB stick into the
               | kernel source and leaves it for others to maintain.
               | Wouldn't the maintainers need the hardware device for
               | testing changes and an understanding of the hardware's
               | quirks to keep the driver updated? Obviously the system
               | works but I don't get how it doesn't suffer from software
               | rot, especially when physical hardware is required for
               | robust testing including testing the hardware against
               | different motherboards.
        
               | toast0 wrote:
               | What I've seen is (more or less) the mainline kernel
               | maintainers for whichever kernel have the objective of
               | making sure the drivers compile.
               | 
               | They can't test what they don't have, and I haven't seen
               | evidence than any open source kernel has a testinfg time
               | with large fleet of hardware for tests. There's more or
               | less an incidental distributed test fleet as users have
               | various testing setups, etc; but driver mistakes in
               | obscure hardware tends to get found late.
               | 
               | It's not unusual for drivers to be found to have been
               | broken for years, and then unless the reporter can fix
               | it, or the fix is obvious from the report, the driver
               | gets removed.
               | 
               | Another path towards driver removal is some interfaces
               | get updated in multiple steps: a new interface is added,
               | drivers are updated to the new interface, the old
               | interface and any drivers that haven't been updated are
               | removed. If nobody has stepped up to update the driver
               | before removal, that's an indication of disuse; sometimes
               | they get added back after removal though.
               | 
               | For the record, Windows isn't necessarily better. I had a
               | printer where there was a working driver, but Windows
               | would prefer the non-working driver from Windows Update,
               | and it was impossible to get that fixed.
        
               | Veserv wrote:
               | You are correct. The answer is that it does suffer from
               | software rot.
        
             | rwmj wrote:
             | Coccinelle semantic patching is pretty cool for making
             | systematic changes across the kernel (eg. adding a
             | parameter to the same function everywhere).
             | 
             | https://coccinelle.gitlabpages.inria.fr/website/
        
             | aseipp wrote:
             | > How does this manage to scale?
             | 
             | "Okay" to "very poorly", depending on your view. And for
             | the record, many times the drivers go untested, frankly.
             | I've absolutely found completely broken drivers in the
             | kernel (as in just obvious lock imbalances where unlock()
             | can't possibly ever get called, null pointer derefs; simple
             | things you can spot in a code review) in the kernel where
             | the break was introduced as some part of a refactoring and
             | the person doing the work made a mistake. Because they also
             | refactored 30 other drivers in the same patch series to
             | match an API change. (The filesystem developers have said
             | many times in patch reviews that certain things are massive
             | chores because they have to go fix 40+ filesystem drivers
             | every time some API break happens.)
             | 
             | Linux actually has tons of hardware _regressions_ because
             | of this whole design choice. When you totally rewrite a
             | piece of code that is shared among multiple components,
             | that can be OK, as long as you preserve the external
             | behavior that the previous interface exposed. An easy way
             | to do that is to establish a contract between the
             | implementation and call sites, which is often reflected as
             | a stable API. The stable API makes some contracts explicit,
             | by construction. But many times API refactorings will
             | happen at the same time, and _that_ in practice typically
             | introduces new behaviors into the downstream code that
             | previously didn 't exist before. These new behaviors, when
             | introduced into an existing driver that was not developed
             | for them, tends to cause bugs when not tested fully.
             | 
             | A big reason Linux chooses not to have stable internal APIs
             | is for agility, more or less. But nothing is free and this
             | is the price that is paid as the project grows.
             | 
             | My personal poster child for this stuff is the amdgpu
             | driver. I use a Navi workstation card (WX5500) in my server
             | and whether or not the amdgpu driver functions correctly on
             | updates is a crap shoot. A while back it went completely
             | headless; when I upgraded to 6.5 like 2 weeks ago, and I
             | had to attach a monitor to look at the kernel logs (network
             | config snafu), my dmesg had 7 kernel faults in its log from
             | amdgpu. Seven! For a 3.5 year old card! With no desktop
             | environment! Despite the fact that the card isn't changing,
             | the _driver_ is changing; new hardware support, expanded
             | interfaces, new features, and those cause regressions. Many
             | subsystems get overhauls and behavioral changes every
             | release, so these things are bound to happen. Testing is
             | not uniform; many parts of the kernel are far more well
             | tested than others. Peripherial drivers are another example
             | of easily broken code (Xilinx code upstream is frankly
             | broken garbage half the time.)
             | 
             | The Linux Kernel is a pretty amazing project in many
             | respects but I'm frankly astonished half the time it
             | actually boots to a working desktop successfully.
        
           | gaganyaan wrote:
           | A knock-on effect of this is that it's easiest to get the
           | drivers into the mainline kernel, where you have to license
           | your code as GPL. I'm not sure that that's intentional, but
           | it does help make source code available for more drivers. A
           | stable ABI would likely see many abandonware driver blobs.
        
             | candiodari wrote:
             | That's 100% intentional. In any case, non-GPL modules
             | cannot be upstreamed (except with a license that is
             | compatible with the GPL, e.g. BSD is okay), and cannot
             | touch a _lot_ of parts of the kernel.
             | 
             | Also, loading a non-GPL module "taints" your kernel (this
             | cannot be undone except through reboot) and tells everyone
             | on the support mailinglists that you've loaded proprietary
             | code.
             | 
             | https://git.kernel.org/pub/scm/linux/kernel/git/stable/linu
             | x...
             | 
             | And there is much hating on Nvidia, for using a
             | "workaround". Including, famously:
             | 
             | https://www.youtube.com/watch?v=tQIdxbWhHSM
             | 
             | Also famously, this is the reason ZFS is not in the linux
             | kernel.
        
               | Nextgrid wrote:
               | > cannot touch a lot of parts of the kernel
               | 
               | Ironically, this is a form of DRM.
        
           | lttlrck wrote:
           | Maybe it's untenable, counterproductive or simply a bad idea
           | :-)
           | 
           | But has there ever been any attempt to create an abstraction
           | layer in the kernel that does provide a stable ABI? Something
           | that could be used for certain classes of driver?
        
             | esaym wrote:
             | I don't know the exact rule, but "abstraction layers" for a
             | driver are disallowed. I believe it was AMD or ATI that
             | added a new driver for a device, but they basically just
             | took the code from their proprietary driver, and then put
             | an "abstraction layer" in the middle. That did not make the
             | kernel overlords happy and that version of the code was
             | never merged.
        
               | skitter wrote:
               | Are abstraction layers still disallowed if they are only
               | able to load GPL code?
        
               | mcny wrote:
               | > Are abstraction layers still disallowed if they are
               | only able to load GPL code?
               | 
               | How would you possibly implement this as a technical
               | restriction?
        
               | 0cf8612b2e1e wrote:
               | Isn't that effectively what NVidia does anyway? Some
               | minimal shim that lets their blob do all of the work?
        
             | rrdharan wrote:
             | https://source.android.com/docs/core/architecture/kernel/ab
             | i...
        
         | maccam94 wrote:
         | The Linux kernel doesn't have a stable ABI. Thus, if a kernel
         | function signature changes, or a subsystem gets refactored,
         | etc, drivers get updated as part of the process. If the drivers
         | lived outside the kernel tree, they would have to be updated
         | separately by their own maintainers. That's less efficient and
         | prone to breakage, so generally driver modules are merged into
         | the kernel tree. Often they can even share code with other
         | hardware devices!
        
           | adrian_b wrote:
           | Not only there is no stable ABI, but there is no stable API.
           | 
           | When only the ABI changes, a recompilation of all sources is
           | enough.
           | 
           | When the API changes, then programmers must manually edit the
           | sources and change function invocations and data structures,
           | to match the new API.
        
       | Jeff_Brown wrote:
       | They need to put more ads on that page. I managed to see some of
       | the text.
        
       | mnd999 wrote:
       | I'm guessing they broke zfs again? Just because they can.
        
       ___________________________________________________________________
       (page generated 2023-10-28 23:02 UTC)