[HN Gopher] Six Great Features with the Upcoming Linux 6.6 Kernel
___________________________________________________________________
Six Great Features with the Upcoming Linux 6.6 Kernel
Author : danielklnstein
Score : 45 points
Date : 2023-10-28 13:42 UTC (9 hours ago)
(HTM) web link (www.phoronix.com)
(TXT) w3m dump (www.phoronix.com)
| ahmedfromtunis wrote:
| Off topic, but I had this question for years now: why does the
| Unix kernel (try to) contain the drivers to every and each
| device?
|
| Wouldn't it be cleaner, easier and simpler to keep those separate
| and add them later as "modules"?
|
| Isn't this how we're supposed to write other types of software?
| Why is this case different?
| woodruffw wrote:
| > Wouldn't it be cleaner, easier and simpler to keep those
| separate and add them later as "modules"?
|
| That is exactly what Linux does. Part of building a Linux
| distribution is determining which modules to ship by default to
| your users. To the best of my knowledge, the overwhelming
| majority of drivers are not installed by default, and the
| average user will never need them.
| MaxBarraclough wrote:
| I think that by 'module' they were thinking of user-space
| drivers rather than kernel modules.
| IshKebab wrote:
| No, I think they were imagining kernel drivers but
| distributed separately and interacting with the kernel
| through a stable ABI.
| chownie wrote:
| There's upsides and downsides to both approaches. I think that
| Linux kernel maintainers continue this approach because this
| forces driver maintainers to meet a minimum level of effort to
| get their work merged.
| joecool1029 wrote:
| Theoretically the monolithic approach is supposed to be more
| performant. Abstracting away everything into modules can be
| more stable, but also usually incurs some overhead. But this
| is holy war territory and I'm trying to ELI5 best as I can.
| rwmj wrote:
| One serious downside (for end users) of having a stable
| driver ABI would be that manufacturers would release a binary
| blob driver for their hardware which just about works, but no
| bug could ever be fixed. Manufacturers wouldn't have any
| incentive to fix bugs and kernel developers couldn't do it
| either.
|
| The situation of GPL'd drivers in a kernel with an unstable
| ABI wasn't really planned, but for users it's probably the
| best outcome.
| nuancebydefault wrote:
| I always thought that ABI was the interface between CPU and
| executable code, ie the spec of the assembly language for a
| CPU architecture: aarch64, x86 etc. I now would guess it is
| also a term used for the interface between libraries
| (kernel/driver) as long as they are binary (without the
| source code or headers) ?
| gdprrrr wrote:
| It's also used for programming languages like Rust and
| Haskell. The Rust compiler not having a stable ABI means
| the way the compiler stores information about function
| signatures, debug symbols etc in libraries may change
| between compiler versions, so you have to recompile all
| libraries.
| joecool1029 wrote:
| You're literally describing something like macOS/iOS which is
| based around a microkernel (XNU, based on Mach). See also QNX
| for a non-unix OS that does this.
| speed_spread wrote:
| Err.. loading drivers dynamically according to a stable ABI
| doesn't make an OS a microkernel. If the drivers all share
| the same address space that's a pretty conventional OS. For a
| microkernel you need to have separate processes for each
| driver. It's more robust but there's more IPC overhead.
| danielklnstein wrote:
| As another comment explains - the vast majority of drivers are
| not compiled as part of a standard Linux distribution.
|
| If you're asking why so many drivers are included in Linux's
| mainline repository and not maintained off-tree - the major
| advantage of being upstreamed is that changes to the kernel
| have to take your driver into account as well. e.g. anyone who
| changes any APIs will have to update your driver as well.
|
| Moreover, being upstreamed certifies that your driver has gone
| through a non-trivial review process to be merged into the
| mainline and is an indication of quality/stability.
| adrian_b wrote:
| Because the Linux kernel does not have a stable API for
| modules.
|
| The modules included in the kernel are updated whenever the API
| changes.
|
| For the external modules, every time when a new kernel version
| is released, they may become broken and there is a lot of work
| to identify how to update the modules, unless one watches the
| kernel mail lists every day, because there is also no
| documentation about how to migrate the old modules to the new
| kernel.
|
| The reasons for module breakage may be just the movement of
| some definitions from one header to another, which break
| compiling, but most frequently some structures gain new members
| or lose old members, or some functions gain new arguments or
| lose old arguments.
|
| When there are new members or arguments it is hard to discover
| what values should be put in them, while when members or
| arguments are deleted it is hard to discover whether their
| absence must be compensated somehow, e.g. by inserting
| invocations to other functions.
|
| In the worst case everything can be solved by reading the
| kernel source, but that takes a lot of time and those who
| maintain out-of-kernel modules usually do not do this as a
| full-time job, so they do not have time to scan every day the
| kernel mail lists, to see if anyone has plans to make changes
| that will break their modules.
| seanwilson wrote:
| How does this manage to scale? With even a modest number of
| drivers, you'd think the maintenance burden would be
| unreasonably high in terms of needing to have someone
| available who understands how a set of drivers work, knows
| how to test them, and keeps them up-to-date.
| adrian_b wrote:
| That is why everybody attempts to push their modules into
| the kernel source.
|
| Those who cannot do that because their modules are not open
| source, like NVIDIA, have a lot of work to do for module
| maintenance.
|
| Nevertheless, there are also open-source modules that
| cannot be pushed into the kernel because they are for
| uncommon hardware with few users, and those also require a
| lot of work for maintenance, so they are frequently
| abandoned and no longer brought up-to-date, to be
| compatible with the latest kernels.
| seanwilson wrote:
| > That is why everybody attempts to push their modules
| into the kernel source.
|
| > Those who cannot do that because their modules are not
| open source, like NVIDIA, have a lot of work to do for
| module maintenance.
|
| Say someone gets a driver for a WiFi USB stick into the
| kernel source and leaves it for others to maintain.
| Wouldn't the maintainers need the hardware device for
| testing changes and an understanding of the hardware's
| quirks to keep the driver updated? Obviously the system
| works but I don't get how it doesn't suffer from software
| rot, especially when physical hardware is required for
| robust testing including testing the hardware against
| different motherboards.
| toast0 wrote:
| What I've seen is (more or less) the mainline kernel
| maintainers for whichever kernel have the objective of
| making sure the drivers compile.
|
| They can't test what they don't have, and I haven't seen
| evidence than any open source kernel has a testinfg time
| with large fleet of hardware for tests. There's more or
| less an incidental distributed test fleet as users have
| various testing setups, etc; but driver mistakes in
| obscure hardware tends to get found late.
|
| It's not unusual for drivers to be found to have been
| broken for years, and then unless the reporter can fix
| it, or the fix is obvious from the report, the driver
| gets removed.
|
| Another path towards driver removal is some interfaces
| get updated in multiple steps: a new interface is added,
| drivers are updated to the new interface, the old
| interface and any drivers that haven't been updated are
| removed. If nobody has stepped up to update the driver
| before removal, that's an indication of disuse; sometimes
| they get added back after removal though.
|
| For the record, Windows isn't necessarily better. I had a
| printer where there was a working driver, but Windows
| would prefer the non-working driver from Windows Update,
| and it was impossible to get that fixed.
| Veserv wrote:
| You are correct. The answer is that it does suffer from
| software rot.
| rwmj wrote:
| Coccinelle semantic patching is pretty cool for making
| systematic changes across the kernel (eg. adding a
| parameter to the same function everywhere).
|
| https://coccinelle.gitlabpages.inria.fr/website/
| aseipp wrote:
| > How does this manage to scale?
|
| "Okay" to "very poorly", depending on your view. And for
| the record, many times the drivers go untested, frankly.
| I've absolutely found completely broken drivers in the
| kernel (as in just obvious lock imbalances where unlock()
| can't possibly ever get called, null pointer derefs; simple
| things you can spot in a code review) in the kernel where
| the break was introduced as some part of a refactoring and
| the person doing the work made a mistake. Because they also
| refactored 30 other drivers in the same patch series to
| match an API change. (The filesystem developers have said
| many times in patch reviews that certain things are massive
| chores because they have to go fix 40+ filesystem drivers
| every time some API break happens.)
|
| Linux actually has tons of hardware _regressions_ because
| of this whole design choice. When you totally rewrite a
| piece of code that is shared among multiple components,
| that can be OK, as long as you preserve the external
| behavior that the previous interface exposed. An easy way
| to do that is to establish a contract between the
| implementation and call sites, which is often reflected as
| a stable API. The stable API makes some contracts explicit,
| by construction. But many times API refactorings will
| happen at the same time, and _that_ in practice typically
| introduces new behaviors into the downstream code that
| previously didn 't exist before. These new behaviors, when
| introduced into an existing driver that was not developed
| for them, tends to cause bugs when not tested fully.
|
| A big reason Linux chooses not to have stable internal APIs
| is for agility, more or less. But nothing is free and this
| is the price that is paid as the project grows.
|
| My personal poster child for this stuff is the amdgpu
| driver. I use a Navi workstation card (WX5500) in my server
| and whether or not the amdgpu driver functions correctly on
| updates is a crap shoot. A while back it went completely
| headless; when I upgraded to 6.5 like 2 weeks ago, and I
| had to attach a monitor to look at the kernel logs (network
| config snafu), my dmesg had 7 kernel faults in its log from
| amdgpu. Seven! For a 3.5 year old card! With no desktop
| environment! Despite the fact that the card isn't changing,
| the _driver_ is changing; new hardware support, expanded
| interfaces, new features, and those cause regressions. Many
| subsystems get overhauls and behavioral changes every
| release, so these things are bound to happen. Testing is
| not uniform; many parts of the kernel are far more well
| tested than others. Peripherial drivers are another example
| of easily broken code (Xilinx code upstream is frankly
| broken garbage half the time.)
|
| The Linux Kernel is a pretty amazing project in many
| respects but I'm frankly astonished half the time it
| actually boots to a working desktop successfully.
| gaganyaan wrote:
| A knock-on effect of this is that it's easiest to get the
| drivers into the mainline kernel, where you have to license
| your code as GPL. I'm not sure that that's intentional, but
| it does help make source code available for more drivers. A
| stable ABI would likely see many abandonware driver blobs.
| candiodari wrote:
| That's 100% intentional. In any case, non-GPL modules
| cannot be upstreamed (except with a license that is
| compatible with the GPL, e.g. BSD is okay), and cannot
| touch a _lot_ of parts of the kernel.
|
| Also, loading a non-GPL module "taints" your kernel (this
| cannot be undone except through reboot) and tells everyone
| on the support mailinglists that you've loaded proprietary
| code.
|
| https://git.kernel.org/pub/scm/linux/kernel/git/stable/linu
| x...
|
| And there is much hating on Nvidia, for using a
| "workaround". Including, famously:
|
| https://www.youtube.com/watch?v=tQIdxbWhHSM
|
| Also famously, this is the reason ZFS is not in the linux
| kernel.
| Nextgrid wrote:
| > cannot touch a lot of parts of the kernel
|
| Ironically, this is a form of DRM.
| lttlrck wrote:
| Maybe it's untenable, counterproductive or simply a bad idea
| :-)
|
| But has there ever been any attempt to create an abstraction
| layer in the kernel that does provide a stable ABI? Something
| that could be used for certain classes of driver?
| esaym wrote:
| I don't know the exact rule, but "abstraction layers" for a
| driver are disallowed. I believe it was AMD or ATI that
| added a new driver for a device, but they basically just
| took the code from their proprietary driver, and then put
| an "abstraction layer" in the middle. That did not make the
| kernel overlords happy and that version of the code was
| never merged.
| skitter wrote:
| Are abstraction layers still disallowed if they are only
| able to load GPL code?
| mcny wrote:
| > Are abstraction layers still disallowed if they are
| only able to load GPL code?
|
| How would you possibly implement this as a technical
| restriction?
| 0cf8612b2e1e wrote:
| Isn't that effectively what NVidia does anyway? Some
| minimal shim that lets their blob do all of the work?
| rrdharan wrote:
| https://source.android.com/docs/core/architecture/kernel/ab
| i...
| maccam94 wrote:
| The Linux kernel doesn't have a stable ABI. Thus, if a kernel
| function signature changes, or a subsystem gets refactored,
| etc, drivers get updated as part of the process. If the drivers
| lived outside the kernel tree, they would have to be updated
| separately by their own maintainers. That's less efficient and
| prone to breakage, so generally driver modules are merged into
| the kernel tree. Often they can even share code with other
| hardware devices!
| adrian_b wrote:
| Not only there is no stable ABI, but there is no stable API.
|
| When only the ABI changes, a recompilation of all sources is
| enough.
|
| When the API changes, then programmers must manually edit the
| sources and change function invocations and data structures,
| to match the new API.
| Jeff_Brown wrote:
| They need to put more ads on that page. I managed to see some of
| the text.
| mnd999 wrote:
| I'm guessing they broke zfs again? Just because they can.
___________________________________________________________________
(page generated 2023-10-28 23:02 UTC)