https://stgraber.org/2023/08/24/stable-linux-mainline-builds/ Stephane Graber's website Skip to content [random] Skip to content * Home * About me * Projects - A month later Stable Linux mainline builds Posted on 2023/08/24 by Stephane Graber Why use a mainline kernel For the past year or so, I've increasingly been using mainline Linux kernels on my various servers and eventually laptop and desktop machines too. That was transitioning from Ubuntu's generic kernel which I feel has sadly decreased in quality over time. The Ubuntu kernel includes a lot of backported fixes and occasionally, those backports go bad, resulting in missing commits, introducing bugs and regressions. Unfortunately the way the Ubuntu kernel is built, tested and published comes with a lot of delays, making fixing such regressions often take weeks if not months (depending on whether security updates show up in between). So I started taking the latest stable bugfix release of the mainline kernel, generate a configuration that's very close to an Ubuntu generic kernel, cherry-pick a few small changes that aren't upstream yet and then build that and push it to my machines. That's been working surprisingly well so far! Those kernels haven't been perfect, I did catch a couple of regressions, but as I'm now working with a mainline kernel, performing a bisect, identifying the offending commit and getting it resolved upstream is very easy, with a revert taking an hour or so at most and a fix taking just a few days to hit mainline. Making them available to everyone Up until now, I've been manually building those kernels from an internal git repository, building them directly on a couple of servers (amd64 and arm64) and then transferring the resulting .debs directly to my other machines. That works, but it's not a particularly clean build environment and installing kernels that way doesn't really scale! That's why I've now spent a few days moving it all to Github and a proper package repository. The kernel tree is now available here: https://github.com/zabbly/ linux [image] For building, I'm using some self-hosted Github runners on my local Incus cluster so I can have access to beefy Debian and Ubuntu builders on both amd64 and arm64. The result is a repository that contains both amd64 and arm64 builds for Ubuntu 20.04 LTS, Ubuntu 22.04 LTS and Debian 12. This is all automatically built and automatically imported into the repository with the only manual step being to update the "linux-zabbly" meta-package after testing the new kernel on some test systems. Using them Installation instructions can be found here: https://github.com/ zabbly/linux#installation Just keep in mind that you'll most likely have to disable UEFI SecureBoot as those kernel builds aren't signed unlike those that come directly from your distribution. The kernel will be updated once a week unless something major happens requiring an intermediate update. It will roll from one kernel version to the next after it has received its first bugfix release which has so far been a good way to avoid some of those initial regressions! ZFS I use ZFS quite extensively to store local containers and VMs on Incus. The Ubuntu kernel ships with a built-in version of ZFS but to keep the Zabbly kernel clean, I opted not to do that. Instead, I maintain a separate ZFS repository at: https://github.com/ zabbly/zfs This currently contains ZFS 2.2rc3 and will be updated with new release candidates and eventually the 2.2 stable release. The decision to ship 2.2 rather than stick to 2.1 is motivated by ZFS 2.2 properly handling VFS idmap shift, a critical feature for Incus. That repository includes both openzfs-zfs-dkms, the package providing the kernel driver as well as the usual set of tools used to manage zfs, openzfs-zfsutils. This entry was posted in Planet Ubuntu, Zabbly. Bookmark the permalink. - A month later 4 Responses to Stable Linux mainline builds 1. [5819] Jarrod Urban says: 2023/08/25 at 6:56 AM Do you have the same issues with the debian 12 kernels? Reply 1. [93d3] Stephane Graber says: 2023/08/25 at 12:24 PM I do not, though to be fair, I've only recently started using Debian again here and there after an almost 20 years long break from the project The main benefit from having the Debian builds is that I now have an absolutely identical kernel on all systems regardless of distribution, making transitioning a fair bit easier. Reply 2. [44b4] Mike P. says: 2023/08/25 at 10:12 AM What instabilities have you experienced? Reply 1. [93d3] Stephane Graber says: 2023/08/25 at 12:30 PM Off the top of my head for the past year or so, the main ones I've hit are: - Regression in KVM support causing a variety of systems to trigger a kernel log loop when starting LXD VMs, very rapidly filling the entire disk with log causing further breakage. ( https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1966499) - Regression in ZFS+shiftfs handling breaking LXD (https:// bugs.launchpad.net/ubuntu/+source/linux/+bug/1990849) - Regression in io_uring handling allowing for unprivileged users to deadlock or panic the kernel (https:// bugs.launchpad.net/ubuntu/+source/linux/+bug/2020901) Those are the ones I remember the most as I was involved in either reporting them or having someone directly fix them. I was lucky to have a kernel engineer working directly on the LXD team that I could temporarily redirect to fixing those issues. The Ubuntu kernel is a massive beast of patches, tracking down what was missed or badly merged is much harder than just doing a normal bisect and fix on a regular mainline kernel. Even with that though, the time to resolution hasn't been great, a "quick" fix usually would still take 6-8 weeks, with some of the more tricky ones like that ZFS one taking several months, all while leaving users broken or running old unsafe kernels... Reply Leave a Reply Cancel reply Your email address will not be published. Required fields are marked * [ ] [ ] [ ] [ ] [ ] [ ] [ ] Comment * [ ] Name * [ ] Email * [ ] Website [ ] [Post Comment] [ ] Notify me of followup comments via e-mail [ ] [ ] [ ] [ ] [ ] [ ] [ ] D[ ] This site uses Akismet to reduce spam. Learn how your comment data is processed. * Social + Github Github + Twitter Twitter + LinkedIn LinkedIn + Mastodon Mastodon * Feeds + RSS feed RSS feed + RSS feed (comments) RSS feed (comments) * Categories + Arkose + Canonical voices + Conferences + Edubuntu + gbtsco + Incus + IPv6 + iTalc + LTSP + LXC + LXCFS + LXD + mirrorkit + pastebinit + Planet Revolution-Linux + Planet Ubuntu + Sandbox + Ubuntu Touch + WebLive + Zabbly * Archives + August 2023 + July 2023 + June 2021 + December 2020 + June 2017 + April 2017 + March 2017 + February 2017 + January 2017 + December 2016 + October 2016 + June 2016 + April 2016 + March 2016 + December 2015 + April 2015 + September 2014 + February 2014 + January 2014 + December 2013 + September 2013 + July 2013 + February 2013 + November 2012 + October 2012 + September 2012 + July 2012 + May 2012 + March 2012 + February 2012 + January 2012 + September 2011 + August 2011 + July 2011 + June 2011 + May 2011 + April 2011 + March 2011 + February 2011 + January 2011 + December 2010 + November 2010 + October 2010 + September 2010 + February 2010 + January 2010 + December 2009 + November 2009 + October 2009 + July 2009 + April 2009 + March 2009 + December 2008 + September 2008 + July 2008 + June 2008 + May 2008 (c) 2023 - Stephane Proudly powered by WordPress. 2010 Weaver Graber's website by WPWeaver.info