Post B5pCAZnVsFRTpwMXLM by azonenberg@ioc.exchange
(DIR) More posts by azonenberg@ioc.exchange
(DIR) Post #B5pAV0GsxH65pKiJWq by azonenberg@ioc.exchange
0 likes, 0 repeats
Using Arch as a CI runner has been quite the interesting journey due to their packaging workflow and the fact that installing any new package effectively brings in a full system upgrade with it.An upgrade that can include some combination of kernel, nvidia kernel module, and userspace Vulkan stack.Which can result in Vulkan context creation failing with errorIncompatibleDriver until you reboot and everything is in sync again.So now the "spawn VM" script has special cased logic for Arch that patches, shuts down, and restarts the VM before actually launching the build on ithttps://github.com/ngscopeclient/scopehal-ci-scripts/blob/main/vm/spawn-vm
(DIR) Post #B5pB5ZocHS5gUQvlLM by azonenberg@ioc.exchange
0 likes, 0 repeats
Do normal arch users just reboot constantly? I'm trying to imagine the UX of installing a random package and having vulkan suddenly stop working until you reboot because your kernel driver and userspace are out of sync or something like that
(DIR) Post #B5pB8phVfwQmct9wJM by azonenberg@ioc.exchange
0 likes, 0 repeats
In debian land i'm used to installing packages whenever I need them, patching every month or so, and rebooting ~annually when I have hardware maintenance to do
(DIR) Post #B5pBPdBzR0M8ksQTqa by jenesuispersonne@piaille.fr
0 likes, 0 repeats
@azonenberg I'm doing 'yay -Syyu' only once a week before shutting down my computer.
(DIR) Post #B5pBWXmHxphZzOhVqa by azonenberg@ioc.exchange
0 likes, 0 repeats
@GyrosGeier I run the nvidia blob drivers and they always keep working for me.If i use nvidia-installer to grab a new version rather than using the Debian packaged version it's not DKMS'd and will break *after a reboot* when I install a new kernel.But I have never had an update cause the currently installed driver for the running kernel to stop working.
(DIR) Post #B5pBYHEgRCzbGRoCSe by jenesuispersonne@piaille.fr
0 likes, 0 repeats
@azonenberg To install new package I just 'yay <package name>'No reboot needed, no unintended upgrade.
(DIR) Post #B5pBqIUwU2hSOL0SNk by gsuberland@chaos.social
0 likes, 0 repeats
@azonenberg enjoying the implication that there's such a thing as "normal arch users"
(DIR) Post #B5pBqhoeLuJouhQ34S by azonenberg@ioc.exchange
0 likes, 0 repeats
@elebertus I'm trying to test on a large cross section of platforms that our users might be on.As of right now the configurations we actively test on are:* aarch64 macos 15.6 / moltenvk* x86-64 arch / nvidia (tried to do llvmpipe but it was segfaulting)* x86-64 debian 12 / nvidia* x86-64 debian 13 / nvidia* x86-64 fedora 43 / llvmpipe* x86-64 ubuntu 24.04 / nvidia* x86-64 ubuntu 26.04 / llvmpipe* x86-64 win11 / nvidiaMy dev machines are x86-64 debian 13, the one in the office has an AMD card and the lab has an nvidia. Eventually I'd like an AMD in the CI cluster but I am not too worried about things breaking with it being my daily driver and last time I put one in the VM server I couldn't get it working.I have no Intel integrated GPUs in the test cluster either, but I do have an iGPU on a laptop that I occasionally spot-check on
(DIR) Post #B5pC5YLlpPSZtpzUQK by azonenberg@ioc.exchange
0 likes, 0 repeats
@jenesuispersonne Yeah see, you reboot once a week that would not fly in my world lol.I usually have 30+ windows open from different applications, long-running host bridges for JTAG/UART on dev boards on the bench, etc. Getting back to where I left off after a reboot is often an hour or two of work at least, plus the disruption of having my work interrupted.I pretty much only reboot when I have to do hardware maintenance or if there is a kernel bug (security issue that actually matters in my environment, or some driver is nonfunctional) that dictates I immediately switch to the new kernel version rather than waiting until a planned outage.
(DIR) Post #B5pCAZMvT6AuVTvIu0 by dianea@lgbtqia.space
0 likes, 0 repeats
@azonenberg I have been using Arch and Gentoo for a long time and the only thing that keeps breaking between versions is systemd, which is why I learned to avoid that. I measure uptime in years and would be upset if a package required rebooting, because they couldn't fix their userspace feature and attempted to patch the kernel (systemd)I will update some packages often, like ca-certificates and valid security issues, but treat my computers as embedded devices that don't change.
(DIR) Post #B5pCAZnVsFRTpwMXLM by azonenberg@ioc.exchange
0 likes, 0 repeats
@dianea How do you get years of uptime with arch? I have a VM snapshot that was pacman -Syu'd yesterday night and doing a pacman -Syu of a random library today broke Vulkan until I rebooted it.
(DIR) Post #B5pCEEwDN6HYIfBBvE by azonenberg@ioc.exchange
0 likes, 0 repeats
@dianea and since that was the snapshot my CI jobs were based on, I had to add a "patch and reboot" to the run script so I would always have working Vulkan when the tests began
(DIR) Post #B5pCnVAf0eiimUkavo by jenesuispersonne@piaille.fr
0 likes, 0 repeats
@azonenberg I only decide by myself to do it once a week.If you choose to go for once a month, it's ok too.(just remind, like all rolling release the less you update, the higher chances to get problems)For now, I never had issues waiting for one month to do a system update.
(DIR) Post #B5pDH9MwdIebqqdMG0 by dianea@lgbtqia.space
0 likes, 0 repeats
@azonenberg I never update all packages in one big sweep every time like many do. If one of the package maintainers went through USA airport security, they could be compromised, that kind of thing. It only takes one out of a thousand packages to compromise the whole system and updating everything everyday is amplifying risk. I do use rolling distributions, but do so on my terms. And that's the way I have done it at least since Gentoo came out. The biggest problem I've had with that is the first storm that lasted longer than the UPS batteries and the hard drive wouldn't spin back up.
(DIR) Post #B5pDH9cXhJ8AdDvoZM by azonenberg@ioc.exchange
0 likes, 0 repeats
@dianea i ran into other breakage when I didn't do that (trying to install a package and it depended on something else and the update didn't pull that in or something).This is a CI runner so it's pretty much a throwaway sandbox anyway.On my real production debian systems I do a full upgrade once a month or so unless something major comes out requiring an immediate update
(DIR) Post #B5pDOJEoL5xsUCF8Do by shironeko@fedi.tesaguri.club
0 likes, 0 repeats
@azonenberg it's a rolling distro, so the expectation is when you install packages your system is already up to date. It's why there's no "install" option only "sync" IMO.
(DIR) Post #B5pDOJWXHC8vNAXHqi by azonenberg@ioc.exchange
0 likes, 0 repeats
@shironeko yeah but like if I'm 24 hours out of date and install a package i wouldn't expect my gpu to stop working until i reboot
(DIR) Post #B5pEJKFlEk9y8osIz2 by shironeko@fedi.tesaguri.club
0 likes, 0 repeats
@azonenberg well, arch sucks for lots of things, computer that can't be rebooted is chief among them.Even if nothing breaks, when a new kernel comes out arch expects you to reboot.
(DIR) Post #B5pEJKQ2cWNoehgW0W by azonenberg@ioc.exchange
0 likes, 0 repeats
@shironeko yeah i am used to installing a kernel during monthly patching then having it sit on deck until next time i do hardware maintenance.My workstation I'm sitting at now had a CPU upgrade earlier in the week but before that, other than one reboot in January that I can't recall the reason for, the last reboot was in Q1 2025
(DIR) Post #B5pEJLE1cgMj9j4Quu by shironeko@fedi.tesaguri.club
0 likes, 0 repeats
@azonenberg I mean realistically, if your computer can't be rebooted when required, it's probably not running a use case that require all bleeding edge software. in which case it shouldn't be running arch.
(DIR) Post #B5pENMzoyrY7o4LcUC by azonenberg@ioc.exchange
0 likes, 0 repeats
@shironeko yeah this is a CI runner so it's something I can blow away whenever I want. the issue is that I want to clone the snapshot, run the build, and shut it down and it seems like if a new kernel came out since the snapshot I need to install that and reboot before anything else
(DIR) Post #B5pEsZDrZZvkhfn8sa by shironeko@fedi.tesaguri.club
0 likes, 0 repeats
@azonenberg actually, I wonder how you are running into this situation for that case, are you running pacman -Sy at any point?
(DIR) Post #B5pEsZQymoQFMLvcK8 by azonenberg@ioc.exchange
0 likes, 0 repeats
@shironeko Yes the build is doing this https://github.com/ngscopeclient/scopehal-apps/blob/master/test-scripts/test-driver-arch.sh
(DIR) Post #B5pEzRHTUxwQ6El2eW by azonenberg@ioc.exchange
0 likes, 0 repeats
@elebertus well it's more that because of the quirks of arch I need to bake a "patch, reboot, then run the test" stage into the CI pipeline rather than simply "clone snapshot, update, run"
(DIR) Post #B5pFOM3O4PgV7x36h6 by penguin42@mastodon.org.uk
0 likes, 0 repeats
@azonenberg Rebooting a bit more often is probably a good idea given kernel security fixes; but yeh a lot of my Debian setups I reboot when I feel like it.
(DIR) Post #B5pFcqbpGKo9wo4I2y by azonenberg@ioc.exchange
0 likes, 0 repeats
@penguin42 I run the vast majority of my stuff that opens a socket to the internet in VMs that I can reboot much more casually. But also I'm mostly running single-user systems. very few kernel bugs are RCEs, those are major headlines and the kind of thing that *would* get me to do an out of cycle reboot. But something that lets you privesc to kernel while running as a normal user really doesn't matter to me.If you pop my workstation you a) can get all of my files and data, root doesn't buy you much else and b) it's a dev box I almost always have a root shell open or am going to su/sudo in the next few minutes to do something you can keylog.The goal is to prevent an attacker from getting code exec as my local user in the first place not preventing root once they do that.
(DIR) Post #B5pG8uRcXPsb6qaD8y by penguin42@mastodon.org.uk
0 likes, 0 repeats
@azonenberg Yep; although remember the kernel does a lot of the VM isolation. Still, I do try not to let anything go as long as a year - you're a bit more likely to find a whole pile of odd changes have happened rather than one or two.
(DIR) Post #B5pGR5xqRkIe39KML2 by azonenberg@ioc.exchange
0 likes, 0 repeats
@penguin42 I'm also not running the vms on my endpoint they're on a xen server across the lab that has a separate reboot schedule. It's much less disruptive to suspend the vms and reboot the host than to reboot an endpoint
(DIR) Post #B5pIJQpPBuO4Ljqy6C by emily@fedi.uni.horse
0 likes, 0 repeats
@azonenberg https://github.com/saber-nyan/kernel-modules-hookwhich really should be part of the base system imobetween that and running LTS kernels I reboot probably once every few months
(DIR) Post #B5pIJR1oRmJOyDesRE by azonenberg@ioc.exchange
0 likes, 0 repeats
@emily I... should clean out some of my old kernel module folders i don't have most of these installed anymore but the modules dirs are somehow sticking around
(DIR) Post #B5pPu9NVjQI4ciGI0u by CliffsEsport@mastodon.social
0 likes, 0 repeats
@azonenberg @penguin42 Can I ask what you tend to use as the Guest OS for the VM? More Debian? I have been slowly working on learning Alpine to use for somewhat similar as Guest though I have a few different Host OS.
(DIR) Post #B5pRgEqumW6fSJVdtQ by IvanDSM@mastodon.social
0 likes, 0 repeats
@azonenberg This is a realy weird problem, whenever I want to install a new package I just do yay -S package and it installs just that and its dependencies. Never had it pull in unrelated upgrades.If your local package DB is outdated and it can't install the package because it's been bumped, do yay -Sy to pull in the latest DBs and you'll be good, no world upgrade needed.If you're not using yay, replace it with pacman in these invocations, with the same flags.
(DIR) Post #B5q0RxeRR2aiRsbPuq by azonenberg@ioc.exchange
0 likes, 0 repeats
@CliffsEsport @penguin42 debian is generally my go-to unless i specifically need something else to test on that distro or something
(DIR) Post #B5q0zJSDOTa3comqy8 by azonenberg@ioc.exchange
0 likes, 0 repeats
@CliffsEsport @penguin42 I've described my setup as "qubes on private cloud". Same idea of isolated vms per security domain except i run them on a shared server and ssh+vnc to them so i can have one mail client, messaging app, etc shared among all of my endpoints
(DIR) Post #B5q1za5hdciBB4wtQ8 by CliffsEsport@mastodon.social
0 likes, 0 repeats
@azonenberg @penguin42 My primary idea is to do similar with VMM on Chromebooks since they natively support VM now. I think using VMM that way is actually a nested VM. I had full MX XFCE working that way on Acer 514 Chromebook+ w i3-n305 8GB RAM and 500 GB NVME. Though also want to do the similar to you with Linux host. IMHO the Chromebook would be more secure but less privacy. Normally use Chromebook w Firefox or Links (text browser) on untrusted networks.
(DIR) Post #B5q2JDeWlygjNVbOAC by azonenberg@ioc.exchange
0 likes, 0 repeats
@CliffsEsport @penguin42 I'm using xcp-ng as the host and debian+xfce in the vms. If i need a browser it's Firefox/noscript/ublock
(DIR) Post #B5qGcxnooNvPMzFz72 by qualia@floofy.tech
0 likes, 0 repeats
@azonenberg yes, and it was abjectly miserable. after almost two decades of Debian exclusivity, i gave Arch a shot for a bit because I was initially under the impression that VR on Linux benefited from maximally up-to-date everything.graphics drivers and VR-specific userspace tooling certainly does, but turns out I can just build all that on Debian stable/testing without much fuss. which is great because having to reboot every 36 hours was infuriating. to say nothing of the constant DDoS attacks on the SPoF that is AUR
(DIR) Post #B5qGlCWnwiwVMCE72O by azonenberg@ioc.exchange
0 likes, 0 repeats
@qualia lol yeah i was just telling somebody else that other than a reboot last week to install a new motherboard/CPU and one back in January for upgrading from Bookworm to Trixie, the last time my office workstation rebooted was in Q1 2025
(DIR) Post #B5qGt9ttbDLTWZp0yW by azonenberg@ioc.exchange
0 likes, 0 repeats
@qualia I dont know the exact time because I don't have logs that far and my monitoring system graphs only go back a year, but interpolating the 1-year uptime trend line back before april 2025 it would probably hit zero somewhere in the jan-feb time frame lol