[HN Gopher] Linux 6.18 Will Fix Lockups When Systemd Units Read ...
___________________________________________________________________
Linux 6.18 Will Fix Lockups When Systemd Units Read Lots of Files
Author : Bender
Score : 29 points
Date : 2025-09-27 20:26 UTC (2 hours ago)
(HTM) web link (www.phoronix.com)
(TXT) w3m dump (www.phoronix.com)
| CaliforniaKarl wrote:
| This seems to me to be a cgroup issue, not a systemd issue,
| though systemd's pervasive use of cgroups make it the most-
| obvious trigger.
| themafia wrote:
| It was a shared queue with high contention and it was being
| sorted more than necessary. The fix is to use independent
| queues and to not sort the dirty list.
| gpm wrote:
| When I stop and think about it writing access times for
| everything seems extremely wasteful... does anything actually use
| this field? Any reason I shouldn't change all my file systems to
| mount with noatime?
|
| It's hard for me to imagine using it for anything myself,
| considering the number of times I do something like run a search
| (or a backup command) across literally every file I care about.
| themafia wrote:
| The canonical example of an application that can break with
| 'noatime' is the "mutt" email client with mbox style single
| file email spools.
|
| Most modern applications are not designed to operate on shared
| files like this so in general 'noatime' is safe for 99.9% of
| software.
| bryanlarsen wrote:
| Mail readers that use the mbox format are pretty much the only
| common user.
| JoshTriplett wrote:
| Ideally, noatime would be the _default_ , and applications that
| still care about atime would be updated to open with a new
| `O_ATIME` flag. Or, better yet, track it themselves
| independently.
|
| It's completely reasonable to turn it on. And also, when you're
| writing applications for Linux, consider using the `O_NOATIME`
| flag in your file opens.
| pengaru wrote:
| It's one of those things that you don't care about until you
| do.
|
| As a former sysadmin through the dotcom booms, we regularly
| depended on atime for identifying which files are actively
| being used in myriad situations.
|
| Sometimes you're just confirming a config file was actually
| reloaded in response to your HUP signal. Other times you're
| trying to find out which data files a customer's cgi-bin mess
| is making use of.
|
| It's probably less relevant today where multi-user unix hosts
| are less common, but it was quite valuable information to
| maintain back then.
| __turbobrew__ wrote:
| > we regularly depended on atime for identifying which files
| are actively being used in myriad situations
|
| You can do that with bpf tooling now, for example the
| `opensnoop` BCC program can capture all file opens on demand.
| You can also write tools which capture all POSIX IOs to
| specific files/directories. I can see atime sometimes being
| useful in some super niche use cases such as hisenbugs you
| cannot reproduce reliably, but I would be reaching for BPF
| tools first.
| NekkoDroid wrote:
| > does anything actually use this field?
|
| Systemd in a way does. One of the systemd-tmpfiles entry option
| is to clean up unused files after some time (it ships defaults
| for /tmp/ after 10 days and /var/tmp/ after 30 days) and for
| this it checks atime, mtime and ctime to determin if it should
| delete the file (I think you can also take a flock on the file
| to prevent it from being deleted as well)
| Avamander wrote:
| I've found it useful for forensic reasons and debugging.
| williadc wrote:
| I use atime to identify archives that can be retired. It's
| common for circuit designer to release a lot of large files for
| their peers to analyze or incorporate into a parent/grandparent
| simulation. They will use that data for as long as it is still
| relevant, which means different things for different types of
| data, and the only consistent thing we've found is that if the
| data hasn't been accessed in awhile, then we can retire it.
| jmclnx wrote:
| I admit, I do not fully understand systemd, but having to add
| logic like this is very odd. If "too many" is reached, couldn't
| they add a pause and throw a message into /var/log/messages ?
|
| This indicates to me a very poor design. _If not_ , it is a
| validation of the old UNIX saying "do one thing and do it well"
| and "keep programs small" (paraphrasing).
| pengaru wrote:
| > I admit, I do not fully understand systemd ... >
| This indicates to me a very poor design. If not, it is a
| validation of the old UNIX saying "do one thing and do it well"
| and "keep programs small" (paraphrasing).
|
| You don't need to fully understand systemd to understand TFA
| describes a _kernel_ fix.
|
| This isn't a systemd problem, systemd just makes use of
| cgroups. The kernel has a degenerate case handling lazy atime
| updates combined with cgroups.
| pizlonator wrote:
| Kinda yeah?
|
| I'd say it's both a systemd issue and a kernel issue. The
| fact that systemd motivates kernel fixes does point to
| systemd being maybe just a bit overengineered
| pengaru wrote:
| > I'd say it's both a systemd issue and a kernel issue. The
| fact that systemd motivates kernel fixes does point to
| systemd being maybe just a bit overengineered
|
| systemd is basically a victim here, you're quasi engaging
| in a tech form of victim blaming.
|
| don't blame systemd for making use of kernel features
| (cgroups)
|
| and without cgroups linux has no sandboxing capabilities,
| and would be largely irrelevant to today's workloads
| pizlonator wrote:
| If I was blaming only systemd then you'd be right.
|
| Look if I wrote a thing that caused kernel lockups then
| I'd blame myself even if the kernel dudes fixed the issue
| lokar wrote:
| The kernel has a very clear API and expected behavior.
| Systemd is not doing anything wrong, it's using the API
| correctly.
|
| It's a kernel bug.
| wrs wrote:
| It's not really specific to systemd, it's about cgroups in the
| kernel. If your code is running as a systemd unit so it gets
| its own cgroup, there you are.
| malkia wrote:
| I was listening to Matt Godbolt's the Two's Complement podcast
| "Squashing Compilers" and this got my attention, I think this was
| Ben Rady sharing his recent systemd issue - seems like related
|
| https://youtu.be/Au15lSiAkeQ?si=sxxP2ia9vUkWY5qy&t=982
|
| From the YouTube transcript:
|
| "I don't know what systemd is doing to take so long cuz this is
| the rub systemd essentially takes 100% CPU twice over. So on our
| two core machine that we run these things on, I can run top that
| when I actually got it, I said to you the machine was
| unresponsive, right? Because all in kernel land, locks are being
| taken out left, right, and center. Um, you know, we're trying to
| mount these things in parallel at sensible levels because we want
| to try and mount"
___________________________________________________________________
(page generated 2025-09-27 23:00 UTC)