Looks like there have been changes to scheduling on Linux (at some point
in the last few years).

The last thing I was aware of was the autogrouping feature. That was
already something like a decade ago, oh my goodness. I had it disabled
ever since, because it doesn't fit my usage pattern.

I just noticed that autogrouping doesn't do anymore what it used to do.
In fact, it appears to have no effect anymore at all these days. Back
when it was introduced, you could run this in one terminal:

    make -j 10  # large build

And then this in another:

    while true; do true; done

And then both these command groups would get 50% CPU time each.

That's no longer the case? They are scheduled "traditionally" again,
i.e. each process of make gets 1/11th of the CPU time and the second
terminal gets the remaining 1/11th.

More so, nice(1) works like it used to work before the autogroup patch.
When I run

    nice -n 19 make -j 10

in the first terminal, then the shell in the second one gets 99% of one
CPU (and the make processes get whatever time is left). (The autogroup
patch broke *this*, that's why I had it disabled.)

So, as of now, I have removed "noautogroup" from my kernel command line.

Maybe this has something to do with the introduction of cgroups v2? I
should catch up on this.


* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *


All that said, nice(1) values still don't apply globally. It appears
they only apply within their respective cgroup/systemd slice. That means
setting "Nice=19" in a system daemon is still irrelevant if that daemon
is competing against some process that runs in my user slice?

The same goes for setting "CPUWeight=idle" in a .service file.

As explained in systemd.resource-control(5), the scheduling is
hierarchical. Below the root slice, there's the system slice and the
user slice. No matter what happens below that point, both these slices
will each get 50% of CPU time (IIUC).

So, when I want to run a system daemon with a lower priority, it appears
I have no choice but to put it into another slice. For now, I have done
this:

    systemctl set-property background.slice CPUWeight=idle

That creates a new slice with absolutely low priority (and also a file
in /etc/systemd/system.control). I can then do

    systemctl edit some-daemon.service

and put this in the override file:

    [Service]
    Slice=background.slice

That does the trick.

(I had first thought about "systemctl set-property background.slice
CPUWeight=1000", i.e. *increasing* the priority of the user slice, but
I'm not sure if that really is a good idea. Better explicitly move the
heavy daemons into the background slice.)