[HN Gopher] Working with Systemd Timers
___________________________________________________________________
Working with Systemd Timers
Author : skwee357
Score : 65 points
Date : 2025-03-10 21:04 UTC (4 days ago)
(HTM) web link (yieldcode.blog)
(TXT) w3m dump (yieldcode.blog)
| MantisShrimp90 wrote:
| It took a minute to setup, but using a combination of rsync and
| timers to backup system files has done wonders to decrease my
| anxiety around upgrades on arch
| bhaney wrote:
| Using a modern CoW filesystem lets you level this up to just a
| snapshot command in a pacman pre-upgrade hook.
| tombert wrote:
| I know everyone hates on systemd, but I've generally been happy
| with it, in no small part because I find the timers to be pretty
| sensible.
|
| They are, in all ways that I care about, simply better than cron,
| and especially in NixOS they're really easy to set up.
| linsomniac wrote:
| I'm convinced that the people who hate systemd are a small
| minority. They also happen to be fairly vocal, which I think
| makes them seem outsized.
| throwaway314155 wrote:
| One might even go so far as to say they're a vocal minority.
| DaSHacka wrote:
| One might even go so far as to say all Linux users are a
| "vocal minority" to begin with.
| jauntywundrkind wrote:
| I'd be excellent to see their complaints collected and
| elaborated. Because God Fucking Damn, these seem like some
| incredible whining Advanced Persistent Threats against
| anything getting better. If I were a pro-discord disruptor,
| I'd pay these people to keep at their endless grudges against
| every single person having to figure out ever single fucking
| capability de-novo.
|
| Systemd makes so so so many incredibly good service
| administration capabilities easy & on tap. The main thing I
| want from systemd haters is to show how how how folks did
| things well before. Mostly I think this is a rebellion of the
| losers, people who hate having to do a good job, people who
| hate learning the many many many many ways we could run
| services better, who valorize each service figuring out their
| own unique special bespoke ways of running their stuff. Its
| fallen as fuck. We need need _need_ the systemd socialization
| of running stuff well, controlling permissions & access.
| There's near zero precedent from the past of anyone else
| being responsible. That's just the situation. Systemd has
| drastically elevated what sysops can do, orthogonally to the
| service by service init launching of the past that got us no
| where.
| DaSHacka wrote:
| > I'd be excellent to see their complaints collected and
| elaborated.
|
| https://web.archive.org/web/20250310201357/https://nosystem
| d...
| pdimitar wrote:
| I challenge the authors of this website to show me a
| software written in the last 10 years that never had a
| bug. I have scanned the first 20-ish items and it seems
| those are either fixed or not considered important
| (unsure about the latter; didn't give them too much
| reading as of the time this comment was written).
|
| In general it seems it's just a collection of GitHub
| issues. The FQDNS thing seem to be people outraged at
| systemd for something that the Linux kernel does not (or
| did not?) do well with certain hostnames -- those ending
| with a dot.
|
| And there are some security issues / CVEs which is
| regrettable and I hate it but again, show me software
| that does not have them.
|
| Seems like a very petty way to just attack something from
| a cursory glance. The author(s) might have as well made a
| ClickHouse remote query directed at systemd's GitHub
| issue tracker. -\\_(tsu)_/-
| MyOutfitIsVague wrote:
| I think in general it's a shrinking minority. I was
| outspokenly critical of systemd in the past, but much about
| it has improved, and I have to admit it has made my life
| significantly easier in many ways. I'm still not a big fan of
| some things being systemd projects that don't really need to
| be under the umbrella (why is resolved even a systemd
| project? Or systemd-timesyncd?) but service definition and
| management, transient services, sockets, timers, and many
| many more things are pretty great. At this point, I pretty
| much love it. Even things that initially annoyed me, like
| timers, I've grown to love, not least because my cronjob
| entries are manageable files that I can commit and backup,
| and I can individually enable and disable them without having
| to deal with stupid commenting and such.
|
| Checking service status and managing services is also
| significantly better with systemd than any other init system
| I've used, and I've used a lot over 15 years of systems
| administration.
|
| I still find journald a bit annoying, though, and I still am
| not completely sold on binary log files. I understand the
| benefits, but working with them is still much more opaque
| than text logs.
| Cyph0n wrote:
| +1 for NixOS. I used to find systemd a bit intimidating, but
| working with NixOS has made it all fit together somehow.
|
| As an example, this is a systemd timer I have that periodically
| runs rclone to sync photos to Backblaze:
| systemd.services.rclone-photos-sync = {
| serviceConfig.Type = "oneshot"; path = [ pkgs.rclone
| ]; script = '' rclone \
| --config ${config.age.secrets."rclone.conf".path} \
| --bwlimit 20M --transfers 16 \ sync
| /mnt/photos/originals/ photos: '';
| unitConfig = { RequiresMountsFor = "/mnt/photos";
| }; }; systemd.timers.rclone-photos-sync = {
| timerConfig = { # Every 2 hours.
| OnCalendar = "00/2:00:00"; RandomizedDelaySec =
| "5m"; Persistent = true; Unit =
| "rclone-photos-sync.service"; }; partOf = [
| "rclone-photos-sync.service" ]; wantedBy = [
| "timers.target" ]; };
| SoftTalker wrote:
| That would all be one line in a crontab.
| Cyph0n wrote:
| The article already explains the difference between cron
| and systemd timers, so I won't rehash that. Yes, systemd is
| more verbose, but that comes with advantages over cron. And
| if cron suits your usecases, it's a fine tool as well.
|
| But with NixOS and this example specifically, you also get
| (at a minimum):
|
| a) A declarative config without requiring an external tool
| like Ansible.
|
| b) Any dependencies (rclone in this case) will be
| _implicitly_ installed if not already present on the
| system.
|
| c) If configuring a remote machine, this will copy over the
| encrypted rclone.conf file and decrypt it on the target.
|
| d) And of course, it's trivial to version control and track
| changes to the config over time.
| ForTheKidz wrote:
| Putting aside that this is all possible with nix and
| without systemd, the additional complexity strikes me as
| having hamstrung the linux community for the last decade.
| I hope reliable service management arises but I have yet
| to see the benefit.
|
| In the meantime I'm just going to keep using void because
| I know how to fix it when it breaks. Systemd i have to
| google how to even find the service logs.... they
| certainly aren't easy to find via the filesystem any
| more.
| Cyph0n wrote:
| Like I said, it is a bit intimidating at the start. But
| you soon realize it's quite intuitive.
|
| Finding logs for a specific service is a one-liner:
| journalctl -u rclone-photos-sync.service
|
| Add -r (reverse) to return newest logs first. Add -b to
| limit logs to the current boot.
|
| Here is a nice overview:
| https://www.digitalocean.com/community/tutorials/how-to-
| use-...
| gf000 wrote:
| Well, above a level tools can't be designed to be
| discoverable without any form of help/learning, and I
| believe that's fine. They should rely on a "common
| knowledge base" as much as possible, and I believe
| systemd commands are done in a POSIX-y style, but you do
| have to occasionally read a man page or Google.
|
| Do we expect an average dude to just pop into a cockpit
| and land a plane with no previous training?
| MortyWaves wrote:
| I want to use NixOS but man it's a mess with its flakes
| vs no flakes debates. I've stuck with learning Ansible
| for now, but if flakes is ever properly built in I'd
| consider it.
| Cyph0n wrote:
| Flakes are as stable as they can get imo. They are
| already widely adopted, so I don't see how they can ever
| completely walk the feature back. The unstable marker is
| only there because there are some edge cases to work
| around afaik.
|
| And I agree that NixOS without Flakes is a worse
| experience overall.
| stephenr wrote:
| That isn't inherently better.
|
| The really nice thing about systemd timers is that as units
| you can get status of them and check the journal output the
| same as any other unit.
| zeec123 wrote:
| How is 'persist' realised with cron? What about 'randomized
| delay'?
| SoftTalker wrote:
| I'm not sure what 'persist' means here but at least some
| cron implementations have support for randomization in
| times or time intervals.
| imp0cat wrote:
| the time when the service unit was last triggered is
| stored on disk. When the timer is activated, the service
| unit is triggered immediately if it would have been
| triggered at least once during the time when the timer
| was inactive. Such triggering is nonetheless subject to
| the delay imposed by RandomizedDelaySec=. This is useful
| to catch up on missed runs of the service when the system
| was powered down.
| tombert wrote:
| I had to write a systemd timer to rescan PCIe ports about ten
| seconds after the computer starts. I was able to add that to
| my NixOS config and it worked how I wanted, and it was great
| because if I had broken anything all I would have to do is
| reboot and choose an older generation.
| freedomben wrote:
| I like so much about Nix and the Nix philosophy, but I'm a
| bit stuck on the perceived ugliness of the Nix language. It
| just feels so ... icky to me. I don't typically feel that way
| about languages. Even Erlang (while being bizarre)
| didn't/doesn't bother me that much.
|
| Does anybody else have this problem? How do you get over it?
| (it seems like such a ridiculous reason to avoid picking up
| an otherwise good tech)
| bitwize wrote:
| You may wish to give Guix a try. Nix philosophy, but the
| language is Scheme.
| bitwize wrote:
| Systemd hater here. My stance is and always was: if you love
| systemd, go nuts with it. I'm not looking to reach into your
| bathroom to set your shower temperature.
|
| But I don't want to have to run it or even think about it.
| Lennart has other plans, hence his campaign to get distros to
| require it and software up the stack to hard-depend on it. I
| don't like its design, I want to opt out of it in favor of
| something better, but one of the development effort's goals is
| making that extremely difficult for modern Linux systems.
| That's my issue with it.
|
| But really it macht nichs for me right now because I run Void,
| btw.
| eth0up wrote:
| I was feeling a bit Kafka-ish with all the gushing systemd
| support until I saw this comment. And same here; I don't care
| what other folks are enamored with, but them "other plans"
| you mention, they concern me and I wish it concerned others
| too.
| Vilian wrote:
| >I don't like its design, I want to opt out of it in favor of
| something better
|
| Others design were already tested extensively, and the one
| that stuck was systemd, don't you already stopped to think
| that the "better design" is the one that systemd are
| currently using?
| bitwize wrote:
| Why are you even using Linux to begin with? Other OS
| designs have already been tested extensively, and the one
| that stuck was Windows. Didn't you consider that Windows is
| the better OS for your needs?
|
| My criteria for a good init system design do not align with
| Lennart's, or with Red Hat's. It's not winner-take-all in
| principle, even if it approaches being so in practice.
| paulddraper wrote:
| > distros to require it
|
| Distros requires a lot of stuff, e.g. libc. This does not
| seem inappropriate given what a distro is.
|
| > software up the stack to hard-depend on it
|
| What does this refer to?
|
| I can see this for session/user/service management programs.
| But not normal single-user programs.
| stephenr wrote:
| I was initially against systemd (around the time Debian started
| _discussing_ using it) for non-technical reasons: the project
| seemed to have some very negative aspects in terms of how
| feedback and concerns are addressed. I don 't know for sure but
| I strongly suspect this was the core issue a good chunk of
| those "against" were concerned about: the way the project is
| run.
|
| Having used it for several major releases of Debian technically
| I'm happy to have it - it's an improvement over the sysv init &
| crontabs. I'm not completely in love with journal compared to
| regular logs but I can see some advantages.
| timrichard wrote:
| When I started using systemd timers, I really liked the systemd-
| analyze calendar facility, to calculate n trigger times for a
| given calendar expression.
|
| For example, show the next five trigger times for the end of the
| last day when the month has 31 days :
|
| systemd-analyze calendar --iterations=5 '*-*-31 23:59:59'
| bhaney wrote:
| I like systemd's timers when they're appropriate, but I really
| think the author's use case here is better suited to cron. The
| "issues" he listed for cron aren't very good either.
|
| > If you want to execute pre/post commands you have to do it
| inside the script itself
|
| So?
|
| > There are no built-in logs
|
| Every cron implementation I can remember using logs each run to
| syslog _and_ emails me the output of the run by default
|
| > There is no built-in status monitoring
|
| I can't think of any built-in status monitoring that systemd has
| for timers that's materially different from cron's
| logging/emailing
|
| > If the system is down when the cron needs to run, the cron will
| be missed
|
| Some cron implementations support this and some don't. Most
| modern ones that I'm aware of do.
|
| Much more significantly, the amount of setup involved in a
| systemd timer is way higher than putting a line in a crontab,
| especially for the author's case of just running a backup script.
|
| Cron only involves running `crontab -e` and adding the line
| "@daily /path/to/script.sh" (which also handles the author's
| issue of cron "skipping" runs if the system was powered off,
| assuming the cron implementation uses something modern like
| anacron)
|
| Systemd involves writing a 7 line timer unit file, an additional
| 5 line service file, running a daemon-reload, then enabling the
| timer. It turns what's usually a 10 second mindless task into a
| much more involved procedure. That can be worth it if there are
| material benefits from it, but I'm not really seeing them here.
| burnJS wrote:
| I pair crons with healthchecks.io
| chatmasta wrote:
| It would be cool if someone made a crontab interface to systemd
| timers. Use crontab syntax but generate systemd boilerplate
| with sane defaults.
| bhaney wrote:
| https://wiki.debian.org/systemd-cron
| 1una wrote:
| IMO Systemd Timers provide much better control over how the
| cron job is run.
|
| Need to distribute lots of cron jobs evenly to avoid
| overloading the system? Use RandomizedDelaySec.
|
| Some cron jobs are flaky and you want to re-run if it fails?
| Add Restart=on-failure to corresponding service.
|
| Some cron jobs conflict with each other? Set
| Conflicts=foo.service or maybe Before & After.
|
| Sure, all above are possible with shell scripts. But systemd
| provides a standard, reliable way to define these properties.
| bhaney wrote:
| > Systemd Timers provide much better control over how the
| cron job is run
|
| Yes, they do. And as the very first sentence in my comment
| says, I like systemd timers when they're appropriate. None of
| the features you've mentioned were used in the author's
| solution, which is the sole thing I'm arguing against.
| Ferret7446 wrote:
| > Much more significantly, the amount of setup involved in a
| systemd timer is way higher than putting a line in a crontab
|
| Counterpoint: this requires root, while you can edit and run
| systemd timers without root. Thus it's also generally more
| secure, both when creating the job and every time it runs.
|
| > Systemd involves writing a 7 line timer unit file, an
| additional 5 line service file, running a daemon-reload, then
| enabling the timer. It turns what's usually a 10 second
| mindless task into a much more involved procedure. That can be
| worth it if there are material benefits from it, but I'm not
| really seeing them here.
|
| You could write a script to automate it, if it's such a big
| deal. Creating timers/cronjobs isn't something that needs to be
| done often enough for this to matter.
|
| If it is, another counterpoint: systemd supports creating
| transient timers, which you could do programmatically.
|
| And final counterpoint: you can make systemd understand crontab
| and convert it into timers (systemd-crontab-generator)
| bhaney wrote:
| > Counterpoint: this requires root
|
| Counter-counterpoint: _No it doesn 't_? Running `crontab -e`
| as a non-root user will edit that user's crontab, and running
| it as root will edit the system crontab. Cron can be
| configured to deny users their own crontabs, but every common
| distro I'm aware of defaults to allowing user crontabs.
|
| > You could write a script to automate it, if it's such a big
| deal
|
| Or I could not bother with that and just use cron?
|
| > another counterpoint: systemd supports creating transient
| timers
|
| What is this a counterpoint to? That the author's particular
| use case of running a backup script once a day is a task
| better suited to cron than systemd timers? I'm not sure how
| transient timers are even relevant here, much less a
| counterpoint.
| rascul wrote:
| > Running `crontab -e` as a non-root user will edit that
| user's crontab, and running it as root will edit the system
| crontab.
|
| Running it as root will edit the root user's crontab (in
| /var/spool/cron), which is separate from the system crontab
| (/etc/crontab), which has a slightly different format.
| dizhn wrote:
| Alpine executes system cron jobs (daily, weekly etc) from
| root's crontab. That simple way of taking care of it
| makes me smile every time I see it.
| Ferret7446 wrote:
| I think support for per-user crontab's is implementation
| dependent. There are implementations (AFAIK) that do not
| support per-user crontabs.
|
| Per Gentoo's wiki, both fcron and cronie have their own
| (different!) ways of whitelisting non-root users to run
| cronjobs.
| bhaney wrote:
| Everything is implementation dependent. There's no ISO
| cron standard or anything.
|
| > Per Gentoo's wiki, both fcron and cronie have their own
| (different!) ways of whitelisting non-root users to run
| cronjobs.
|
| Both allow any user in the cron group to have their own
| crontabs by default, and both support optional
| .allow/.deny files
| rascul wrote:
| > There's no ISO cron standard or anything.
|
| It's part of the Single Unix Specification.
|
| https://pubs.opengroup.org/onlinepubs/9799919799/utilitie
| s/c...
| bhaney wrote:
| That's a light spec for a subset of the interface and
| behavior of the crontab command line frontend, not a
| standard for cron in general. It's not really relevant to
| a discussion about how distros set up their cron defaults
| for per-user access.
|
| Thanks for sharing it though. I wasn't aware it was in
| the posix spec and it explains why pretty much every
| implementation supports .allow/.deny files even when most
| already implement better access control mechanisms.
| burnJS wrote:
| This is cool and after learning of this I'm tempted to convert my
| crons. I currently use systemd for running a redis queue and it's
| worked great for years.
| notepad0x90 wrote:
| There is one huge advantage of cron that is usually missed with
| such comparisons. cron is dead-easy to create and maintain.
| Systemd timers make sense in certain situations, but in 90%+ of
| my use cases so far, the added complexity really adds up the
| amount of time I'm administering my systems. I've been in a
| situation where I needed to troubleshoot why a systemd timer
| wasn't triggering, and I didn't like that experience at all. To
| me, it is something I would use if I actually needed it, it isn't
| a 1-to-1 replacement for cron.
| packetlost wrote:
| I actually disagree with simplicity. cron does weird things
| with the environment and it can be hard to correctly reproduce
| that environment when creating a job and debugging. Conversely,
| systemd timers are very easy to manually trigger and otherwise
| behave as a normal oneshot service.
| SoftTalker wrote:
| My approach to cron is to assume that you have an empty
| environment. Just set everything you will need.
| encoderer wrote:
| Yes this is correct. Something most developers don't know
| is that you can set env vars directly in your crontab file.
| encoderer wrote:
| I'm the developer of crontab.guru and Cronitor.
|
| We have a free tool called CronitorCLI that includes a cron-
| like shell. You can run and test your scripts in an
| environment that matches how they will be run by cron itself.
| notepad0x90 wrote:
| Dude! you're my hero! This is why I like HN :)
|
| Thank you for many years of reassuring me when I doubted my
| cron-expressions.
| Ferret7446 wrote:
| There is also one huge disadvantage of cron: it requires root.
| Users can create and run systemd timers without root/sudo.
|
| And quite frankly, your experience is likely due to your
| personal knowledge bias. Systemd timers are quite easy to
| debug, in fact easier than cron in my opinion. systemctl list-
| timers lists all the timers, when they last ran, when they will
| run next, and you can use other commands to inspect each timer
| in detail and all associated logs.
|
| In contrast, I find cron much harder to debug. Starting with
| the first problem, you must first figure out what cron is
| running, as different implementations have different behavior!
| raincom wrote:
| One can set up cron jobs under non-root.
| Ferret7446 wrote:
| You cannot edit /etc/crontab without root. I'm not talking
| about dropping privileges for the job.
|
| Although speaking of which, cron dropping privileges is not
| as secure as systemd running as the user before parsing the
| user's timers, from a defense-in-depth perspective.
| bigstrat2003 wrote:
| But non-root users still have their own crontab, which
| you don't need sudo to edit.
| Ferret7446 wrote:
| That depends on which cron you're using, not all of them
| support user crontabs (and the ones that do probably have
| subtle differences in behavior).
| bigstrat2003 wrote:
| I was unaware of that, so thank you for the education.
| But for what it's worth, I have _never_ seen a Linux
| system which didn 't have user crontabs. So even if not
| universal, it is at least very widespread to the point
| that you can bet on its presence.
| encoderer wrote:
| It's true that root can disable user crontabs but the
| cron that's shipped with all popular Linux distributions
| does have built-in support for them.
| vaylian wrote:
| I found the implicit column-based configuration layout of cron
| rather cubersome. I prefer the key-value configuration layout
| of systemd timers.
| kinglawrence wrote:
| Really liked this read. Is anyone able to explain how the
| backup.timer runs the backup.service? It wasn't obvious to me
| where the trigger was defined. I guess it's just inherent to what
| that unit type does, and the fact that both units are named
| backup? What is the name for that "package" of units that make up
| the whole backup program?
| bhaney wrote:
| > I guess it's just inherent to what that unit type does, and
| the fact that both units are named backup?
|
| Correct. If there's no `Unit=` specified in the timer unit, it
| defaults to the service unit of the same name. See
| https://www.freedesktop.org/software/systemd/man/latest/syst...
|
| This is also a common pattern for the socket units of socket
| activated services
| akeck wrote:
| I used systemd user timers, a shell script, and an HDHomerun to
| make a simple VCR.
| rs_rs_rs_rs_rs wrote:
| I use them all the time, my only wish is they get rid of the
| .service file and make it possible to define the service inside
| the .timer file.
| xyse53 wrote:
| I initially had a similar reaction, but there are so many cases
| like this for unit files. I think a higher level DSL for
| generating them would be useful.
| paulddraper wrote:
| The separation of concerns is legitimate.
| merpkz wrote:
| Minor nitpick - shouldn't you first define the service and only
| then a timer for it? Otherwise since you enabled timer and are
| still trying to figure out how to write service, systemd won't
| have anything to run when timer triggers. Maybe I am wrong, but
| that just feels like logical order. Anyways, after years on
| hating on systemd I also started to embrace it and porting my
| cron jobs to systemd timers and I must admit it's really nice,
| the overall overview with list-timers, next execution timestamp,
| total execution time, ordering of services so one can run after
| another is completed and of course the logging in journal so I
| can filter output and keep track of everything it's just
| wonderful experience.
|
| EDIT: yea, the email reporting is certainly missing, but it was
| hard to control it since whole STDOUT was shipped, which is not
| what I wanted most of the time anyways. It would be good to come
| up with some way to still have small overview emails sent about
| important jobs done, maybe a dependency service which starts when
| important job finished and just sends an email about that
| dmd wrote:
| I would love to see something like https://github.com/isd-
| project/isd but for systemd timers.
___________________________________________________________________
(page generated 2025-03-14 23:02 UTC)