[HN Gopher] Running systemd without systemd-journald
___________________________________________________________________
Running systemd without systemd-journald
Author : axy
Score : 59 points
Date : 2022-06-19 07:30 UTC (15 hours ago)
(HTM) web link (declassed.art)
(TXT) w3m dump (declassed.art)
| saurik wrote:
| Would doing something like this work around the "journald drops
| the most important error messages" issue that has been
| known/outstanding for ten years (bug moved to GitHub six years
| ago), or is that more of a fundamental design mistake in systemd
| itself?
|
| https://github.com/systemd/systemd/issues/2913
|
| https://bugs.freedesktop.org/show_bug.cgi?id=50184
| jiripospisil wrote:
| It's not accurate to say it "drops" error messages. The bug
| causes these messages to not be attributed to a particular unit
| - you can still see them with `journalctl` but not with
| `journalctl -u foo`. Still pretty annoying and should
| absolutely be fixed (although I'm not sure if systemd is the
| right place to do it).
| puffoflogic wrote:
| > although I'm not sure if systemd is the right place to do
| it
|
| It's truly impressive how systemd turns out the be the right
| place to do absolutely anything and everything from
| bootloader to init to dhcp to ntp to network shares...
| _except_ fix systemd bugs. systemd doesn 't seem to be the
| right place to do that ever. Someone else needs to do it. For
| another example of this phenomenon, see the nohup bug.
| saurik wrote:
| This is actually _extremely useful_ for me to know, so thank
| you so much for pointing this out!
| andrewstuart wrote:
| systemd is incredibly useful and powerful.
|
| If you are a developer on Linux then you really owe it to
| yourself to learn as much as you can about systemd.
|
| If you really understand systemd then you'll find yourself
| architecting your software around its capabilities.
|
| systemd, if you understand it, can mean you can completely avoid
| large chunks of development you might otherwise have assumed you
| need to do.
|
| socket activated services, nspawn, traffic accounting, the list
| of juicy goodness goes on and on....
|
| Ignore the haters, they wouldn't hate if they dedicated their
| energy to understanding systemd instead of hating on it.
|
| In 2022 you are a really an incomplete full stack developer if
| systemd is not one of the technologies you know very well.
| ghoward wrote:
| systemd was the first init/supervision system I used. I was
| able to understand how to use it pretty well.
|
| I still hate it.
|
| Sorry, but understanding systemd does not preclude hating it.
|
| In fact, it's because of the "haters" that I decided to dive
| deep into what init/supervision systems are and can be. Without
| that deep dive, I would have always thought that systemd was
| great. Afterward, however, I know just how wrong that is.
|
| If a developer's only interaction with init systems was SysV
| init, yes, of course, systemd is the greatest thing since
| sliced bread.
|
| But systemd could have been so much better...
|
| Disclaimer: I'm writing an init/supervision system to be so
| much better. And simpler. Orders of magnitude simpler. Oh, and
| one that doesn't reach its fingers into every part of your OS.
|
| tl;dr: systemd is better than SysV init, but it's not the end-
| all-be-all of init/supervision systems.
| coredog64 wrote:
| > systemd is incredibly useful and powerful.
|
| In that one sentence I think you've highlighted what garners
| hate from the haters. It's powerful, and it's great when that
| power is used for good. Those times when it's not is where
| people get agitated.
| qbasic_forever wrote:
| Do you have an example of when systemd is not used for good?
| egberts1 wrote:
| it kills production daemons having long running stateful
| data whenever the netdev goes offline., systemd-networkd,
| that is.
| viraptor wrote:
| Only if you configure it to do that. You can use Wants=
| and After= for your service to start it after the
| network, but not restart otherwise.
|
| The behaviour you describe is not unreasonable to want:
| If your network device goes away, what exactly are you
| binding the socket to? How are you restoring the
| listening when it comes back up? But it's up to you to
| say which one you want.
| houzi wrote:
| It's quite annoying to not be able to use common tools for
| DNS queries like `dig`, when using `systemd-resolved` for
| DNS. I think you might have to sometimes flush the caching
| feature of `systemd-resolved` as well.
|
| It's fine, I guess. But it did take a while to learn the
| new ways.
|
| I think this is the main complaint from users with decades
| of experience. Their scripts and old knowledge stops
| working.
| [deleted]
| midislack wrote:
| Systemd is absolute GARBAGE, I stopped using Linux as it subsumed
| more and more.
| AnonymousPlanet wrote:
| What are you using instead?
| [deleted]
| frankharv wrote:
| I have but one word for this guy.
|
| Devuan
|
| I see no reason not to consider it.
|
| He is a Debian user who hates the way systemd works.
|
| Devuan is for you.
| Arch-TK wrote:
| I recently got a new work laptop and decided to take the
| opportunity to switch from using windows 10 with a linux VM
| (please don't try to sell me WSL2, I'm not interested) to plain
| linux. I decided to pick devuan stable since the base machine
| needs to be... stable. I have no interest in using systemd on
| anything which needs to be stable. I feel happy for anyone who
| has never encountered severe stability issues with systemd but
| I am not that person. Devuan comes with sysvinit which is also
| trash but it offers the opportunity to use other inits. At this
| point in time I have switched it from sysvinit to runit and
| eventually ripped out the entire runit infrastructure that
| comes with devuan and replaced it with something heavily
| inspired by void's runit. This isn't great, it would be nice if
| devuan took things other than sysvinit+initscripts more
| seriously. Maybe even just switch to OpenRC since, while it is
| still not great (please stop using pid files) it's a hell of a
| lot more sane than the mess of init scripts devuan ships with.
| seclorum_wien wrote:
| Three cheers for Devuan! :). Completely agree, this would've
| been an easy distro switch-a-rooni ..
| egberts1 wrote:
| I switched all my Debian to Devuan. Havent looked back.
| egberts1 wrote:
| if you run a Debian-based production server, then Devuan is for
| you.
|
| Devuan can allow your precious daemon to stay up despite netdev
| going offline, unlike systemd-networkd which would kill the
| daemon.
|
| This is quite important if your large ling-running daemon has
| statefulness data.
| dralley wrote:
| As someone pointed out elsewhere in the thread, so can
| systemd, your service did what it was configured to do. If
| you want the service to stay up when netdev goes down, then
| don't tell systemd that netdev is a hard requirement for your
| service.
| egberts1 wrote:
| Its never a good systemd-networkd default to be killing
| daemon whenever a netdev goes " _cricket_ ".
| TacticalCoder wrote:
| It depends what you're doing but for my "workstation", Devuan
| is indeed working flawlessly. I'm running it since six months
| or so on my main PC (a little Ryzen 3700X / 32 GB or RAM) and
| I'm very happy with it.
|
| Now if you're a sysadmin and have come to rely on systemd and
| are now locked in, Devuan is obviously not for you. But if
| you're running Linux on a desktop or on a laptop and aren't a
| fan of systemd, Devuan is great.
| dark-star wrote:
| He doesn't even know how to read man pages (journald.conf(5)
| describes the setup he is looking for), and he doesn't know how
| memory usage on Linux works (shared /mmap()ed memory is counted
| for each process). Do you really think his issues go away when
| he switches to Devuan? He'll probably just yell at different
| clouds
|
| Other than that, Devuan is a solid choice for people who want
| to get rid of systemd. It comes with the Debian-typical rather
| old versions of most programs, but I guess for a file server it
| doesn't matter much if you run kernel 4.19 or 5.15
| Arch-TK wrote:
| The systemd man pages are a book. And, just for avoidance of
| doubt, that's not a good thing. I am never surprised when
| someone can't find out how to configure systemd to do what
| they want. It's just too enterprise grade.
| enriquto wrote:
| Spam omelette with spam sausage and spam does not have too much
| spam in it!
| rob_c wrote:
| Can we just admit jourald is a thorn in the side of people taking
| systems seriously. The default behaviour even on polished distros
| is just bad and it's mainly because of the mindsets behind it...
| pengaru wrote:
| PSA: systemd-journald uses _shared_ file-backed mappings via
| mmap() for its journal IO.
|
| You must subtract its shared memory use from its resident memory
| use before judging how much memory it's consuming. The file-
| backed shared mappings are _reclaimable_ , because they are file-
| backed. The kernel will just evict the mapped journal pages at
| will, since they can always be faulted back in from the
| filesystem.
|
| TFA is much ado about nothing, learn to measure memory use
| properly before breaking out the pitch forks.
|
| Full disclosure: I've hacked a bunch on journald upstream.
| bragr wrote:
| I was wondering about this exact thing when the article didn't
| break down the ram usage. Thanks!
| viraptor wrote:
| I was waiting for the author to check the memory usage of
| rsyslog and becoming enlightened... but it didn't happen.
| Reminder: check your assumptions/result after changes. He
| could learn that rsyslog uses way more shared memory than
| journald (>900M on my system) and it doesn't matter.
| jeroenhd wrote:
| This is true, but the author seems to be running their services
| on several Raspberry Pi like devices whose flash storage may be
| unstable or quick to wear out. Eliminating unnecessary writes
| and swap space (depending on the application), those megabytes
| of extra memory may be just enough what tricks the system into
| committing memory into swap.
|
| You can run quite a lot in 512MB of RAM if you use the right
| languages to write code in. I was surprised about how little
| RAM my moderately complex daemon written in Rust uses, for
| example; I expected to have to allocate a gigabyte of RAM to
| the VM running it (based on what other tools similar to what I
| was doing needed) but the entire system turned out to be quite
| comfortable with just a quarter of that. I didn't even try to
| optimise for memory usage, which is what made this so
| surprising. I stil had to give it some more RAM because
| unattended upgrades tended to get stuck, but I learned a lesson
| that day.
|
| Ever since I've been meaning to try to mess with Firecracker +
| bare bones daemons to run virtual machines services with
| absolutely minimal overhead. I like the virtualisation
| boundaries from a security standpoint much more than container
| boundaries and now I wonder how much I can shrink my overhead
| by.
| Redoubts wrote:
| > This is true, but the author seems to be running their
| services on several Raspberry Pi like devices whose flash
| storage may be unstable or quick to wear out. Eliminating
| unnecessary writes and swap space (depending on the
| application), those megabytes of extra memory may be just
| enough what tricks the system into committing memory into
| swap.
|
| Well the author seems to want text logs instead, which seems
| much much worse for this.
| pengaru wrote:
| If you're concerned about storage wear you'd just run
| journald without /var/log/journal so it's volatile (tmpfs)
| only. At least that way you still have journals for your
| current boot and functionality like `systemctl status
| $service` can still tell you some journal information.
| Spivak wrote:
| Yeah this is a lot of work to avoid reading journald.conf,
| switching the storage to volatile, and capping the memory
| usage to whatever you want.
| seclorum_wien wrote:
| >You can run quite a lot in 512MB of RAM if you use the right
| languages to write code in.
|
| I recently delivered a production-ready embedded system
| running Armbian with 512megs RAM, and indeed disabled
| systemd-journald for our uses, also .. but even with it
| enabled, our Lua-based app was (science/data analysis on
| sensor network) running in the best environment it has ever
| run, so I can confirm: 512MB is enough for a lot of things.
| RedShift1 wrote:
| Windows 95 ran an entire OS with decent UI in 8 MB of RAM.
| One really has to wonder, where is all the RAM going these
| days? I think the knowledge of doing anything with only 8
| MB of RAM has gone away, we don't know how to do it
| anymore.
| HL33tibCe7 wrote:
| Your comment brings the following Monty Python sketch to
| mind:
|
| "What did the Romans ever do for us?
|
| ... All right, but apart from the sanitation, the
| medicine, education, wine, public order, irrigation,
| roads, a fresh water system, and public health"
|
| Replace "Romans" with "Increased RAM usage".
| mid-kid wrote:
| It's not the knowledge - it's the increased complexity of
| the entire stack, all the way down to the hardware. A
| modern linux kernel image is easily bigger than 8MB, and
| that needs to be in memory at all times. Why? Because of
| all the functionality it has these days, to fit all the
| possible usecases people need. Windows 95 didn't have
| Swap, didn't support many filesystems, didn't have
| central logging, didn't have ASLR, let alone support for
| containers, and many other features I'm forgetting along
| the way.
|
| Sure you could strip away a lot of that functionality,
| even at the distribution level (by for example not using
| an init system at all, instead just one shell script to
| initialize things), but then you'd end up with an
| operating system that's not general purpose for today's
| standards anymore.
| rasz wrote:
| > Windows 95 didn't have Swap
|
| it did have virtual Memory and swap
| jacquesm wrote:
| 512MB is absolute overkill for the application that you
| built, it is the choice of OS + the tooling used that
| resulted in that requirement. Not all that long ago 32 MB
| served a whole bank, and embedded systems used kilobytes of
| RAM, not megabytes. We've gotten so used to slapping a full
| unix server into stuff that we hardly even think about it
| any more and just take that kind of power completely for
| granted. I'm not saying you made any wrong choices, it's
| just that most of the embedded stuff that I come across
| would be just as feasible on a fraction of the CPU (and
| power) budget than what we typically choose because for
| instance Lua is such a convenient choice for a platform
| like that.
| axy wrote:
| Memory use has always been a mystery to me and I can easily
| miss some things. Thanks for pointing to. Anyway, the right
| solution for me is tldr, all the rest is shit.
| iforgotpassword wrote:
| Yeah, the rest just shows that op likes to do things like
| they've always done it. How you can prefer to poke around
| syslog and ps output to determine the state of a service
| instead if just doing systemctl status is beyond me for
| example.
| hedora wrote:
| Did the Linux kernel ever fix the thing where it evicts code
| pages at the same priority as files mapped read/write?
|
| I haven't checked in the last 4 years or so, but, before that,
| every time I've worked with a Linux-based storage system that
| used mmap to write to files, I've ended up rewriting it to use
| pread/pwrite.
|
| Each time, there was no perceptible CPU hit, but there was a
| massive page cache / memory pressure win. It turns out that
| aggressively evicting warm code pages then faulting them back
| in is bad for system performance, even with a fast SSD.
| jcalvinowens wrote:
| There's nothing to "fix" here, in some cases what you want is
| not optimal. It is perfectly reasonable for the kernel to
| prioritize data pages you touched more recently than code
| pages by default. It's essentially a big LRU, always has
| been.
|
| If you don't like that, you can always use mlock(). You can
| also tune things like writeback sysctls and readahead
| behavior. But I disagree it's "broken" because it doesn't do
| what you want by default.
| quotemstr wrote:
| It's not quite that simple: while clean file-backed pages are
| _cheaper_ than, say, private dirty pages (which the kernel must
| preserve as long as anyone references them), they 're not
| _free_ : you're still paying an opportunity cost. That is, the
| kernel is, at least for a time, keeping each clean file-backed
| page resident when it could be keeping some other page, perhaps
| a more useful one, in RAM instead. If systemd-journald is
| append-mostly, it'd be useful to MADV_FREE (after msync) any
| pages behind the current write pointer so as to give the kernel
| a hint that it can get rid of those clean file-backed pages
| early. I'd actually suggest getting rid of the use of memory
| mapping entirely, but doing so would likely be a bigger ask.
| viraptor wrote:
| > the kernel is, at least for a time, keeping each clean
| file-backed page resident when it could be keeping some other
| page
|
| It's almost the same result for the standard page cache when
| you're reading a file, isn't it?
| db48x wrote:
| Weird; the systemd journal is the feature I want most! It would
| be the last thing I would ever consider disabling.
| MrStonedOne wrote:
| heretogetout wrote:
| I don't know if this is still an issue but the last time I used
| journald the logs would occasionally become corrupted and
| journalctl would refuse to read them. The fix was to just
| delete the logs. I have no idea how logging got so screwed up
| that corruption in part of the file could make the rest of the
| log file unreadable. I mean, it's a journal, it's right in the
| name.
|
| Ever since then I switched to rsyslogd and the like. Rock
| solid.
| db48x wrote:
| Honestly, that sounds more like a disk problem than a systemd
| problem.
| viraptor wrote:
| Keep in mind that rsyslog doesn't even attempt to verify
| logs. An alternative explanation is: my system is corrupting
| logs, I changed to a logging daemon which doesn't tell me
| about it.
|
| I mean, there could definitely be a bug in journald, but I
| haven't seen any fixes mentioned in changelog for the last 5
| years and if it was happening in standard usage, people would
| notice.
|
| For recovering corrupted logs - you can still "less" them as
| usual. They have some extra markers, but the text is
| available as text. Journalctl has some special options for
| that too.
| heretogetout wrote:
| IIRC when I experienced this the logs were all in some
| binary format that couldn't easily be less'd. tbh I didn't
| do much investigation other than to see the "delete all
| your logs" resolution suggestion. There could have been a
| better option.
| lokar wrote:
| The fact that it can't seem to recover from a few bad
| records and gives up on the whole file demonstrates what
| terrible software it is.
| viraptor wrote:
| Have you tried it? Journalctl does skip bad entries and
| prints out the rest automatically. If you've found a case
| where it doesn't, you should report that as a bug.
___________________________________________________________________
(page generated 2022-06-19 23:01 UTC)