Post B6dAKSs1zyZFFB3kcS by hyc@mastodon.social
(DIR) More posts by hyc@mastodon.social
(DIR) Post #B6cxSUoXMDBgBUvNSa by equinox@chaos.social
0 likes, 0 repeats
Cool.Updated Debian stable on our hacker ISP main DFZ routers.Check 'top' output a little later… systemd-networkd is in a restart loop continuously consuming an entire CPU core. (Of course none of the systemd restart-limit options are being applied.)Only relevant thing in the logs: "Could not enumerate links: Connection timed out"Oh, and, pretty sure (can't check ofc) systemd-network was disabled prior to upgrading.Quality engineering.
(DIR) Post #B6cxSV2MWoFKsNOQ0e by equinox@chaos.social
0 likes, 1 repeats
Ah yes. systemd-timesyncd and systemd-resolved have somehow been enabled as well, pretty sure those were disabled before too.Time for some strategic exorcism.
(DIR) Post #B6cxXmF5zDjHgYrhku by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@equinox can’t have systemd-timesyncd and systemd-resolved if the system is systemd-free 😸 like those couple of Debian systems I run.
(DIR) Post #B6cyMtN9Em8QZE0Nm4 by equinox@chaos.social
0 likes, 0 repeats
@mirabilos we're trying to hit a balance of minimum ongoing maintenance effort, and completely removing systemd I believe is notably more effort than keeping it quarantined.…this update experience, however, is making another argument…
(DIR) Post #B6cyMtX4ds4h40eJFI by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@equinox removing it is mostly once-only effort tho
(DIR) Post #B6cyRuvGttDmp7MWdk by jakob@mastodon.chaosfield.at
0 likes, 0 repeats
@equinox @mirabilos weird that jt got enabled during an upgrade. But networkd is not bad at all anymore (as with all poettering projects it took a while to settle before getting useable but imo we've reached that point ^^). Certainly better than ifupdown2 or netplan :P
(DIR) Post #B6cyRv5YHfRdL0AjfE by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@jakob @equinox eh no, it requires USERLAND.EXE and is being ensloppified
(DIR) Post #B6cyX3YGFJML2839Yu by equinox@chaos.social
0 likes, 0 repeats
@jakob @mirabilos considering the error message, best I can guess is that it's not good enough for a system with 1.25M routes and a separate management VRF.
(DIR) Post #B6cyX3itblrlZ71e8e by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@jakob @equinox it’s good enough for Poettering’s laptop; how dare you have different requirements!
(DIR) Post #B6cybrQmjmRgtEqOHY by equinox@chaos.social
0 likes, 0 repeats
@mirabilos yes and no. We're a bunch of people running this ISP, each of us with very little time, and removing systemd also means bringing everyone up to speed on how to deal with that.
(DIR) Post #B6cybrc83bWHSQ9Rxo by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@equinox but everyone knows how to run systems the classic way, which was default until only a few years ago
(DIR) Post #B6cz98sQtXMMYOe6IC by equinox@chaos.social
1 likes, 1 repeats
@mirabilos @jakob my primary complaint around the systemd ecosystem remains the incredibly poor observability. In this case: single error message and nothing to go on. It fails just like a Windows system, obtusely and leading you on a dig for actual error messages. And I don't know where to dig.It works as long as it works, if something goes wrong you're kinda fucked.I don't expect FRRouting levels of log messages, but how about maybe half of that?
(DIR) Post #B6czYyU29zitqdYEfw by linear@nya.social
0 likes, 1 repeats
@equinox@chaos.social @mirabilos@toot.mirbsd.org @jakob@mastodon.chaosfield.at this has been only one of many issues i've encountered over the years trying to use systemd on embedded systems. when i get the choice, i stick with a sysvinit-like and a real syslog daemon because then i can get actual useful information, and even central collection of logs from multiple devices.even android is better than systemd in this regard. you can deploy a fleet of special-purpose embedded android devices as a budding vendor and, with very little effort, get a centralized web dashboard of all logcat messages and be able to filter through all messages in the entire fleet and track down causes of issues without having to get a shell on a single device.
(DIR) Post #B6czeGwkwRaeYhVnyy by linear@nya.social
1 likes, 1 repeats
@equinox@chaos.social @mirabilos@toot.mirbsd.org @jakob@mastodon.chaosfield.at systemd also has all sorts of weird bugs in edge cases that make it entirely unsuitable for certain embedded use cases, and trying to report them upstream or provide sensible fixes only gets you a "you are doing it wrong, your use case is wrong and should not exist" from upstream
(DIR) Post #B6czs0Sl1bpzUO1sVk by linear@nya.social
0 likes, 0 repeats
@equinox@chaos.social @mirabilos@toot.mirbsd.org @jakob@mastodon.chaosfield.at thankfully i don't have to do that anymore at my current job
(DIR) Post #B6dAKSs1zyZFFB3kcS by hyc@mastodon.social
0 likes, 1 repeats
@linear @jakob @mirabilos @equinox ah yes, like my ever favorite "systemd-journald hangs due to firehose of syslog output from slapd" - You're Doing It Wrong, stop trying to call syslog so quickly.
(DIR) Post #B6ejs7VVclFeO1Ghoe by jakob@mastodon.chaosfield.at
0 likes, 0 repeats
@equinox @mirabilos you'll probably want to set```ManageForeignRoutingPolicyRules=noManageForeignRoutes=noManageForeignNextHops=no```in the networkd config. That way it doesn't touch any routes it did not set itself.With that set it handles multiple fulltables without any issues (as it doesn't care about them).
(DIR) Post #B6ejs7iyog1j3nZSoS by equinox@chaos.social
0 likes, 0 repeats
@jakob @mirabilos I mean, sure, but… why would I want to run systemd-networkd to begin with? Why would I want to run *any* daemon? It has nothing to do. This is a router, the only way the network config changes is if we tell it to.
(DIR) Post #B6ejs7tGCSFZZgNfpw by jakob@mastodon.chaosfield.at
0 likes, 0 repeats
@equinox @mirabilos because it is by far the best thing to set up the network? (also there are cases that require active intervention like interface state changes).Even if you don't need anything longrunning there are no good alternatives. Everything sucks to some degree but networkd sucks the least for static configs (for dynamic connections networkmanager is the best system imo).I really hope you are not writing sysv init scripts that run iproute2 commands :P
(DIR) Post #B6ejs84FXb2a7lWRxw by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@equinox @jakob why would interface state changes "require active intervention"?no, they sure as hell stay they way they are configured so things continue working when the state changes again
(DIR) Post #B6ezNmb6sWa9TjC8Js by jakob@mastodon.chaosfield.at
0 likes, 0 repeats
@mirabilos @equinox I've had many cases in the past starting with software binding to specific interfaces being unhappy about an interface going down and requiring a restart. I regularly have things that require intervention after interface state changes (also on routers/servers). Yes it probably should not be necessary. But it is.Also proper statekeeping is nice to have as it allows you to make changes without interruption and without manually applying the changes.
(DIR) Post #B6ezNmow37doAbfArw by equinox@chaos.social
0 likes, 0 repeats
@jakob @mirabilos the box is configured with ifupdown2, which is working perfectly fine. systemd-networkd has somehow gotten started and just failed immediately. Therefore, systemd-networkd is clearly not the best thing to configure the network…The only thing I can really relate with "software needs poking after state changes" is explicit binding to IP addresses. That's an admin decision, and when doing that the IP should be added to loopback. And addresses generally don't just disappear…
(DIR) Post #B6ezNn4t5oOwy57ujY by equinox@chaos.social
0 likes, 2 repeats
@jakob @mirabilos this thing about the network stack doing random things… no. It doesn't. It does things when told to, aside from bugs and design fuckups. It's not an unknowable black box.In most cases, it's userspace (e.g. DHCP) messing with things that then requires other userspace pieces to be fixed. Exceptions to this are IPv6 RA, proxy neigh and nexthop groups. The latter two are irrelevant for userspace that isn't routing related, and RAs can be dealt with (by not hardcoding addrs)
(DIR) Post #B6ezWgUovtXXA4Uu9Y by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@equinox @jakob oooooh yes, trixie’s dhcpcd-base just does IPv6 things despite being told not to, and it just breaks things; isc-dhcp-client at least was v4-only…(IPv6 works perfectly fine with the kernel’s built-in rtsol, TYVM)
(DIR) Post #B6f9P3tj2bmwxv9Oc4 by tknarr@mstdn.social
0 likes, 0 repeats
@equinox @jakob @mirabilos I've always wondered how much of that is someone finding a new need, not wanting to wait for an official solution so they out in a hack to do it, then when the real solution arrives the hack doesn't get removed?
(DIR) Post #B6f9P4M5LATQNsQ2oi by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@jakob @equinox @tknarr and the hack is "modern" and "minimal"… and the real solution "uses legacy tech"…