[HN Gopher] Go is portable, until it isn't
___________________________________________________________________
Go is portable, until it isn't
Author : khazit
Score : 139 points
Date : 2025-12-07 11:17 UTC (6 days ago)
(HTM) web link (simpleobservability.com)
(TXT) w3m dump (simpleobservability.com)
| necovek wrote:
| This seems to imply that Go's binaries are otherwise compatible
| with multiple platforms like amd64 and arm64, other than the
| issue with linking dynamic libraries.
|
| I suspect that's not true either even if it might be technically
| possible to achieve it through some trickery (and why not risc-v,
| and other architectures too?).
| khazit wrote:
| Of course you still need one binary per CPU architecture. But
| when you rely on a dynamic link, you need to build from the
| same architecture as the target system. At that point cross-
| compiling stops being reliable.
| necovek wrote:
| I am complaining about the language (phrasing) used: a
| Python, TypeScript or Java program might be truly portable
| across architectures too.
|
| Since architectures are only brought up in relation to
| dynamic libraries, it implied it is otherwise as portable as
| above languages.
|
| With that out of the way, it seems like a small thing for the
| Go build system if it's already doing cross compilation (and
| thus has understanding of foreign architectures and
| executable formats). I am guessing it just hasn't been done
| and is not a big lift, so perhaps look into it yourself?
| arccy wrote:
| they're only portable if you don't count the architecture
| specific runtime that you need to somehow obtain...
|
| go doesn't require dynamic linking for C, if you can figure
| out the right C compiler flags you can cross compile
| statically linked go+c binaries as well.
| swills wrote:
| I happily and reliably cross build Go code that uses CGO and
| generate static binaries on amd64 for arm64.
| vbezhenar wrote:
| Is it some tooling issue? Why is is an issue to cross-compile
| programs with dynamic linking?
| dekhn wrote:
| In general, cross compilers can do dynamic linking.
| spijdar wrote:
| In my experience, the cross-compiler will refuse to link
| against shared libraries that "don't exist", which they
| usually don't in a cross compiler setup (e.g. cross
| compiling an aarch64 application that uses SDL on a
| ppc64le host with ppc64le SDL libraries)
|
| The usual workaround, I think, is to use dlopen/dlsym
| from within the program. This is how the Nim language
| handles libraries in the general case: at compile time, C
| imports are converted into a block of dlopen/dl* calls,
| with compiler options for indicating some (or all)
| libraries should be passed to the linker instead, either
| for static or dynamic linking.
|
| Alternatively I think you could "trick" the linker with a
| stub library just containing the symbol names it wants,
| but never tried that.
| dwattttt wrote:
| You just need a compiler & linker that understand the
| target + image format, and a sysroot for the target. I've
| cross compiled from Linux x86 clang/lld to macOS arm64,
| all it took was the target SDK & a couple of env vars.
|
| Clang knows C, lld knows macho, and the SDK knows the
| target libraries.
| cxr wrote:
| It's a tooling issue. No one has done the work to make
| things work as smoothly as they could.
|
| Traditionally, cross-compilers generally didn't even work
| the way that the Zig and Go toolchains approach it--
| achieving cross-compilation could be expected to be a much
| more trying process. The Zig folks and the Go folks broke
| with tradition by choosing to architect their compilers
| more sensibly for the 21st century, but the effects of the
| older convention remains.
| cxr wrote:
| For a single binary that will actually run across both
| architectures, see <https://cosmo.zip/>.
|
| Original discussion:
| <https://news.ycombinator.com/item?id=24256883>.
| daviddever23box wrote:
| This is an (organizational) tooling problem, not a language
| problem - and is no less complicated when musl libc enters the
| discussion.
| laladrik wrote:
| The conclusion of the article says that it's not the language
| problem either. Under the title "So, is Go the problem?" Or do
| you mean something else here?
| saghm wrote:
| Given that the title implies the opposite, I think it's a
| fair criticism. Pointing out clickbait might be tedious, but
| not more so than clickbait itself.
| bitbasher wrote:
| Once you use CGO, portability is gone. Your binary is no longer
| staticly compiled.
|
| This can happen subtley without you knowing it. If you use a
| function in the standard library that happens to call into a CGO
| function, you are no longer static.
|
| This happens with things like os.UserHomeDir or some networking
| things like DNS lookups.
|
| You can "force" go to do static compiling by disabling CGO, but
| that means you can't use _any_ CGO. Which may not work if you
| require it for certain things like sqlite.
| swills wrote:
| You can definitely use CGO and still build statically, but you
| do need to set ldflags to include -static.
| tptacek wrote:
| You can even cross-compile doing that.
| swills wrote:
| Yes, indeed, I do.
| ncruces wrote:
| There are at least a couple of ways to run SQLite without CGO.
| tptacek wrote:
| I think the standard answer here is modernc.org/sqlite.
| apitman wrote:
| Careful, you're responding to the author of a wasm-based
| alternative.
| ncruces wrote:
| No need to be careful. I won't bite. ;)
| silverwind wrote:
| > This happens with things like os.UserHomeDir or some
| networking things like DNS lookups.
|
| The docs do not mention this CGO dependency, are you sure?
|
| https://pkg.go.dev/os#UserHomeDir
| purpleidea wrote:
| I was surprised too, that I had to check the docs, so I
| assume the user was misinformed.
| bitbasher wrote:
| Perhaps I misremembered or things changed? For instance,
| the os/user results in a dynamicly linked executable:
| https://play.golang.com/p/7QsmcjJI4H5
|
| There are multiple standard library functions that do it..
| I recall some in "net" and some in "os".
| telotortium wrote:
| os.UserHomeDir is specified to read the HOME environment
| variable, so it doesn't require CGo. os/user does, but
| only to support NSS and LDAP, which are provided by libc.
| That's also why net requires CGo- for getaddrinfo using
| resolv.conf
| PunchyHamster wrote:
| > Which may not work if you require it for certain things like
| sqlite.
|
| there is cgo-less sqlite implementation
| https://github.com/glebarez/go-sqlite it seems to not be
| maintained much tho
| IceWreck wrote:
| You're linking to a different version - this is the one that
| most people use https://github.com/modernc-org/sqlite
| debugnik wrote:
| Yes and no, the package above is a popular `database/sql`
| driver for the same SQLite port you linked.
| hiAndrewQuinn wrote:
| You don't need CGO for SQLite in most cases; I did a deep dive
| into it here.
|
| https://til.andrew-quinn.me/posts/you-don-t-need-cgo-to-use-...
| mmulet wrote:
| I ran into this issue when porting term.everything[0] from
| typescript to go. I had some c library dependencies that I did
| need to link, so I had to use cgo. My solution was to do the
| build process on alpine linux[1] and use static linking[2]. This
| way it statically links musl libc, which is much friendlier with
| static linking than glibc. Now, I have a static binary that runs
| in alpine, Debian, and even bare containers.
|
| Since I have made the change, I have not had anyone open any
| issues saying they had problems running it on their machines.
| (Unlike when I was using AppImages, which caused much more
| trouble than I expected)
|
| [0] https://github.com/mmulet/term.everything look at
| distribute.sh and the makefile to see how I did it.
|
| [1]in a podman or docker container
|
| [2] -ldflags '-extldflags "-static"'
| cxr wrote:
| I didn't see an explanation in the README that part of what the
| first GIF[1] shows is an effect created by video editing
| software (and not a screencapture that's just demonstrating the
| program actually running). "Screen images simulated" are the
| words usually chosen to start off the disclaimers in fine print
| shown at the bottom of the screen when similar effects appear
| in commercials. I think that it would make sense to adopt a
| similar explanation wrt the effect used for the GIF.
|
| 1. <https://github.com/mmulet/term.everything/blob/main/resourc
| e...>
| plufz wrote:
| Why would an open source project need to have any disclaimer?
| They are not selling anything.
| nofriend wrote:
| Because lying is wrong even when open source projects do
| it.
| plufz wrote:
| I think it is a big stretch calling this visual effect
| lying.
|
| I don't know if it is a cultural American thing or just
| difference in interpretation but I had no difficulty
| understanding that this was a visual effect. But in my
| country ads don't come with disclaimers. Do you feel like
| these disclaimers are truly helpful?
| cxr wrote:
| I don't feel that the person I responded to is lying or
| being intentionally deceptive.
| nebezb wrote:
| > "in commercials where such effects appear"
|
| Good thing this isn't a commercial then.
| jchw wrote:
| IMO this is the best approach, but it is worth noting that musl
| libc is not without its caveats. I'd say for most people it is
| best to tread carefully and make sure that differences between
| musl libc and glibc don't cause additional problems for the
| libraries you are linking to.
|
| There is a decent list of known functional differences on the
| musl libc wiki:
|
| https://wiki.musl-libc.org/functional-differences-from-glibc...
|
| Overall, though, the vast majority of software works perfectly
| or near perfectly on musl libc, and that makes this a very
| compelling option indeed, especially since statically linking
| glibc is not supported and basically does not work. (And
| obviously, if you're already using library packages that are
| packaged for Alpine Linux in the first place, they will likely
| already have been tested on musl libc, and possibly even
| patched for better compatibility.)
| apitman wrote:
| Note that you don't have to compile on an Alpine system to
| achieve this. These instructions should work on most distros:
|
| https://www.arp242.net/static-go.html
| tasuki wrote:
| Huh. Does term.everything just work, or are there some gotchas?
| This seems like it could be supremely useful!
| mmulet wrote:
| It works so far! No major gotchas that I know of yet. From
| the perspective of the apps, they are just talking to a
| normal Wayland compositor, so everything works as expected.
| Just try it for your workflow, and if you run into any
| problems just open an issue and I'll fix it.
| nickcw wrote:
| That is a nice approach. I'll have to give that a try with
| rclone. I tried lots of things in the past but not using Alpine
| which is a great idea
|
| Another alternative is
|
| https://github.com/ebitengine/purego
|
| You can use this to dynamic load shared objects / DLLs so in
| the OP example they could disable systemd support if the
| systemd shared object did not load.
|
| This technique is used in the cgofuse library (
| https://github.com/winfsp/cgofuse ) rclone uses which means
| rclone can run even if you don't have libfuse/winfsp installed.
| However the rclone mount subcommand won't work.
|
| The purego lib generalizes this idea. I haven't got round to
| trying this yet but it looks very promising.
| kokada wrote:
| I am using purego indirectly in two pet projects of mine.
| While it has its own issues it definitely solves the issue of
| cross-compilation.
|
| In this particular case it may be that they will need to
| write a wrapper to abstract differences between the systemd C
| API if it is not stable, but at least they still can compile
| a binary from macOS to Linux without issues.
|
| The other issue as other said is to use journalctl and just
| parse the JSON format. Very likely that this would be way
| more stable, but not sure if it is performant enough.
| nly wrote:
| > and even bare containers.
|
| Strange, i thought the whole point of containers was to solve
| this problem.
| jdub wrote:
| Depends how much you care about the size and security
| footprint of your container images.
| johnisgood wrote:
| I use `-ldflags '-extldflags "-static"` as well.
|
| From the .go file, you just do `// #cgo LDFLAGS: -L. -lfoo`.
|
| You definitely do not need Alpine Linux for this. I have done
| this on Arch Linux. I believe I did not even need musl libc for
| this, but I potentially could have used it.
|
| I did not think I was doing something revolutionary!
|
| In fact, let me show you a snippet of my build script:
| # Build the Go project with the static library if go
| build -o $PROG_NAME -ldflags '-extldflags "-static"'; then
| echo "Go project built with static library linkage" else
| echo "Error: Failed to build the Go project with static
| library" exit 1 fi # Check if the
| executable is statically linked if nm ./$PROG_NAME | grep
| -q "U "; then echo "Error: The generated executable is
| dynamically linked" exit 1 else echo
| "Successfully built and verified static executable
| '$PROG_NAME'" fi
|
| And like I said, the .go file in question has this:
| // #cgo LDFLAGS: -L. -lfoo
|
| It works perfectly, and should work on any Linux distribution.
| mmulet wrote:
| I use alpine for this [1] reason, but I will admit that this
| is a premature-optimization. I haven't actually ran into the
| problem myself.
|
| ----
|
| Your code is great, I do basically the same thing (great
| minds think alike!). The only thing I want to add is that cgo
| supports pkg-config directly [2] via // #cgo
| pkg-config: $lib
|
| So you don't have to pass in linker flags manually. It's
| incredibly convenient.
|
| [1]https://stackoverflow.com/questions/57476533/why-is-
| statical...
|
| [2]https://github.com/mmulet/term.everything/blob/def8c93a3db
| 25...
| imcritic wrote:
| What troubles did you have with AppImages?
| mmulet wrote:
| List of troubles:
|
| [1]https://github.com/mmulet/term.everything/issues/28
|
| [2]https://github.com/mmulet/term.everything/issues/18
| (although this issue later gets sidetracked to a build issue)
|
| [3]https://github.com/mmulet/term.everything/issues/14
|
| [4]https://github.com/mmulet/term.everything/issues/7
| pansa2 wrote:
| > _do the build process on alpine linux and [...] statically
| link musl libc_
|
| IIRC it used to be common to do builds on an old version of
| RHEL or CentOS and dynamically link an old version of glibc.
| Binaries would then work on newer systems because glibc is
| backwards compatible.
|
| Does anyone still use that approach?
| Xylakant wrote:
| If you need glibc for any kind of reason, that approach is
| still used. But that won't save you if no glibc is available.
| And since the folks here want to produce a musl build anyways
| for alpine, the easier approach is to just go for musl all
| the way.
| kccqzy wrote:
| Interesting that it uses the C API to collect journals. I
| would've thought to just invoke journalctl CLI. On platforms like
| macOS where the CLI doesn't exist it's an error when you exec,
| not a build time error.
| ajross wrote:
| That's really not such a weird choice. The systemd library is
| pervasive and compatible.
|
| The weird bit is the analysis[1], which complains that a Go
| binary doesn't run on Alpine Linux, a system which is
| _explicitly and intentionally_ (also IMHO ridiculously, but
| that 's editorializing) binary-incompatible with the stable
| Linux C ABI as it's existed for almost three decades now. It's
| really no more "Linux" than is Android, for the same reason,
| and you don't complain that your Go binaries don't run there.
|
| [1] I'll just skip without explaination how weird it was to see
| the author complain that the build breaks because they can't
| get systemd log output on... a mac.
| khazit wrote:
| The macOS bit wasn't about trying to get systemd logs on mac.
| The issue was that the build itself fails because libsystemd-
| dev isn't available. We (naively) expected journal support to
| be something that we can detect and handle at runtime.
| ajross wrote:
| Well... yeah. It's a Linux API for a Linux feature only
| available on Linux systems. If you use a platform-specific
| API on a multiplatform project, the portability work falls
| on you. Do you expect to be able to run your Swift UI on
| Windows? Same thing!
| amiga386 wrote:
| That's also what gopsutils does, IIRC: it tries to look up
| process information with kernel APIs but can fall back to
| invoking /usr/bin/ps (which is setuid root on most systems) at
| the cost of being much less performant.
| LtWorf wrote:
| I did this a while ago but it only reads journal files
| sequentially and I didn't implement the needed stuff to use the
| indexes.
|
| https://github.com/appgate/journaldreader
| nunez wrote:
| You hit this real quick when trying to build container images
| from the scratch. Theoretically you can drop a Go binary into a
| blank rootfs and it will run. This works most of the time, but
| anything that depends on Go's Postgres client requires libpq
| which requires libc. Queue EFILE runtime errors after running the
| container.
| nateb2022 wrote:
| > anything that depends on Go's Postgres client requires libpq
| which requires libc
|
| Try https://github.com/lib/pq
| AlbinoDrought wrote:
| I've also seen https://github.com/jackc/pgx used in many
| projects
| mxey wrote:
| > For users that require new features or reliable resolution
| of reported bugs, we recommend using pgx which is under
| active development.
| ghola2k5 wrote:
| I've had some success using Zig for cross compiling when CGO is
| required.
| hansvm wrote:
| That's Uber's approach, right?
| sgt wrote:
| Is Uber using Zig for other things by now?
| hansvm wrote:
| I haven't heard.
| dilyevsky wrote:
| There's still some bugs when interacting with gold and cross-
| compiling to linux/arm64 but fixable with some workarounds...
| jen20 wrote:
| From the article:
|
| > In the observability world, if you're building an agent for
| metrics and logs, you're probably writing it in Go.
|
| I'm pretty unconvinced that this is the case unless you happen to
| be on the CNCF train. Personally I'd write in Rust these days, C
| used to be very common too.
| cosmin800 wrote:
| Well, that was pretty obvious that the portability is gone,
| especially when you start linking into systemd, even on the host
| system you have to link with the shared libs into systemd, you
| cannot link statically.
| CGamesPlay wrote:
| Use dlopen? I haven't tried this in Go, but if you want a binary
| that optionally includes features from an external library, you
| want to use dlopen to load it.
| immibis wrote:
| It only works in a dynamically-linked binary, because the
| dynamic linker needs to be loaded.
| nurettin wrote:
| Go is portable until you have to deploy on AS/400
| cyberax wrote:
| Cgo is terrible, but if you just want some simple C calls from a
| library, you can use https://github.com/ebitengine/purego to
| generate the bindings.
|
| It is a bit cursed, but works pretty well. I'm using it in my
| hardware-backed KMIP server to interface with PKCS11.
| pjmlp wrote:
| And a set of people rediscovered why cross compiling only works
| up to certain extent, regardless of the marketing on the tin.
|
| The point one needs to touch APIs that only exists on the target
| system, the fun starts, regardless of the programming language.
|
| Go, Zig, whatever.
| dwattttt wrote:
| You're thinking of cross platform codebases. There's nothing
| about cross compilation that stops the toolchain from knowing
| what APIs are present & not present on a target system.
| pjmlp wrote:
| Cross compilation and cross platform are synonymous in
| compiled languages, in regards of many issues that one needs
| to care about.
|
| Cross platform goes beyond in regards to UI, direction
| locations, user interactions,...
|
| Yeah, if you happen to have systemd Linux libraries on macOS
| to facilitate cross compilation into a compatible GNU/Linux
| system than it works, that is how embedded development has
| worked for ages.
|
| What doesn't work is pretending that isn't something to care
| about.
| IshKebab wrote:
| > Cross compilation and cross platform are synonymous in
| compiled languages
|
| Err, no. Cross-platform means the code can be compiled
| _natively_ on each platform. Cross-compilation is when you
| compile the binaries on one platform for a different
| platform.
| pjmlp wrote:
| Not at all, cross platform means executing the same
| application in many platforms, regardless of the hardware
| and OS specific features of each platform.
|
| Cross-compilation is useless if you don't actually get to
| executed the created binaries in the target platform.
|
| Now, how do you intend to compile from GNU/Linux into
| z/OS, so that we can execute the generated binary out
| from the C compiler ingesting the code written in
| GNU/Linux platform, in the z/OS language environment
| inside an enclave, not configured in POSIX mode?
|
| Using z/OS, if you're feeling more modern, it can be UWP
| sandboxed application with identity in Windows.
| IshKebab wrote:
| > cross platform means executing the same application in
| many platforms, regardless of the hardware and OS
| specific features of each platform.
|
| That is a better definition yes. But it's still not
| synonymous with cross-compilation, obviously. Most cross-
| platform apps are _not_ cross-compiled because it 's
| usually such a pain.
| t43562 wrote:
| Systemd. Binary logs are wonderful aren't they?
| LtWorf wrote:
| It's not that hard to read them without linking their library.
| The format is explained on their documentation.
|
| https://github.com/appgate/journaldreader
| 0xbadcafebee wrote:
| There's no such thing as a portable application; only programs
| limited enough to be lucky not to conflict with the vagaries of
| different systems.
|
| That said, in my personal experience, the most portable programs
| tend to be written in either Perl or Shell. The former has a
| crap-ton of portability documentation and design influence, and
| the latter is designed to work from 40 year old machines up to
| today's. You can learn a lot by studying old things.
| nasretdinov wrote:
| Go was never truly portable on Linux unfortunately due to its
| dependency on libc for DNS and user name resolution (because of
| PAM and other C-only API). Sure, pure Go implementation exists,
| but it doesn't cover all cases, so, in order to build a "good"
| binary for Linux you still needed to build the binary on (oldest
| supported) Linux distro.
|
| If your production doesn't have any weird PAM or DNS then you can
| indeed just cross-compile everything and it works
| liampulles wrote:
| Well now you've gone and linked to a fascinating tool which I'm
| going to have to dive into and learn: https://kaitai.io/
|
| Thanks.
| larusso wrote:
| I think this is true for nearly all compiled languages. I had the
| same fun with rust and openSSL and glibC. OP didn't mentioned the
| fun with glib-c when compiling on a fairly recent distro and
| trying it to run on an older one. There is the "many Linux"
| project which provides docker images with a minimum glib c
| version installed so it's compatible with newer ones. The switch
| to a newer open ssl version on Debian/Ubuntu created some issues
| for my tool. I replaced it with rust tls to remove the dynamic
| linked library. I prefer complete statically linked binaries
| though. But that is really hard to do and damn near impossible on
| Apple systems.
| Rucadi wrote:
| If you really need a portable binary that uses shared libraries I
| would recommend building it with nix, you get all the
| dependencies including dynamic linker and glibc.
| ksajadi wrote:
| I tink the title is a bit misleading. This is about very low
| level metrics collection from the system which by definition is
| very system dependent. The term "portable" in a programming
| language usually means portability for applications but this more
| portability of utilities.
|
| Expecting a portable house and a portable speaker to have the
| same definition of portable is unfair.
| davvid wrote:
| > We did not want to spend time maintaining a backward compatible
| parser or doing code archaeology. So this option was discarded.
|
| Considering all of the effort and hoop-jumping involved in the
| route that was chosen, perhaps this decision might be worth
| revisiting.
|
| In hindsight, maintaining a parser might be easier and more
| maintainable when compared to the current problems that were
| overcome and the future problems that will arise if/when the
| systemd libraries decide to change their C API interfaces.
|
| One benefit of a freestanding parser is that it could be made
| into a reusable library that others can use and help maintain.
| bb88 wrote:
| That's what I was thinking too. A go native library is 10 times
| better in the go ecosystem than a c library linked to a go
| executable.
|
| Also in the age of AI it seems possible to have it do the
| rewrite for you, for which you can iterate on further.
| khazit wrote:
| There is an existing pure Go library [1] written by someone
| else. The issue is that we weren't confident we could ship a
| reliable parser. We even included an excerpt from the systemd
| documentation, which didn't exactly reassure us:
|
| > Note that the actual implementation in the systemd codebase
| is the only ultimately authoritative description of the format,
| so if this document and the code disagree, the code is right
|
| This required a lot of extra effort and hoop-jumping, but at
| least it's on our side rather than something users have to deal
| with at deploy time.
|
| [1]: https://github.com/Velocidex/go-journalctl
| hollow-moe wrote:
| Hashicorp's Vault go binary is a whopping 512Mb beast. Recently
| considered using its agent mode to grab secrets for applications
| in containers but the size of the layer it adds is unviably big.
| And they don't seem interested into making a split server/client
| binary either...
| vb-8448 wrote:
| i wonder, for their use case, why not just submit journal in
| binary format to the server and let the serve do the parsing?
| xmodem wrote:
| It's crucial to be able to do some processing locally to filter
| out sensitive/noisey logging sources.
| trashburger wrote:
| Cross-compiling doesn't work because you're not defining your
| dependencies correctly and relying on the existence of things
| like system libraries and libc. Use `zig cc` with Go which will
| let you compile against a stub Glibc, or go all the way and use a
| hermetic build system (you should do this always anyhow).
| ifh-hn wrote:
| This stuff is out of my frame of reference. I've never used Go
| before and have never had the need to go this low level (C APIs,
| etc); so please keep this in mind with my following questions,
| which are likely to sound stupid or ignorant.
|
| Can this binary not include compiled dependacies along side it?
| I'm thinking like how on windows for portable apps they include
| the DLLs and other dependant exes in subfolders?
|
| Out of interest, and in relation to a less well liked Google
| technology, could dart produce what they are after? My
| understanding is dart can produce static binaries, though I'm not
| sure if these are truly portable compile once run everywhere
| sense.
| combiBean wrote:
| Was there not a third option: Calling the journalctl CLI as a
| child process and consume the parsed logs from the standard
| output? This might have avoided both the requirement to use CGO
| and also to write a custom parser. But I guess I am missing
| something.
| dardeaup wrote:
| This was the first thought that occurred to me too when I saw
| this post.
| cookiengineer wrote:
| Also there is a --json (or -o json) flag for journalctl which
| will output line based json log entries. And it can simply be
| called with a Command as you pointed out.
| redrove wrote:
| Yeah looks like they missed the forest for the trees.
|
| I see this kind of thing in our industry quite often; some Rube
| Goldberg machine being invented and kept on life support for
| years because of some reason like this, where someone clearly
| didn't do the obvious thing and everyone now just assumes it's
| the only solution and they're married to it.
|
| But I'm too grumpy, work me is leaking into weekend me. I had
| debates around crap like this all week and I now see it
| everywhere.
| IshKebab wrote:
| It's generally less robust to run CLI tools and scrape the
| output. Usually it isn't intended to be machine readable, and
| you have to handle extra failure modes, like incompatible tool
| versions, missing tools, incorrect parsers, etc.
|
| It's the lazy-but-bad solution.
| sigwinch wrote:
| journalctl with -o export produces a binary interchange
| format. Would you rather have bugs or API rot from that, or
| in an internal tool?
| jeremyjh wrote:
| journalctl is designed for these use cases and has options to
| solve those issues. The lazy part here is you not doing any
| research about this tool before dismissing it as "not best
| practice", which is exactly what the fuckups who wrote this
| article did.
| khazit wrote:
| We dismissed using journalctl at the very start. We've had
| similar experiences with other CLI tools: the moment you
| start embedding them inside a program, you introduce a
| whole new class of problems. What if journalctl exits? What
| if it outputs an error? What if it hangs? On top of that,
| you have to manage the subprocess lifecycle yourself. It's
| not as easy as it may seem.
|
| You can also argue that sd_journal (the C API) exists for
| this exact reason, rather than shelling out to journalctl.
| These are technical trade-offs, doesn't mean we're fuckups
| jeremyjh wrote:
| Does Go really not have any libraries capable of
| supervising an external program? If you'd considered
| journalctl, why didn't you mention it in the article? As
| many have pointed out here, it is the obvious and
| intended way to do this, and the path you chose was
| harder for reasons that seemed to surprise you but were
| entirely foreseeable.
| zbentley wrote:
| I think a lot is riding on that "generally". You're right
| that the default approach/majority of cases should avoid
| shelling out wherever possible, but there are a large
| minority of situations where doing that does make sense,
| including:
|
| Calling a CLI tool which will be present everywhere your
| program might reasonably be installed (e.g. if your program
| is a MySQL extension, it can probably safely assume the
| existence of mysqld).
|
| The CLI tool you want to call is vendored into or downloaded
| by your wrapper program, reducing installation requirements
| overhead (this is not always a good idea for other reasons,
| but it does address a frequently cited reason not to shell
| out).
|
| The CLI tool's functionality is both disjoint with the rest
| of your program and something that you have a frequent need
| to hard-kill. (Forking is much more error prone than running
| a discrete subprocess; you can run your own program as a
| subprocess too, but in that case the functionality is
| probably not disjointed).
|
| Talking to POSIX CLI tools in a POSIX compatible way (granted
| most things those tools do are easier/faster in a language's
| stdlib).
| tkone wrote:
| Ironically this is EXACTLY what the journald receiver for
| OpenTelemetry does, which, as they noted, is written in go.
|
| Specifically because you're only supposed to use that OR the c
| bindings by design because they want the ability to change in
| the internal format when it's necessary.
| arianvanp wrote:
| FWIW I maintain an official implementation of the journal wire
| format in go now.
|
| https://github.com/systemd/slog-journal so you can at least _log_
| to the journal now without CGO
|
| But that's just the journal Wire format which is a lot simpler
| than the disk format.
|
| I think a journal disk format parser in go would be a neat
| addition
| mbrock wrote:
| I've got a pure Go journald file writer that works to some
| extent--it doesn't split, compress, etc, but it produces
| journal files that journalctl/sdjournal can read, concurrently.
| Only stress tested by running a bunch of parallel integration
| tests, will most likely not maintain it seriously, total newbie
| garbage, etc, but may be of interest to someone. I haven't
| really seen _any_ other working journald file writers.
|
| https://github.com/lessrest/swash/tree/main/pkg/journalfile
| nicman23 wrote:
| so like every other language
| novoreorx wrote:
| This article reminds me of the days before LLMs ruled the world,
| when the word "agent" was most commonly used in the DevOps area,
| representing the program that ran on a remote machine to execute
| dispatched jobs or send metrics. Now I wonder how many developers
| would look at "agent" and think of this meaning.
| orochimaaru wrote:
| So you can't pull in c libraries built for different
| distributions and expect this to work.
|
| If you use pure go, things are portable. The moment you use C
| API, that portability doesn't exist. This should be apparent.
| jeremyjh wrote:
| My assumption was that they were using a C API just from
| reading the headline. I don't use Go but these sorts of
| problems are common to any project doing that in just about any
| language.
| colonwqbang wrote:
| Has nothing to do with go. You added a dependency which is not
| portable. It is well known that systemd project only targets
| Linux.
|
| Vendorise systemd and compile only the journal parts, if they are
| portable and can be isolated from the rest. Otherwise just shell
| out to journalctl.
| thomashabets2 wrote:
| The portability story for Go is awful. I've blogged about this
| before: https://blog.habets.se/2022/02/Go-programs-are-not-
| portable....
|
| It's yet another example of Go authors just implementing the
| least-effort without even a slight thought to what it would mean
| down the line, creating a huge liability/debt forever in the
| language.
| r_lee wrote:
| more like C is portable, until it isn't
| regularfry wrote:
| All of this, every last bit of complexity and breakage and sweat,
| is downstream of this:
|
| > Journal logs are not stored in plain text. They use a binary
| format
|
| And it was _entirely predictable and predicted_ that this sort of
| problem would be the result when that choice was made.
| valbaca wrote:
| Unix philosophy strikes again
| p0w3n3d wrote:
| Basically everything is portable unless it isn't. Java - the
| same. We fly in abstractions unless you need to delete a file
| mbrumlow wrote:
| I really hate this type of blog. It pollutes the world with this
| attitude of "I messed up, how I have to frame the problem in a
| way, and write a blog that lets my ego stay intact", which
| results in blogs like this showing in decision making process as
| "why you should not use go". And mostly people never look past
| the title.
|
| The fact is go is portable, it provides the ability to cross
| compile out of the box and reasonably executed on other platforms
| it supports. But in this case, a decision that had little to do
| with go, the desire to use c code, a non go project, with their
| go project made things harder.
|
| These are not "just a set of constraints you only notice once you
| trip over them", this is trivializing the mistake.
|
| Entire blog can be simplified to the following.
|
| We were ignorant, and then had to do a bunch of work because we
| were ignorant. It's a common story in software. I don't expect
| everybody to get it right the first time. But what we don't need
| is sensational titled blogs full of fluff to try to reason
| readers out of concluding the obvious. Somebody in charge made
| decisions uninformed and as a result the project became more
| complicated and probably took longer.
___________________________________________________________________
(page generated 2025-12-13 23:01 UTC)