[HN Gopher] Gokrazy - Go Appliances
___________________________________________________________________
Gokrazy - Go Appliances
Author : lagniappe
Score : 247 points
Date : 2023-12-19 05:23 UTC (17 hours ago)
(HTM) web link (gokrazy.org)
(TXT) w3m dump (gokrazy.org)
| denysvitali wrote:
| Presentation by Michael: https://youtu.be/mv_THRgLRHY
| drux wrote:
| The 25 Gbit/s at home guy
| https://michael.stapelberg.ch/posts/2022-04-23-fiber7-25gbit...
| jpgvm wrote:
| More importantly the author of i3. :)
| vasergen wrote:
| thanks, I didn't know that, I love i3!
| denysvitali wrote:
| More importantly: the author of Gokrazy :)
| lyxell wrote:
| The showcase of projects is (a bit unintuitively) listed under
| Available packages and is helpful if you want to get a picture of
| what you can build on top of gokrazy:
| https://gokrazy.org/packages/showcase/
| cedws wrote:
| I am very fond of the idea of a from-scratch userspace written
| entirely in Go. I've been dreaming of building something like
| gokrazy for servers. Or possibly something like Bottlerocket OS
| in Go.
|
| I experimented a few weeks ago with packaging a custom kernel and
| minimal Go init binary into a disk image bootable on GCE. No
| bootloader needed, just EFI loading the kernel directly.
| hyperthesis wrote:
| To some extent, android is this with java.
|
| For a low-powered device, would a Go usespace bring particular
| benefits? (Though, even the cheapest phones today aren't really
| "low-powered" anymore)
| yjftsjthsd-h wrote:
| Go has memory safety (via GC rather than fancy ownership, but
| if it works it works); that's an asset anywhere that can
| afford the overhead.
|
| Edit: In order to preempt the argument others are having:
| Safer than C, anyways, while being _much_ easier than rust.
| pjmlp wrote:
| If only we didn't need the const dance for enumerations,
| even Pascal like ones, to yet another 10 years, I guess.
| jrockway wrote:
| I don't have a big problem using codegen to get "syntax"
| for enumerations. "go generate ./..." before you checkin,
| save yourself some typing.
|
| I have used this in the past:
|
| https://github.com/abice/go-enum
|
| But mostly use protos as much as possible these days.
| (The enum is going to be in the API, might as well use
| the same object inside the app.)
| pjmlp wrote:
| I left that kind of gimmicks behind me in C++ compilers
| for MS-DOS.
| pjmlp wrote:
| There are USB security keys running Go bare metal,
|
| https://www.withsecure.com/en/solutions/innovative-
| security-...
|
| Look for TamaGo on the page.
| marcus_holmes wrote:
| Same - I've been playing around with Unikernels for the same
| reason. Running my web server as the only process on the
| machine really appeals. Attack surface is tiny, basically non-
| existent. If the web server falls over, reboot the machine
| (which should be fast because it only has to load the kernel &
| the Go program).
|
| It's basically the opposite of containers, which I like hehe.
| fuzztester wrote:
| Interesting. Which unikernel do you use?
| preisschild wrote:
| Talos Linux basically implements their entire userspace in Go
| and its similar to BottleRocketOS, because it is designed to
| host Kubernetes.
|
| https://www.talos.dev/ https://github.com/siderolabs/talos
| cedws wrote:
| Wow cool, thanks.
| pjmlp wrote:
| See TamaGo, https://github.com/usbarmory/tamago
|
| Or TinyGo, https://tinygo.org/docs/reference/microcontrollers/
| lifty wrote:
| Another project that aims to deliver this is Linuxkit
| (https://github.com/linuxkit/linuxkit). All the components they
| ship are written in memory safe languages (usually Go) and run
| as containers under containerd. You can build a custom image
| very easily, fully defined as a YAML file.
| 0xdeadbeefbabe wrote:
| There is also u-root which is "A fully Go userland with Linux
| bootloaders! u-root can create a one-binary root file system
| (initramfs) containing a busybox-like set of tools written in
| Go." https://github.com/u-root/u-root
| espe wrote:
| this looks really cool. might be perfect for an old odroid. hope
| that doesn't stray too far from the happy paths.
| withinboredom wrote:
| Go simply can't really do well unless it has globs of memory or
| properly set kernel parameters (that affect the whole global
| machine). To get it to run reliably in low-memory conditions also
| requires careful tuning of GOGC + GOMEMLIMIT + MADV_DONTNEED. Go
| is geared more towards servers with tons of available memory, not
| IoT devices. Further, it's networking stack does really well on
| low-latency connections (ie, a datacenter) and starts to do
| poorly if you throw a in link with lots of data loss (this
| includes clients across the internet on crappy wifi).
|
| I love that they basically reimplemented libc to get more
| performance, but they also threw away decades of edge cases that
| it doesn't (yet) handle very well. I'm confident they'll get
| there, eventually, especially when people do things like this and
| report issues.
| denysvitali wrote:
| That's why you have TinyGo: https://tinygo.org/
| withinboredom wrote:
| This looks like a really interesting project. Thanks for
| sharing it! I'm going to have to try this out with a couple
| of projects. Just a cursory experiment reveals that it's GC
| seems to be much better than Go's GC and I'm very curious as
| to why that is.
| dolmen wrote:
| https://tinygo.org/docs/reference/lang-support/#garbage-
| coll...
| withinboredom wrote:
| Interesting that it says it is slower. It may actually be
| slower, but tends to not suffer as much degradation as
| the Go's GC. In other words, on my test setup using Go, I
| get ~1k rps once I hit memory limits, but with tinygo I
| get 7k rps once I hit memory limits. So it might be
| "slower" but it looks like it may be more efficient as
| well.
| rob74 wrote:
| Also good for WASM...
| sapiogram wrote:
| TinyGo is not Go, it's more accurate to see it as a dialect
| rather than a separate Go implementation imo. Many packages
| (even the stdlib) don't compile under it, and as I understand
| it, certain language features involving runtime reflection
| are likely to never be supported.
| pjmlp wrote:
| Just like what many people call C is not ISO C, and
| similarly many ISO C libraries don't compile with "C"
| compilers for the same environments as TinyGo.
| hamburglar wrote:
| The differences between TinyGo and Go are vast compared
| to "C vs ANSI C" though.
| pjmlp wrote:
| Depends on which embedded toolchain we are talking about.
| oefrha wrote:
| The targets of this project are RPis (with 512MB+ RAM) or
| similar. That's tons of RAM compared to "IoT devices" you have
| in mind, and not an issue at all for Go.
| withinboredom wrote:
| 2Gi of ram is nearly the bare minimum to run a single Go
| server without suffering (huge) performance penalties --
| we're talking a 50-99% degradation of performance while GC
| sucks up all the CPU power to gain more memory.
| bradfitz wrote:
| That's not true in general. Sounds like you have a really
| inefficient server that allocates too much.
| withinboredom wrote:
| It's not a matter of allocation since there isn't a way
| to manually free memory in Go. The GC basically doesn't
| do anything if there isn't memory pressure. Once your
| applications starts feeling some memory pressure, the GC
| will start spending CPU to free up memory. This is a non-
| disputable fact of how Go works, in general.
| logicchains wrote:
| >It's not a matter of allocation
|
| It's absolutely a matter of allocation, since the more
| you allocate, the more work the GC will have to do. If
| you write clean, memory-friendly code (e.g. reusing
| buffers, favouring stack allocation), then you'll
| generate a lot less garbage and the GC will spend less
| time collecting it.
| 4ad wrote:
| > It's not a matter of allocation
|
| > Once your applications starts feeling some memory
| pressure
|
| I rest my case.
| Yasuraka wrote:
| >The GC basically doesn't do anything if there isn't
| memory pressure
|
| It will run every 2 minutes, if it hasn't run otherwise
|
| https://github.com/golang/go/blob/9b4b3e5acca2dabe107fa2c
| 3ed...
| d-z-m wrote:
| I think bradfitz knows how Go works.
| oefrha wrote:
| Without talking about your use case, this is such an absurd
| claim that doesn't deserve a response. The rest of us can
| easily run a dozen or more typical light load HTTP services
| within 512MB.
| withinboredom wrote:
| I think the keywords there are "light load", you probably
| won't notice any issues as long as your requests are
| spaced out.
| oefrha wrote:
| Light load as in serves a couple rps persistently without
| problem (complete with SQLite database operations), and
| I'm not talking about highly optimized code. Enough for
| most home or small business intranet use cases. You're
| not going to serve google.com in 50MB, obviously. If your
| code has to allocate ~100 megs for a request without an
| obvious reason that requires that much allocation, that's
| on your code, not the language.
| BillinghamJ wrote:
| I don't know how you'd define "light load", but I have a
| few services that average about 100rps. Only 1 active
| container most of the time, they're allocated 128MiB RAM,
| with a p99 utilization of 55%. They don't have any GC
| config or special optimization - just operating with
| regular JSON POST requests, making MongoDB queries
|
| Go just does incredibly well in these kinds of
| conditions. The only reason we allocate 128MiB is because
| that's the minimum Google Cloud Run allows - we'd
| probably go for 64MiB on most systems otherwise
| jerf wrote:
| No, request rate doesn't really matter. Increased request
| rate will increase the rate of the GC running, but it
| doesn't intrinsically increase the amount of RAM used.
| (More _simultaneous_ requests can, of course, but that
| isn 't necessarily implied by an increased request rate.)
|
| Look, let me blunt, since you have insistently posted
| several times: You don't know what you're talking about.
| The code is open source, the algorithms publicly
| available, and it simply _is not the case_ that Go is
| programmed to simply run up to gigabytes of memory usage
| before going "hurr durr I guess I should free some
| stuff". Like many other posters here, I also in
| possession of numerous 24/7 Go servers that run all sorts
| of things through them without needing gigabytes of RAM.
|
| I can tell you what happened to you with high
| probability. You wrote a memory leak, and mistook the
| result for Go's error rather than yours. Hit your leaking
| code with the built-in memory profiler and figure out
| what you're leaking and fix it. It's OK, it happens to
| all of us, I've written many such leaks in many different
| languages, with many different allocation schemes.
|
| RPis are abundantly capable of running Go. They can run a
| lot of Go. They're way, way above minimum spec. Of course
| Go won't let you magically hold a decompressed video
| stream in memory any more than anything else does or work
| other magic, but it's a heck of a lot more efficient than
| Python, and can run an awful lot of server in the
| resources a web browser takes just to display a blank
| window.
| acheong08 wrote:
| Skill issue. A 512 MB server is able to run gitea and
| dendrite along with a bunch of other of my own services
| just fine with acceptable performance.
| devjam wrote:
| I'm bemused by these sorts of disparaging comments that
| come from people who seem to have no real world experience
| running Go applications at scale.
|
| It's entirely possible to run a server that will handle
| tens of thousands of req/sec on hardware with comparatively
| small amounts (by todays standards) of RAM e.g. 512MB
| withinboredom wrote:
| I don't see how you think my comments are "disparaging,"
| this is just code doing what it is programmed to do.
|
| > It's entirely possible to run a server that will handle
| tens of thousands of req/sec
|
| Indeed, you can do this. I'm talking about what happens
| once the Go starts getting memory pressure and the GC
| starts dominating the CPU. This is what it is designed to
| do. If your code can prevent that from happening, you are
| correct then, as it won't be an issue.
| veqq wrote:
| > Go starts getting memory pressure and the GC starts
| dominating the CPU
|
| That does not happen. You must be thinking of Java 15
| years ago.
| pjmlp wrote:
| Even Java 17 years ago is ironic, considering the
| hardware requirements of J2ME, Java Embedded, Blue Ray
| players, Ricoh and Xerox copiers, Cisco phones, and so
| forth.
| jrockway wrote:
| Prior to GOMEMLIMIT being a thing, people would get
| themselves into this situation occasionally. Their steady
| state working set size is something like 2.1GB, but they
| want to fit into a 4GB container limit, so GOGC=100 led
| to OOMs occasionally. They would bring GOGC down to 10 or
| something (why not 90!?), and then they are running GC
| every time they allocated 10% or 200MB, and then their
| CPU profiles would show they were burning a lot on GC,
| because it's pretty easy for an RPC server to generate
| 200MB of garbage.
|
| GOMEMLIMIT definitely improves things, taking the very
| blunt instrument GOGC out of people's hands, and does
| what they actually want instead, which is to prioritize
| GC only when you're about to run out of memory.
|
| People dislike tunables like this, and rightfully so, but
| a similar problem exists somewhat insidiously in non-
| garbage collected languages. Calling malloc() and free()
| is not a zero-cost operation, and you can certainly waste
| CPU by calling free() when nothing actually needs the
| memory you are freeing. GC lets cases like "this is a CLI
| app, the system has 128GB of RAM, the working set size is
| 2GB, and it's over in 1 second" basically never do any
| memory freeing. You allocate as much as you need, GC
| never runs, and the app exits. That kind of optimization
| is actually pretty neat. But, for steady state servers
| that run for an unbounded period of time, it is
| inevitable that you are going to spend time trying to
| reuse memory that you no longer need, because the system
| only has a finite amount of memory. Then the compromises
| start to balance throughput with efficiency. gc +
| GOMEMLIMIT is a very nice compromise that is excellent
| for many applications, but no heuristic is perfect for
| every possible computer program.
| curioussavage wrote:
| I've done well over 100k req/s and I'd have to ask former
| coworkers but I'm pretty sure we were using under 2Gb of
| ram.
| bborud wrote:
| Is it likely that you would run servers "at scale" on a
| RPi?
| oefrha wrote:
| A newer rpi is in fact more than enough scale for most
| websites in existence.
| pjmlp wrote:
| A RPi is more powerfull than most PCs I owned between
| 1990 until 2002.
| bborud wrote:
| Come to think of it, in terms of computing power, the
| ESP32 is probably more powerful than what you used before
| 2000.
| jrockway wrote:
| I have written a bunch of tiny Go apps, and agree with
| you. In general, 64MB is where I set the limit on these
| things and there are no problems. I have some big
| applications that literally allocate 1GB at once and they
| still run OK within 2GB memory limits. (Obviously, you
| can't do two of these operations at once, though.)
| tgv wrote:
| That's bollocks. I've got a 4GB server which runs 5
| production services, and (maximally) 20 test services, and
| only one of those ever goes over 40MB, and then drops below
| that again. The production services handle about 65k
| requests per day. Not much, but it shows that you can
| easily run go applications in less than 2GB.
| sapiogram wrote:
| It's not an exact comparison, but at work we have tons of
| Go apps running in Docker containers with a 50MB memory
| limit. For lighter workloads, it works just fine.
| tedunangst wrote:
| What networking stack?
| 1vuio0pswjnm7 wrote:
| This project always seemed to be swimming against the current
| by using Go but for the longest time no one would state the
| obvious. Finally. Now I know I'm not krazy.
| 4ad wrote:
| The whole point of an appliance is that it runs a single
| application. Go does not have any sort of networking stack.
| This comment is 100% nonsense.
| Thaxll wrote:
| The Go processes I'm currently working on do thousand of
| requests per seconds doing tls and DB calls / serialization
| with less than 128MB of memory.
|
| Go does not need me much memory, it's in the same ballpark as a
| C++ server, using way way less memory than Java / C#.
| psanford wrote:
| I'm running gokrazy on a raspberry pi zero 2 w with 512MB of
| ram. I'm using it as a record-on-motion security camera[0]. The
| actual recording happens with ffmpeg, but the Go server does
| motion detection, hls streaming, and cloud upload of video
| chunks. It also does this all in-memory, the sdcard in the pi
| is mounted read-only.
|
| It works great! No memory pressure/GC issues at all. Its really
| not that hard to write go that can perform well in moderately
| constrained environments.
|
| [0]: https://github.com/psanford/rom-cam
| dsff3f3f3f wrote:
| I've seen fairly naive Go servers that do 10,000+ RPS that
| issue database requests on every request that consistently use
| less than 50MB of RAM without issue. Several of them barely
| ever break 30MB of RAM.
| dolmen wrote:
| Looks interesting.
|
| However, I don't like to have a Go compiler on the production
| machine for security reasons, as well as to reduce even more the
| system base.
|
| I think instead that the compiler should be needed only to deploy
| binaries and should stay on the syadmin's machine. The Go
| toolchain is very good at cross-compiling, so there should not be
| a need for the compiler on the target machine.
| secure wrote:
| I don't know where you got the impression that the Go compiler
| would run on "the production machine".
|
| gokrazy images are built whereever you like (typically your PC,
| or some more automated setup) and then deployed to the target
| device. There is no Go toolchain on the target device.
| __jonas wrote:
| Oh this seems great, I actually just wrote a small Go program
| that I was planning on running on a Raspberry Pi as the only
| thing running on the device, looks like this could make it a bit
| simpler to deploy, very nice!
| asim wrote:
| The idea is not bad. Execution is always hard with this stuff.
| I've done a pure Go app platform with this sort of desire for a
| native tool chain but it's very hacky. I do wonder whether wasm
| will solve some of this but sometimes it's a solution in search
| of a problem. Still the simple idea of a server that will pull
| and run your code on demand with a simple "run x" is always
| appealing.
| nyadesu wrote:
| Couldn't someone achieve this by using a very lightweight Linux
| distro, setting up ssh and using Ansible?
| theshrike79 wrote:
| Yes, but this is even lighter weight. There is no distro or
| kernel, only the program.
| AHTERIX5000 wrote:
| gokrazy does use Linux kernel though.
| rickcarlino wrote:
| Reminds me very much of the Elixir Nerves framework*. I'm glad to
| see there are more options out there now.
|
| How does Go fare on single core machines like older RPi0 models?
|
| * https://nerves-project.org/
| Thaxll wrote:
| Go is pretty performant, faster than Nerves for sure, I have an
| old RPB+ from 2014 and some go code on it, works just fine. The
| Go compiler can target most popular arm platforms by default.
| rcarmo wrote:
| Except that you need older versions of Go to target ARMv6,
| like the pre-2 revs and the Zero.
| Thaxll wrote:
| Aren't those are arm11? They can all be compiled with any
| version of Go afaik GOARM=6
| whalesalad wrote:
| They are different beasts. Nerves does OTA updates
| (blue/green deploys), and a lot of other cool stuff. Needs
| beefier hardware but I started building a pretty wicked
| little tool on rpi4 with nerves to try and accurately track
| motorcycle lap times using gps and sensor fusion. It was
| pretty snappy in my experience.
| IshKebab wrote:
| This is the right way to do Raspberry Pi projects, but I think
| tying it so strongly to Go is a mistake. Is that really
| necessary?
| asylteltine wrote:
| This is actually quite useful to something I am working on thank
| you. I love Go.
___________________________________________________________________
(page generated 2023-12-19 23:02 UTC)