[HN Gopher] The case of a leaky goroutine
___________________________________________________________________
The case of a leaky goroutine
Author : surprisetalk
Score : 62 points
Date : 2024-03-25 12:01 UTC (11 hours ago)
(HTM) web link (brainbaking.com)
(TXT) w3m dump (brainbaking.com)
| jerf wrote:
| It's a pity Go didn't have structured concurrency:
| https://vorpus.org/blog/notes-on-structured-concurrency-or-g...
|
| There's a library for it: https://github.com/sourcegraph/conc
|
| But this goes to one of the things I've been kind of banging on
| about languages, which is that if it's not in the language, or at
| _least_ the standard library right at the beginning, sometimes it
| almost might as well not exist. Sometimes a new language can be
| valuable, even if it has no "new" language features, just to get
| a chance to reboot the standard library it has and push for
| patterns that older languages are theoretically capable of, but
| they just don't play well with any of the libraries in the
| language. Having it as a much-later 3rd party library just isn't
| good enough.
|
| (In fact if I ever saw a new language start up and that was
| basically its pitch, I'd be very intrigued; it would show a lot
| of maturity in the language designer.)
| typesanitizer wrote:
| Structured concurrency is part of the Swift standard library,
| and was added at the same time when first-class support for
| concurrency was added.
|
| TaskGroup in the standard library -
| https://developer.apple.com/documentation/swift/taskgroup
|
| Explore structured concurrency in Swift (WWDC 21) -
| https://developer.apple.com/videos/play/wwdc2021/10134/
| Cthulhu_ wrote:
| > The concept was formulated in 2016 by Martin Sustrik (creator
| of ZeroMQ) with his C library libdill, with goroutines as a
| starting point.
|
| It's fairly new; the thing (and I think you address it too) is
| that the pattern did not exist yet when Go was introduced. Go
| is averse to adding more things to its standard library, or
| indeed changing its core fundamentals; I think it's better to
| have one well-defined way of doing things in a language,
| instead of adding the mental overhead of deciding between one
| or the other.
|
| And I doubt Go will remove support for their original
| concurrency, like, ever. I'd love to see forks of Go made with
| core elements (like concurrency) swapped out though.
| masklinn wrote:
| > It's fairly new
|
| That specific formulation is new but the concept had been
| floating around for a while. For instance Rust originally got
| scoped thread in 2015 (before they had to be removed for
| being unsound).
| jerf wrote:
| Yes, I do know Go predates a solid description of the
| paradigm, though I hid that in what otherwise looks like a
| bizarre verb tense in my first sentence. :)
|
| Part of the reason I rhapsodize about new languages off of
| that observation is precisely that Go can't add it. It almost
| wouldn't even matter if they tried to put it in the language
| proper, because by backwards compatibility the old ways would
| still work, and it would take a very long time to get the
| entire ecosystem to the new way.
| roguecoder wrote:
| Guy Steele's talk "Growing A Language" gets into this. It's
| definitely worth looking up if you haven't seen it.
|
| I think we've seen two ways languages have successfully been
| open to evolution:
|
| Java was specifically designed to allow it to build on the
| standard library over time in fully backwards-compatible ways.
| It has required the central governance committee to adopt
| proposals, because reflection is slow and poorly-optimized, but
| it is a far more fully-featured language today than it was an
| inception without ever needing a reset. By keeping the surface
| area small & the strong "Everything Is An Object" paradigm in
| place, it has had remarkable longevity and has avoided the
| Python versioning pain.
|
| The second are the "sharp knives" languages: Javascript, Ruby
| and to a lesser extent C++ (only because DLL hell is very
| real).
|
| All three of these can be used to write software in any
| paradigm (including Aspects, if one is a masochist), and so let
| engineers invest in their own productivity. Languages where the
| standard libraries are indistinguishable from custom libraries
| require more skill and collective team alignment to use
| productively and safely, but also allow for solutions highly-
| opinionated languages can't support.
| paulddraper wrote:
| Structure concurrency is to concurrency what structured program
| control are to program control.
|
| I.e. unstructured concurrency is like GOTOs. Not necessarily
| wrong, but certainly nerve-wracking.
| peterashford wrote:
| That's a lovely analogy, and I concur :o)
| lxe wrote:
| Thread leaks, which happen more frequently due to threaded async
| abstractions, such as the goroutine, are less often discussed
| than memory or CPU leaks, but are much more dangerous in multi-
| tenant container environments.
|
| A thread leak can lock up your entire node, including all the
| control plane processes. A container spec doesn't provide an easy
| way to control thread/nproc/ulimit limits (you can still do it,
| but it's not straightforward), which in turns leaves pretty much
| every k8s deployment misconfigured and vulnerable to thread
| leaks.
| hedora wrote:
| My #1 complaint about Rust is that leaking a future is safe. It
| means the compiler can't check for async coroutine leaks, and
| it breaks the borrow checker's ability to say "nothing else has
| a reference to this any more".
|
| Anyway, we're using golang for some stuff at work, and holy
| crap, I forgot how terrible it was to work in high level
| languages that don't statically check for correct
| synchronization.
|
| If C++-style concurrency is like a chainsaw, then golang
| concurrency is like a chainsaw in a bouncy castle.
| f_gergo wrote:
| What is a "correct synchronization"?
| kccqzy wrote:
| Synchronization such as the whole program is well-defined
| according to some memory model.
| Thaxll wrote:
| Which is the case for Go:
|
| https://go.dev/ref/mem
| kccqzy wrote:
| A quote from your link:
|
| > programmers are strongly encouraged to use appropriate
| synchronization to avoid data races
|
| Any time you need to "encourage" programmers to do the
| right thing, you have already failed in your language
| design.
|
| And I think OP agrees with me here. OP says "static
| checking of correct synchronization" which is
| irresponsibly absent from Go.
| Thaxll wrote:
| So it's absent from every language but Rust?
| sapiogram wrote:
| Javascript is even better, by not having multithreading
| at all. I am not joking, Go is much worse than Java and
| C#, but Javascript and Rust are the only mainstream
| languages where I've seen non-experts reliably write
| correct concurrent code. Maybe it's true for langauges
| like Elixir as well, but I haven't tried.
| masklinn wrote:
| > My #1 complaint about Rust is that leaking a future is
| safe.
|
| There's no other way given the leakpocalypse decision. You'd
| need an entirely new leak-proof language to fix that, and
| that means you need alternatives for Rc and Arc (or a way to
| prevent them creating a cycle).
| duped wrote:
| > My #1 complaint about Rust is that leaking a future is
| safe.
|
| Do you mean futures that aren't polled to completion, tasks
| that aren't joined, or literal memory leaks that happen to
| own futures?
| js2 wrote:
| Isn't this addressed by CPU requests/limits + pid limiting?
|
| https://kubernetes.io/docs/concepts/policy/pid-limiting/
| lxe wrote:
| It's just more annoying to set up and isn't as widely known,
| and doesn't work on a per-pod basis.
|
| > PID limiting is a an important sibling to compute resource
| requests and limits. However, you specify it in a different
| way: rather than defining a Pod's resource limit in the .spec
| for a Pod, you configure the limit as a setting on the
| kubelet. Pod-defined PID limits are not currently supported.
| kbolino wrote:
| Goroutines in a single process map onto a fixed number of
| threads. Even if you have goroutine leaks, you should not have
| thread leaks. Your program may deadlock or run out of memory,
| but it will not take the whole system down (at least, not in
| this way).
| usrnm wrote:
| > Goroutines in a single process map onto a fixed number of
| threads
|
| Not necessarilly true if you're using cgo
| mholt wrote:
| This article doesn't go into depth about capturing the profile to
| identify the leak, so if you want more instruction on one way to
| do that, I wrote an article recently called Profiling Caddy,
| which is geared toward Caddy but really works for any Go
| programs: https://caddyserver.com/docs/profiling
| shp0ngle wrote:
| Didn't Uber have some leaky goroutine detector? I vaguely
| remember seeing something like that, 5 years ago...
|
| Ah yeah it's here.
|
| https://github.com/uber-go/goleak
| latchkey wrote:
| Uber also made something called fx, which is fantastic.
|
| You don't have to use it, but when you do, it helps ensure that
| you organize your code in a way that becomes very easily
| testable. It enforces a modular approach to composing together
| golang services.
|
| Being more easily testable helps prevent bugs, like these leaky
| goroutines.
| Karrot_Kream wrote:
| `fx` is mostly just Dependency Injection in Go which has been
| a thing in Java forever.
|
| I'm curious though, when do you reach for `fx` in a non-
| industrial project and when do you not? I still use the same
| patterns of separating out the implementation from the
| interface but I've been wiring in the dependencies by hand.
| I'm curious if folks reach for `fx` immediately or if it's
| something that requires thought to add. There's also Google's
| wire library [1] that does similar stuff but takes a compile
| time approach so it's a little easier to reason about if
| struct initialization screws up due to weird implicit things.
|
| I still wire dependencies up by hand, but I'm curious what
| others do.
|
| [1]: https://github.com/google/wire/tree/main
| latchkey wrote:
| I co-founded Java @ Apache, so my background is Java and DI
| was the first thing I was looking for when I started down
| the golang path. I tried out a bunch of different DI
| options for golang and settled on fx. fx is actually more
| than just DI, it is a whole framework for starting and
| stopping "services" as well.
|
| I realized quickly it wasn't absolutely necessary to use it
| since most people just make a package in golang, in order
| to get the separation they need. But when I started using
| it more and more, I noticed that taking advantage of the DI
| features of fx, also ensured that I wrote code that had
| clear separation of concerns.
|
| In golang, it is too easy to just cross package include
| `new` things you need right in the function, instead of
| passing it in as an argument. This of course, makes it much
| harder to write tests for since you can't mock what you
| need easily.
|
| The binary I built was distributed across tens of thousands
| of servers in multiple data centers, and had to run
| perfectly on every release as it took a lot of time/effort
| to even do updates. This meant comprehensive testing before
| deployment, so I wanted to optimize my unit/integration
| tests as much as possible.
|
| I'm not sure it would be necessary for just simple api
| endpoint microservices, but for a complicated application
| binary that needs perfect testing, I can't imagine writing
| golang code without it. The benefits far outweigh the
| negatives.
| Karrot_Kream wrote:
| Yeah when I've written Go at scale we've used fx for
| similar reasons. But for smaller projects I go back-and-
| forth. Fx makes it quick and easy to start using DI and
| avoid the repetitive hand initialization and injection,
| but for smaller codebases it's also much easier to reason
| about. FWIW I haven't used wire before as if I'm at a
| scale smaller than fx I'm just wiring in structs by hand.
| bakul wrote:
| If Go allowed something like "handle = go foo()", goroutine could
| be automatically terminated when handle goes out of scope or
| becomes dead and is garbage collected. You can also use handle to
| cancel a goroutine etc.
|
| Go designers specifically avoided this model (of having a
| goroutine "id") for reasons I don't remember any more (may be to
| avoid making them heavier weight?) but this would be one way to
| stop leaky goroutines.
| atombender wrote:
| It's because just terminating a goroutine isn't safe with
| respect to I/O and defer chains.
|
| I _think_ Go has the internal plumbing to theoretically support
| this, though it might require inserting checks more often.
| Another way would be to make contexts first-class and
| automatically insert context checks even when not done (e.g.
| selects). And also all I /O has to be cancellable.
|
| I suspect Go's designers prefers the current way in which
| cancellation is explicit.
| bakul wrote:
| > just terminating a goroutine isn't safe with respect to I/O
| and defer chains.
|
| Agreed. The idea is to panic() if a goroutine has to be
| forcibly terminated due to GC, instead of a slow leak.
| Requires more thought though.
| masklinn wrote:
| So the idea is to randomly take down the program when the
| GC runs?
| bakul wrote:
| Let me try to explain. The current way (which must
| continue working the same way even if the language is
| changed) does not allow you to distinguish a goroutine
| that should terminate but hasn't due to some bug versus a
| goroutine that can legitimately run for a long time.
| Making the goroutine "id" explicit can allow you
| distinguish the two cases. Store the id in some global or
| long lived variable or array if you want the goroutine to
| run for a long time. Otherwise carefully control the
| scope of such an id so that the runtime has a chance to
| catch the first case. That is my initial thinking but it
| would need to be fleshed out more. For instance, there
| should be a way to test that goroutine has terminated.
| Currently you do this explicitly by passing a channel and
| waiting for a message.
| dagss wrote:
| The Go way is to pass the long running goroutines a
| different ctx from the short running.
|
| Or even one ctx per goroutine and cancel them dynamically
| according to whatever logic.
| ikiris wrote:
| If you actually code in go there, they expect you to handle it
| yourself though by not orphaning goroutines. Err groups or
| similar will take care of this. Basically it's people writing
| bad code because they can. There's already options to handle it
| available.
| dagss wrote:
| The convention of passing ctx does almost the same thing
| though. Make a new context.WitCancel and pass it to the
| goroutine.
|
| It just requires programmer cooperation, but as long as you
| pass ctx all through the stack down and handle err on the way
| back, it is not often you deal with it explicitly.
| masklinn wrote:
| The explanation of the issue in ToDoneInterface really is not
| clear to me because of this:
|
| > The defer close() seems to close well, but it's on the wrong
| channel.
|
| The `done` input channel is supposed to be closed by a caller,
| and the goroutine is closing the output channel, surely that's
| the point?
|
| Now from what I know of go channels and understand of the code
| involved, the `done` channel may never get closed by the parents
| (and can never get closed at all if it's nil?), in which case the
| goroutine never receives a signal, never terminates, and leaks.
| But the explanation below the snippet confuses me completely.
|
| And if that's that... what's the fix? Aside from not doing this
| sort of conversions? Just "git good scrub", try to make sure you
| don't rely on cancellation for progress, and hope you don't use
| raw background channels again, and don't forget to cancel your
| non-background channels?
| samatman wrote:
| A lot of these problems come from accepting a line in the OP, "A
| Goroutine is essentially a coroutine". The rest of the sentence
| is "...that maps onto green threads that map onto real native
| threads on your OS in an NxM way".
|
| This is not a coroutine at all, calling them goroutines was a
| clever hacker pun along the lines of "GNU's Not Unix". If you
| treat a preemptively-scheduled primitive as though it's a
| cooperatively-scheduled primitive, you're going to have a bad
| time.
|
| Goroutines are threads, basically, with all the memory-management
| headaches that implies. Caveat emptor.
| kbolino wrote:
| I was under the impression that goroutines were cooperative but
| that they yield pretty aggressively (on blocking channel
| operations, disk and network I/O, cgo calls, and certain
| syscalls). What makes you think they're preemptive?
| masklinn wrote:
| New implicit yield points started getting added from 1.2,
| which added a yield point in the function prologue (so any
| function call, even with no IO whatsoever, could yield). I
| think later releases also added yield points on allocation
| and stack growth.
|
| This culminated in 1.14, which made the runtime preemptive on
| most platforms (https://go.dev/doc/go1.14#runtime) in order
| to fix the last sticking point where a goroutine might not
| yield: a tight numerical loop might never yield.
|
| This was an issue, because the GC relied on scheduling to
| slip in its STW pauses, so the GC would trigger STW,
| progressively pause every goroutine reaching a yield point,
| but would be unable to ever pause the last goroutine, and the
| program would pretty much grind to a halt until it was done.
|
| There are ways to handle this (e.g. insert trapping reads in
| various control structures), but ultimately preemption was
| considered a better and more useful solution.
| atombender wrote:
| I wish Go recorded the timestamp of goroutine and let you access
| them.
|
| An app I work on recently had a bug where goroutines would slowly
| build up over time. Turns out the bug is in the Growthbook SDK
| [1]. We can monitor the number of goroutines, but having a large
| number of goroutines waiting in the location that gets stuck is
| normal -- we can only see such a problem over multiple days, in
| that the minimum value slowly goes up.
|
| If Go could tell you the timestamp of the oldest goroutines as
| part of the pprof dump, we could have an alert, and it would work
| for any such leak.
|
| [1] https://github.com/growthbook/growthbook-golang/pull/28
| jmholla wrote:
| Language support would be great, but could you add logs that
| record the creating and destruction of goroutines, giving them
| a unique UUID so you can track which one's haven't exited?
|
| Edit: Also, maybe the tool at this comment could've helped you?
| https://news.ycombinator.com/item?id=39817775
| codethief wrote:
| I don't have any experience with Go but to my untrained eyes this
| looks very much like a general problem that I've noticed with
| coroutines in other languages, e.g. JavaScript or Python1:
| Coroutines are so lightweight that people tend to "fire & forget"
| them, when in reality coroutines take up memory and can easily
| leak. One _should_ keep track of them and garbage-collect them
| but last time I checked there wasn 't a great out-of-the-box
| solution for that.
|
| 1 Same thing in frameworks like RxJS, where observers in some
| sense take on the role of coroutines.
| psnehanshu wrote:
| Just wondering if threads in Rust can suffer such problems?
|
| Background: I am coming from the JS/TS/Node world, and have
| decided to jump onto a compiled language. I narrowed down my
| choices to Go and Rust and eventually decided to go with Rust,
| because it didn't use GC for memory management.
| Filligree wrote:
| Less likely, but it certainly can. The problem isn't the
| garbage collector; it's the overall approach to threading. Rust
| has a different culture that makes problems like this
| _probably_ less of an issue, but nothing stops you implementing
| a memory leak.
|
| In fact, since it doesn't have a GC, you can trivially create a
| memory leak by creating a reference loop... though the
| ownership checker makes that in itself really difficult, and so
| it's again less likely to happen than it otherwise would be. At
| the cost of loops being hard to make even if you want them.
| Thaxll wrote:
| Every language can leak memory and threads.
| whateveracct wrote:
| People hate on Haskell async exceptions (with good reason), but
| one cool thing about them and the Haskell RTS is that you can
| almost [1] always cancel a thread from the outside. No need for
| the thread to cooperate like in Golang.
|
| The entire `async` package is built on this. The `race`
| combinator is an especially cool application.
|
| After doing a big project in Golang, I appreciated this more. We
| had our fair share of goroutine leaks.
|
| [1] iirc, if the thread is not blocking on a syscall or
| allocating memory, it will not be yielding to the RTS.
| treyd wrote:
| Rust's futures also get this through being poll-based. You
| cancel a future by dropping it. Futures compose really easily,
| so you can combine a bunch of futures in interesting trees of
| selects and joins and any that are not completed get cleaned up
| automatically and with little/no overhead when they go out of
| scope / when the task they're a part of completes. You don't
| think about them like separate chunks of work, you just think
| about them like types you can await on and yield a value, and
| the compiler flattens it all out into a state machine. All of
| the sync and composition combinators are implemented just in
| the traits/types of the tokio/futures libraries because the
| poll/waker abstraction is low level and versatile enough. As
| long as you don't go out of your way to write bad async code,
| there's no leaks for the same reasons there's no leaks in Rust
| code.
|
| It feels a lot like the monadic composition of Haskell even if
| the means it achieves it are very different.
| iforgotpassword wrote:
| I'm gonna be that guy. The old man yelling at cloud. I don't get
| all this high level crap. Coroutines, goroutines, fibers,
| async/await. It's supposed to make concurrency easy and safe. But
| I just fail to build a working mental model for it. I get the
| rough idea, but every time there's an await I wonder where
| execution might jump next. And then you read stuff like this, how
| these super high level comfortable languages fuck you over if
| you're holding them wrong, and even a Go dev has to admit they
| needed to stare at the code for an hour to get it. I don't
| understand half the words in that post, but it makes me want to
| stay away from that language.
|
| I prefer multi threaded programming. Everything is off to the
| races (pun intended), you need to think long and hard about
| lifecycle management, who creates a resource, who will clean it
| up, how do you synchronize, where do you synchronize. It might be
| hard to get right sometimes. But the concept is simple. The tools
| you have available are simple. There's no "well everything works
| fine automagically unless it doesn't because these 10 lines of
| code".
| doctor_eval wrote:
| I only skimmed the article but if I understand correctly, the
| problem was in a dependency (library), not in the code. That
| could happen to anyone and is (arguably) not really a fault in
| the language.
|
| From the other comments here it's not clear that there is a
| modern language that doesn't have this problem.
| cherryteastain wrote:
| A goroutine is practically just a thread
| vonwoodson wrote:
| While The White House (and actual Rust and Go enjoyers) are
| advocating for these safe memory-safe languages; what is really
| going on is that the warts these languages have are just not as
| well known yet. In a few years time Go will be just as hated as
| C++, and there'll be some new darling programming language
| that'll "solve all out problems".
|
| To be fair, I do look forward to when logic programming languages
| get their time in The Sun.
___________________________________________________________________
(page generated 2024-03-25 23:02 UTC)