[HN Gopher] Ergo: Erlang/OTP patterns implemented in Golang
___________________________________________________________________
Ergo: Erlang/OTP patterns implemented in Golang
Author : rapnie
Score : 93 points
Date : 2023-01-28 12:43 UTC (10 hours ago)
(HTM) web link (cloud.ergo.services)
(TXT) w3m dump (cloud.ergo.services)
| yumraj wrote:
| Does anyone know what tool was used to make these diagrams.. I
| really like the style.
| Existenceblinks wrote:
| I wasn't impressed but I look at it again .. it could be more
| pretty with right colors. Anyone has answer?
| prophesi wrote:
| This is just a tongue-in-cheek response that I have no stock in,
| but I find it funny that the page says "You don't have to
| reinvent the wheel" after they rewrite Erlang processes in
| another language.
| tete wrote:
| Being compatible with what's there is exactly the opposite of
| reinventing the wheel (mechanisms, protocols, etc.).
| prophesi wrote:
| That's true, but it'd be even simpler to implement a node in
| x language that can communicate with Erlang/OTP, and let
| Erlang work its fault-tolerant and scalable magic.
| gfodor wrote:
| This is why analogies like the wheel analogy are silly - to
| stretch it closer to reality, sometimes you are eyeing a
| wheel in Go and a wheel in Erlang, so you've gotta pick
| which one to port.
| moomoo11 wrote:
| dumbo here, what are some use cases for this?
| fishgoesblub wrote:
| I originally thought this had something to due with the IRCd
| written in Go also named Ergo https://github.com/ergochat/ergo
| lll-o-lll wrote:
| There was another recent Erlang post which posited the suggestion
| that the _power_ of erlang was not in lightweight processes and
| message parsing, but in generic components erlang calls
| _behaviours_. These generic patterns are what allows for building
| reliable distributed systems, and Joe Armstrong wrote a PhD
| thesis about it.
|
| I gave an anecdote of having built a c++ framework that
| unwittingly followed the same principles for "big corp" long ago
| (convergent design), and posited that the same concepts could
| probably be built in any language. Here it is for _go_. So is it
| a good thing?
|
| Previously I suggested it would be good to capture these core
| concepts in standardized libraries for a range of languages, but
| now I am not so sure. The problem I have is that as you start
| looking at these higher level patterns involving distributed
| computing, the landscape of frameworks seems to explode. Is this
| because the erlang/otp concepts are insufficient or outdated?
| jerf wrote:
| "Is this because the erlang/otp concepts are insufficient or
| outdated?"
|
| Insufficient or outdated is a pretty loaded term.
|
| What they are is really, really tied to the Erlang worldview,
| of immutable functions, mailbox APIs that can receive messages
| out of order, the way they have "behaviors" as a really limited
| kind of class system, and other details of Erlang. This isn't a
| criticism, it's really a good thing. An Erlang library _should_
| harmonize with Erlang, after all.
|
| But when you port them into environments that don't have all
| those characteristics, the optimal library that harmonizes with
| _that_ environment looks different. In Go, for instance, rather
| than the whole gen_server thing, the idiomatic approach to this
| sort of thing tends to look a lot more like:
| for { select { case msg1 := me.Chan1:
| me.handleMsg1(msg1) case msg2 := me.Chan2:
| me.handleMsg2(msg2) // etc. }
| }
|
| Trying to jam Go code into the gen_server framework is very
| klunky.
|
| Another reason is that while Go and Erlang both have fairly
| weak type systems, they are weak in _very_ different ways. It
| is idiomatic in Erlang to label what kind of message something
| is by making it the first atom in a tuple. Then you can use
| Erlang 's pattern matching in functions to implement many
| different message types based on what you send to the
| gen_server. This ports into Go terribly. Go does it in very
| different ways.
|
| "I suggested it would be good to capture these core concepts in
| standardized libraries for a range of languages, but now I am
| not so sure."
|
| No, it really doesn't work. Even two languages in the same
| basic family can end up with significant differences in
| patterns (witness Ruby versus Python, for instance, virtually
| the same language, significant difference in what is
| idiomatic). Trying to stretch a library from a language that is
| based on a super weak type system (on purpose; it is part of
| the Erlang solution to cross-version communication),
| immutability, pattern matching, and mailboxes into a language
| with a weak type system in a different way, mutability,
| conventional function calls, channels (very different than
| mailboxes), and all the other relevant differences may be
| theoretically possible, but it will always result in foreign
| environments in the language.
|
| You can actually see this in the real world by the way that C
| is the _de facto_ cross-language in-process bus on Linux. It
| bends everything that has to use it. Personally I consider it
| one of the biggest impediments to a new language getting off
| the ground because of the design decisions it forces on a new
| language at a young age. We are, slowly, getting past this in a
| variety of ways, but it 's hard.
| lll-o-lll wrote:
| This is interesting. When I think of a programming language I
| tend to consider it a _tool_ to express an idea. The _ideas_
| from Erlang Behaviours (not being an Erlang developer),
| seemed to be a fairly generalisable. So I would have thought
| the language is not so important.
|
| The example of C as the de facto cross-language standard
| (it's not just in Linux), is also interesting. Creating a C
| interface imposes on a library author a set of restrictions
| in expression, but is anything truely lost in doing this?
|
| I am much more interested in the concepts/ideas being
| expressed than I am the language being used to express it.
| di4na wrote:
| Re the C part. Yes. A lot. You lose all the expressivity of
| your type systems and the way you manage memory. Plus you
| have to deal with the lack of specification of the C ABI.
|
| See https://faultlore.com/blah/c-isnt-a-language/
| lll-o-lll wrote:
| You loose the _expressiveness_ , no argument there. The
| _poetry_ is not visible. Still, that's just a matter for
| a wrapper in the consuming language.
|
| E.g. _beautiful Rust wrapper_ - > "Ugly C interface" ->
| "Ugly C interop layer" -> _beautiful Go library_
|
| I mean, it's a pain to maintain "ugly C interop", but as
| these tend to be built after the fact anyway, other than
| cost I don't see anything lost.
| di4na wrote:
| Nope, you cannot because there is information lost in the
| middle that means that some of what you wanted to express
| in order to make it beautiful in Go is not there anymore.
| The medium is lossy and you cannot recover from it.
| lll-o-lll wrote:
| What is beautiful in one language is often hideous in
| another. Translation is required.
|
| Much like with human translations of poetry. A good
| translation is not word for word, the translator captures
| the _essence_ and _re-expresses_ it in the other
| language. When done well, it is frankly a work of genius.
|
| So the C interface is just a medium a "translation
| dictionary" if you will. To do it well requires the
| author of the wrapper to _express_ the ideas in a form
| palatable to users of the other language. I get that this
| is not always done well, requires a bunch of work, and
| may raise the question "why not re-write?". I'm happy to
| consider better alternatives.
| di4na wrote:
| You realise that the medium language is incapable of
| expressing that?
|
| And yes, there are wip to a superset ABI
| linuxftw wrote:
| The example for OPEN(2) isn't for Linux, it's for glibc.
| Languages are free to implement system calls and bypass
| glibc if they wish. This isn't the case on BSD as the
| libc implementation is the official interface, but nobody
| is forcing anyone to write C. The fact is, lot's of
| interesting software is written in C, and most people
| want to use it rather than re-implement things that have
| worked for decades.
| yumaikas wrote:
| They are fairly generalizable, to a point.
|
| The thing about Erlang behaviors is that they rely on
| several other pieces of Erlang to work _well_.
|
| One big one is being able to be notified when another
| process goes down, or is aborted.
|
| The other big one is being able to reason about state in
| the face of such failure.
|
| Erlang decided to go with immutable data structures, shared
| nothing processes, and the OTP behaviors generally expect
| you to decompose the behavior of the various bits into
| functions that represent a single atomic step.
|
| The more of those properties that you don't share with
| Erlang, the harder it will be to adopt OTP style semantics
| in your system.
| pdimitar wrote:
| Can you elaborate on how are Erlang process mailboxes
| different than channels, please? I view them as hear-
| identical but I haven't thought deeply about it.
| klabb3 wrote:
| Me too! I don't know Erlang enough, but Golang channels
| accept anything, including mutable data, file descriptors
| etc. If you want to send to different machines you need
| glue and refactoring in order to get some similar behavior,
| since channels are in-process only.
|
| My understanding is that Erlang require messages to be
| self-contained/immutable (even serializable?) which would
| make it possible to simply move an actor (process) to a
| different machine, or even multiple machines? If you know
| Erlang, please fact check!
| weatherlight wrote:
| Messages between Erlang processes are simply valid Erlang
| terms. That is, they can be lists, tuples, integers,
| atoms, other pids (process identifiiers), strings, and so
| on.
|
| However, Erlang and Elixir's AST is also composed of
| valid Erlang terms, so as I understand. you can send
| entire Erlang/Elixir programs as messages.
|
| To answer your question, Yes, it is possible, but there
| is no "Move process to node" call. However, if the
| process is built with a feature for migration, you can
| certainly do it by sending the function of the process
| and its state to another node and arrange for a spawn
| there. To get the identity of the process right, you will
| need to use either the global process registry or gproc,
| as the process will change pid.
|
| There are other considerations as well: The process might
| be using an ETS table whose data are not present on the
| other node, or it may have stored stuff in the process
| dictionary (state from the random module comes to mind)
| digilypse wrote:
| Maybe it's because the frameworks themselves are less important
| than the *human language used to describe them. To someone who
| knows what a Genserver is, or has used one before, what does it
| matter which of the many frameworks they're using?
| lll-o-lll wrote:
| It just seems like we reinvent the same things again and
| again. Maybe that's because at this level you can't have a
| standard and things need to be bespoke.
|
| It could, on the other hand, be an indication that there is a
| huge amount of wasted effort in revisiting long ago solved
| problems.
| digilypse wrote:
| I agree, I just think it's part of the evolutionary
| process.
|
| We reinvent them because the core concepts are right, but
| there isn't enough connective tissue to re-use the
| implementations directly.
|
| I don't see it as a bad thing. The concepts persist and
| gain momentum and with each iteration we get closer to
| something that can actually be relied on consistently
| without replication.
|
| There's wasted effort, but something about the original was
| missing that would have made it the obvious choice. I think
| it feels wasted mostly because it isn't yet obvious what
| the missing pieces were.
| lll-o-lll wrote:
| I just don't think this is true. Erlang has existed since
| the early 80s, open source in the late 90s. Joe Armstrong
| wrote his thesis in 2003. It's been 2 decades, and yet
| reliable distributed systems are still something
| companies fail at consistently. The latest trend with
| kubernetes and web interfaces everywhere is, in my
| opinion, an example of these ideas never gaining
| popularity so we outsourced it to infrastructure (then
| realised it needed code and made a whole new class of
| network/developer, devops).
|
| Erlang basically posited that "process isolation" as a
| building block was fundamental to reliable distributed
| systems in the presence of software errors. I don't think
| that's ever been really challenged, but the final
| solution we have today seems almost ludicrously
| inefficient in comparison. I just hope lunatic gets
| traction!
| kaycebasques wrote:
| I will forever think of Will Ferrell's parody of the Matrix
| whenever I hear "ergo": https://youtu.be/iRkiyy3EDl4?t=360
| cutler wrote:
| So that's it for Elixir then.
| gfodor wrote:
| Anything like this for Python?
| rcarmo wrote:
| Not for the patterns, but Thespian is a pretty decent actor
| library.
| jerf wrote:
| On the one hand, this framework's front page almost undersells
| the biggest feature it has, which is that I believe it can
| function as an actual Erlang node. You can use this to join a Go
| system to your Erlang cluster directly. This is pretty impressive
| because that's nontrivial functionality.
|
| On the other hand... this is a pure an example of a
| transliteration as an idiomatic port as I can think of. There is
| simply no need, in pure Go at least, to blindly copy every last
| detail of the Erlang implementation for gen_server for general
| usage, and more than you should copy every last such thing for
| Javascript or Python or any other language. If you _are_ joining
| to an Erlang node and working in the Erlang ecosystem, you have
| no real choice, but if you _aren 't_ doing that, this library is
| going to add a lot of friction to a lot of things in Go, all over
| the place.
|
| It's a very niche thing. If you need it it's probably worth all
| the money, but I'd pay to _not_ get it put into a greenfield Go
| project.
| Idk__Throwaway wrote:
| As someone without a lot of knowledge of GO and with only a
| trivial understanding of OTP/the gen_server impl, how would
| this add friction? What exactly is wrong with it and what would
| be a better alternative?
| jerf wrote:
| I go over this fairly deeply in
| https://www.jerf.org/iri/post/2930/ .
|
| The Big Idea is that it's rarely a good idea to transliterate
| a library. You should _translate_ a library. When I was done
| translating into Go, I found that what was idiomatic in Go
| was much simpler than Erlang. To some extent this is because
| Go simply _can 't do_ some things Erlang can do (in
| particular Erlang is one of the very short list of languages
| that can safely use "asynchronous exceptions", or to put it
| another way, can let one thread simply reach out and _nuke_
| another thread, so safely it can be incorporated into the
| core design of systems rather than treated as a dangerous
| special case)... but if that is the case, why port over the
| aspects of the Erlang design that are now dangling free in
| space, unconnected to anything?
| lll-o-lll wrote:
| Maybe the answer to the challenge of asynchronous
| exceptions is that "threads" are not well suited to robust
| parallelism.
|
| Certainly, I reached this conclusion in the past. A better
| paradigm is the "process". Unlike a thread a process can be
| killed arbitrarily (your asynchronous exception), and
| completely safely (all resources are released etc).
| Software can be composed of a collection of processes and
| thus be much more robust (e.g. Chrome). It's great that
| Erlang built this into the language itself, but any
| language can leverage processes for the same effect.
|
| As an aside, Lunatic (leveraging wasm to create "process"
| like sandboxes inside one process), looks very promising as
| a way of leveraging the power of process isolation without
| all the fuss.
| weego wrote:
| I cautiously agree, the only point I'd make from my own
| perspective is that the other reference point for this would be
| Akka, which was/is a behemoth of boilerplate and torturous
| implementation, so it begs the question of whether its a
| pattern that just has friction and is it worth trying to
| reinvent again.
| hinkley wrote:
| We did a book reading of a Scala book with a bunch of senior
| Java developers, and even they thought Scala was too baroque.
| If Java developers think something is too complicated you've
| really, really gone off the deep end.
|
| I've heard that Scala has spent a lot of time since then
| trying to simplify itself, so I can't speak to now, but I do
| know that Akka came into being when Scala was at its worst.
| When in Rome, do as the Romans.
| weatherlight wrote:
| Shouldn't the patterns transcend the language implementation?
| The fact that this exists in Go should be proof of that.
| avianlyric wrote:
| Go's concurrency model and Erlangs concurrency model are
| fundamentally the same.
|
| Erlangs mailbox and Go's channels are the same primitive.
|
| And Erlangs processes and Go's Goroutines are also the same
| primitive.
|
| The big difference between the languages (from a concurrency
| perspective, there are other differences) is that idomatic Go
| is generally built around spinning off many short lived
| goroutines passing objects back and forth via channels,
| whereas Erlang generally spins off many long lived processes
| that represent objects, communicating mutation requests via
| mailboxes.
|
| But you can do go style concurrency in Erlang, and Erlang
| style concurrency in go. You'll just be swimming upstream the
| entire time, because your concurrency style won't mesh well
| with any of the libraries out there which will be written in
| the idiomatic go/Erlang concurrency style.
|
| So the idea you can create an Erlang style concurrency
| framework in Go isn't actually surprising. Go and Erlang
| concurrency models are just opposite sides of the same coin.
| jibbit wrote:
| I actually think it's rather difficult to mimic Erlang
| message passing in go, but trivial (and not uncommon) to
| mimic go channels in Erlang.
| yumaikas wrote:
| So, the big difference between Go and Erlang style
| concurrency is that go style concurrency doesn't give you a
| readily available reliable way to keep an eye on other
| goroutines, as I understand it, where there are mechanisms
| in Erlang (links and monitors come to mind) to help
| accomplish that, and, in fact, it's one of the core parts
| of the Erlang model of concurrency
| throwawaymaths wrote:
| It's not about concurrency, it's about failure domains.
|
| The go VM doesn't natively let you take out monitors/links,
| so if "the right thing to do" when some part of a set of
| concurrent tasks fails is to "just give up" (suppose task
| 3/5 reaches out to an external service and comes back with
| a 500 error)... Handling the teardown of the whole thing
| (and potentially rollback state) is gnarly in go. In Erlang
| it's often zero lines of code. In Elixir, this architecture
| let the anointed db engine roll back a partially completed
| database transaction by default (0loc) in such a scenario.
| fpoling wrote:
| Idiomatic Erlang and Go are fundamentally different. In
| Erlang there is one mailbox per process and multiplexing is
| archived via polymorphic messages. In Go there can be
| multiple channels and one multiplexes them using the select
| statement.
|
| Erlang message queue is a priority queue, while in Go a
| channel is a strict FIFO. In fact implementing a priority
| queue on top of channels in Go is rather hard. A much
| better approach is to code a priority queue directly using
| mutexes.
|
| In Erlang (at least the last time I checked this) the
| message queue is unbound while Go gives a lot of control
| over the channel queue size.
|
| As was already pointed, Erlang also supports reliable
| cancelation of threads, while in Go a thread has to be
| coded explicitly to support cancelation.
|
| All of this leads to very different style of multithreaded
| code in the two languages.
___________________________________________________________________
(page generated 2023-01-28 23:00 UTC)