[HN Gopher] Erlang's not about lightweight processes and message...
___________________________________________________________________
Erlang's not about lightweight processes and message passing
Author : stevan
Score : 385 points
Date : 2023-01-27 11:56 UTC (11 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| hosh wrote:
| I think it is misleading to frame this as "behavior". I think
| what we really have are battle-tested concurrency patterns, which
| then uses the language feature called "behavior" to implement.
| This collection of battle-tested concurrency patterns that has
| been in use for a long time, in a variety of settings, is called
| OTP. It's why you'll hear long-time Erlang folks talk about how,
| it's really about OTP.
|
| You can implement those same patterns, if you also have
| lightweight processes, messages, and message queues. There's a
| Rust library that aims to do exactly that. The new WASM runtime
| (Firefly, formerly known as Lumen) aims to bring those patterns
| into the WASM ecosystem, potentially using this kind of
| concurrency pattern client-side.
|
| What's potentially novel is adding in newer concurrency patterns
| that are in use in the Kubernetes community -- use of selectors,
| services, scaling with replicasets -- so that, instead of a
| static supervision tree, it can be a dynamic supervision tree,
| spread out across the entire pool, even as that pool expands or
| shrinks.
| imachine1980_ wrote:
| Please, can you share the rust library??
| hosh wrote:
| I don't remember the name and I don't remember if it is tokio
| or something else. However, I did find this:
| https://docs.rs/genserver/latest/genserver/
|
| I will keep looking. Keep in mind, BEAM uses a preemptive
| actor rather than async, so it avoids the "storms" that can
| happen when an async reactor runs out of resources. EDIT: but
| I guess Tokio is capable of preemption through work-stealing
| too?
| verdagon wrote:
| Can you say more about these storms? Sounds pretty
| interesting. How does preempting solve it?
| hosh wrote:
| I don't know Python or Rust well, but I think this is the
| problem:
|
| https://betterprogramming.pub/the-dangers-of-async-in-
| python...
|
| And this is how some people are solving it:
|
| https://async.rs/blog/stop-worrying-about-blocking-the-
| new-a...
|
| This is how it works in BEAM:
|
| https://github.com/happi/theBeamBook/blob/master/chapters
| /sc...
|
| The commonality is to limit execution time and yield
| (preempt) so something else can run too.
| hosh wrote:
| This looks interesting too, though no OTP out-of-the-box:
| https://docs.rs/axiom/latest/axiom/
| bkolobara wrote:
| I'm the author of something related
| https://github.com/lunatic-solutions/lunatic
| [deleted]
| linkdd wrote:
| Not sure this is what GP is talking about but to implement
| the actor model in https://letlang.dev I use tokio.
| mcphage wrote:
| > This collection of battle-tested concurrency patterns that
| has been in use for a long time, in a variety of settings, is
| called OTP.
|
| OTP seems like a pretty overloaded acronym--what does it mean
| here?
| butterisgood wrote:
| Or Outlaw Techno Psychobitch for the Bananarama of Languages.
| https://www.youtube.com/watch?v=rRbY3TMUcgQ
| jaggederest wrote:
| OTP in an Erlang context refers to the Open Telecom Platform
| - a bit of a misnomer since you can do a lot of things with
| it that do not inherently relate to telecoms, but that's how
| it evolved.
| chc wrote:
| For context, Erlang itself was originally created at the
| telecom company Ericsson -- the name is short for " _Er_
| icsson _Lang_ uage". But as concurrency and networking
| became bigger parts of software in general, the qualities
| that made Erlang good for telecom software turned out to be
| generally useful.
| gilbetron wrote:
| Open Telecom Platform, I think?
| hinkley wrote:
| I started using worker threads for some batch processing and
| development tools recently. We have a couple of tricky bits
| that pull data from different environments to compare them, so
| the code has to run 'as' two or more environments and forcing a
| state change was bug ridden as hell, and spawning a child
| process was slow enough that we limited the scope of how much
| we scanned (running n worker threads for n environments dropped
| a lookup from 1.5s to 350ms).
|
| Armed with those successes I turned to thinking about applying
| something similar to a production Node app, and every time I
| think about it I realize what I really want is OTP. Each type
| of processing we do has its own caches, and its own CPU uses,
| which _could_ dovetail well with worker threads, but I have a
| chicken and egg problem with trying to move any of that work
| out of process because I don 't get any benefit until 1) none
| of the workers can silently crash and leave the application
| nonfunctional and 2) before I'm handling half a dozen
| concurrent requests per Node process instead of one or two. and
| 3) before message passing overhead is cheaper
|
| It's either too late or too early to rewrite this whole thing
| in Elixir.
| hosh wrote:
| Would running Nodejs in WebAssembly along with something like
| Firefly or Lunatic help make the transition smoother?
| orf wrote:
| Perhaps Node is the problem, rather than Elixir being the
| solution?
| toolz wrote:
| If node is the issue here that would make elixir a
| solution, no? Changing languages (i.e. node is the issue)
| to get the features described would put you looking at
| elixir (among other languages that can fill the described
| properties).
|
| Maybe what you mean is that elixir isn't the only solution?
| orf wrote:
| Yeah, that's what I meant: "elixir being _the_ (only)
| solution". Guess it was worded ambiguously.
| hinkley wrote:
| Sometimes your problem is that you're using a reciprocating
| saw when you should be using an angle grinder, sometimes
| your problem is that you bought Ryobi instead of Milwaukee
| or Makita.
| orf wrote:
| Indeed, but sometimes you just need to use the right tool
| for the job. That could be a precision guided flying
| laser chainsaw axe. Usually it's a bog standard trusty
| dumb hammer that everyone else manages to bash similar
| looking nails with quite effectively.
| dangoor wrote:
| Here's an attempt at implementing OTP in Go:
| https://github.com/ergo-services/ergo
| metadat wrote:
| Why do you say "attempt"? Does it not work sufficiently well
| [yet]?
|
| Edit: Thanks for the additional context, dangoor. Cheers.
| dangoor wrote:
| I haven't tried it, it's just on my radar. It may be great!
| But OTP has been around a long time, so I'm not going to
| assume that they've nailed it unless I've either tried it
| myself or seen folks talking about using it successfully.
| dmitriid wrote:
| The main insight is in Joe Armstrong's thesis. "Making reliable
| distributed systems in the presence of software errors"
| https://erlang.org/download/armstrong_thesis_2003.pdf (I think
| the original title was "In the presence of _hardware_ errors ",
| emphasis mine)
|
| All the other things flow from that thesis and understanding. You
| can recreate behaviours described in the repo doc using Erlang
| primitives very easily, and they are very hard to recreate in
| pretty much any other language.
|
| Because Erlang is very much literally about lightweight processes
| and message passing. Only:
|
| - every part of the system knows about lightweight processes and
| messages
|
| - every part of the system is engineered around them
|
| - the underlying VM is not only re-entrant in almost every single
| function, it's also extremely resilient, and can almost guarantee
| that when something dies, only that process dies, and the rest of
| the system receives a guaranteed notification that this process
| dies
|
| There are more, but without at least that you can't really
| recreate Erlang's standard behaviours. For example, you can't
| recreate this in Go or Rust because `panic` will kill your entire
| program unconditionally.
| neonsunset wrote:
| Most modern async runtimes let you do that.
|
| A (worker) thread dying isn't an issue in Rust, Go, C# and etc.
| Sure, each goes about error handling in a slightly different
| way either opt-in or opt-out or enforced (when unwrapping
| Error<T, E>) but other than that the advantages of
| Erlang/Elixir have faded over time because the industry has
| caught up.
|
| p.s.: C# has not one but two re-entrancy syntax options -
| 'async/await' and 'IEnumerable<T>/yield return'. You can use
| latter to conveniently implement state machines.
| jayd16 wrote:
| C#'s weakness here is that those two patterns are cooperative
| multitasking only. Under the hood they retain control of a
| thread until they yield execution. By default resource
| management is something that needs to be considered and the
| default thread pool is not an uncontested resource.
|
| I don't use Erlang but my understanding is that while it is
| not exactly fully pre-emptive, there are safeguards in place
| to ensure process fairness without developer foresight.
| neonsunset wrote:
| C# async runtime is mixed mode, threadpool will try to
| optimize the threadcount so that all tasks can advance
| fairly-ish. This means spawning more worker threads than
| physical cores and relying on operating system's thread
| pre-emption to shuffle them for work.
|
| That's why synchronously blocking a thread is not a
| complete loss of throughput. It used to be worse but
| starting from .NET 6, threadpool was rewritten in C# and
| can actively detect blocked threads and inject more to deal
| with the issue.
|
| Additionally, another commenter above mistakenly called
| Rust "bare metal" which it is not because for async it is
| usually paired with tokio or async-std which (by default,
| configurable) spawn 1:1 worker threads per CPU physical
| threads and actively manage those too.
|
| p.s.: the goal of cooperative multi-tasking is precisely to
| alleviate the issues that come with pre-emptive one. I
| think Java's project Loom approach is a mistake and made
| sense 10 years ago but not today, with every modern
| language adopting async/await semantics.
| dmitriid wrote:
| > but other than that the advantages of Erlang/Elixir have
| faded over time because the industry has caught up.
|
| It really hasn't. People fixate on the idea of _just_ running
| some processes, and _just_ catching _some_ errors.
|
| And yet, non of the languages that "solved this" can give you
| Erlang's supervision trees that are built on Erlang and
| Erlang VM's basic functionality. Well, Akka tried, and re-
| implemented half of Erlang in the process :)
|
| But other advantages did fade: multi-machine configurations
| are solved by kubernetes. And it no longer matters that you
| can orchestrate multiple processes doing something when even
| CI/CD now looks like "download a hundred docker containers
| for even the smallest off tasks and execute those".
|
| > p.s.: C# has not one but two re-entrancy syntax options -
| 'async/await' and 'IEnumerable<T>/yield return'.
|
| What I meant by re-entrancy in the VM is this:
|
| Every process in Erlang gets a certain number of reductions.
| Where a reduction is a function call, or a message passed.
| Every time a function is called or a message is passed, the
| reduction count for that process is reduced by one. Once it
| reaches zero, the process is paused, and a different process
| running on the same scheduler is executed. Once that reaches
| zero, the next one is executed etc.
|
| Once all processes are executed the reduction counter is
| reset, the process is woken up and resumed.
|
| On top of that, if a process waits for a message, it can be
| paused indefinitely long, and resumed only when the message
| it waits for arrives.
|
| So, all functions that this process executes have to be re-
| entrant. And it doesn't mean just the functions written in
| Erlang itself. It means _all_ functions, including the ones
| in the VM: I /O (disk, network), error handling, date
| handling, regexps... You name it.
| gamache wrote:
| Sometimes the advantage is not just the ability to do
| something -- I mean, by that logic, only bare-metal systems
| languages like Rust, C, Zig etc could have advantages over
| other languages.
|
| Erlang (and by extension Elixir) has the advantage that the
| programmer can think at a higher level about their system.
| You don't have to write or configure a scheduler. You don't
| have to invent supervision trees. You can be sure that the
| concurrently-running parts of your system cannot possibly
| affect each other's memory footprint (though Rust gives a
| robust answer to this problem as well).
|
| It doesn't make a perfect fit for every problem, but there is
| still a decent-sized space of problems -- I'd say "highly
| concurrent, but not highly parallel" -- where Erlang gives
| the programmer a headstart.
| valenterry wrote:
| The thing is, we mostly try to keep applications stateless,
| unless it's necessary to keep state for performance (think
| realtime onlinegames, hft, ...).
|
| And in those cases there is simply no need for e.g.
| supervision trees because there is nothing to restart. You
| still need stuff like retrying, but this is supported by
| all major concurrency libraries / effect systems. In fact,
| Erlang has fallen behind here in terms of what the language
| offers (not what the BEAM offers, the BEAM is still top
| notch imho)
|
| State is mostly moved to either the database or message
| queues or similar, which is pretty good.
| OkayPhysicist wrote:
| We try to keep applications stateless because state
| handling in most programming languages is pathological.
| Erlang presents a different solution to the underlying
| problem of state management, rather than the trying to
| solve the higher-level of "how best to support this one
| particular solution to the problem of state management".
|
| Namely: The message queues are part of the language.
| They're built right in, for ease of use. Your caching
| layer is built into the language, for ease of use and
| faster performance.
| valenterry wrote:
| > We try to keep applications stateless because state
| handling in most programming languages is pathological.
|
| It's true that handling state is hard in most programming
| languages. But that's neither the only nor the most
| important reason why people try to keep applications
| stateless.
| ThemalSpan wrote:
| I agree up to the last point. You can catch panics at the
| thread level right? I mean, more generally, what is Erlang
| implemented in?
| Idiot_in_Vain wrote:
| >> I mean, more generally, what is Erlang implemented in?
|
| Pure magic, that can't be recreated in any regular language
| :)
| dmitriid wrote:
| > I mean, more generally, what is Erlang implemented in?
|
| "Any sufficiently complicated concurrent program in another
| language contains an ad hoc informally-specified bug-ridden
| slow implementation of half of Erlang." - Virding's Law ;)
|
| So first you have to implement all those things. And then use
| them.
| d3ckard wrote:
| In C. You don't operate on thread level directly, threads are
| just executors of processes managed by VM, usually the same
| amount as vCPU cores.
| octacat wrote:
| You can in C/C++. Though, the point is that erlang has a
| strategy how to clean and recover from such error (some data
| can be lost, rarely we can crash the whole system for some
| errors).
|
| I mean, you can write erlang-way in almost any language, just
| in this case you need to adopt all the libraries to follow
| the same principles.
|
| Some languages implement similar error handling strategy by
| just creating a separate process per request (hello, php). We
| know how to clean after a worker dies (just let the worker
| die). Just in that case supervisor strategy is very simple.
| Arch-TK wrote:
| Erlang is implemented on top of BEAM which has its own
| scheduler and schedules its own idea of lightweight
| processes.
| dolmen wrote:
| It looks like features that the Go runtime provides.
| ModernMech wrote:
| Of course you could implement a language like Erlang in Rust,
| but I think the point is that you would have to do exactly
| that in order to do in Rust what Erlang does at the language
| level.
| EdwardDiego wrote:
| 18 or so years ago, I was a phone monkey for a government
| department who was initially learning to code for fun, but
| quickly realised programming could be a great way to earn a
| living.
|
| I read about Erlang on /r/programming, which was having a new
| fad for a new shiny language, as was the custom back then.
|
| And I desperately followed all the /r/programming fads because
| I was worried that I'd end up irrelevant if I wasn't skilled up
| in the latest Haskell web framework.
|
| But one of those fads was Erlang, and it intrigued me so much,
| that I ended up printing Mr Armstrong's thesis, stuck it in a
| manila folder, and read it, slowly, cover to cover while
| waiting for, or riding on, my bus to my contact centre job, and
| I've still got it in my bookcase.
|
| His thinking on resiliency in the face of inevitable failures,
| and on safe concurrency, has shaped my thinking, and proved
| invaluable repeatedly and is very relevant today. It's like
| their phone switches were distributed microservices running in
| a container orchestrator, before it was cool.
| gjvc wrote:
| _But one of those fads was Erlang, and it intrigued me so
| much, that I ended up printing Mr Armstrong 's thesis, stuck
| it in a manila folder, and read it_
|
| I am now inspired to do the same, thank you!
| ilyt wrote:
| > For example, you can't recreate this in Go or Rust because
| `panic` will kill your entire program unconditionally.
|
| You can recover() panic() just fine in Go.
| ramchip wrote:
| Yeah, OP took a shortcut here :)
|
| The nuance is that most languages let you build reliable
| programs _if your code is correct_ - if you're using defers,
| context handlers, finalizer, cleaning up state in shared data
| structure, etc.
|
| Erlang's goal is to be reliable "in the presence of software
| errors", that is, even if your code is buggy. If a request
| handler process dies inside an Erlang web app, whatever file
| or socket it opened, memory it allocated, shared resource it
| acquired (e.g. a DB connection) will be reclaimed. This is
| true without having to write any error handling in your code.
|
| The way it's done is that the VM handles the basic stuff like
| memory and files, and it provides signals that libraries
| (like a DB connection pool for instance) can use to know if a
| process fails and clean up as needed. In other words the
| process that fails is not responsible for its own clean up.
|
| At some point _some_ code must of course be correct for this
| to work. Like, if the DB connection pool library doesn 't
| monitor processes that borrow connections, it could leak the
| connection when such a process dies. But the point is that
| this part (the "error kernel") can be a small, well-tested
| part of the overall system; whereas in a classic program, the
| entire codebase has to handle errors correctly to guarantee
| overall stability.
| dmitriid wrote:
| These are like sledgehammers when you need a screwdriver.
|
| It's very hard to build Erlang-like versions of supervision
| trees using those tools.
| masklinn wrote:
| > These are like sledgehammers when you need a screwdriver.
|
| Also by the very nature of Go your application state is as
| likely as not to be fucked, so even if you did handroll
| monitors and links, and thus could build supervision trees,
| they wouldn't be of much use.
| komadori wrote:
| > You can recover() panic() just fine in Go.
|
| You can catch_unwind() panic!() in Rust too :-).
| wongarsu wrote:
| > For example, you can't recreate this in Go or Rust
|
| If you're willing to make your "lightweight processes" OS
| threads you could kind of make it work. E.g. Rust gives you
| both panic hooks (to notify everyone else that you died) and
| catch_unwind to contain the effect of a panic (which generally
| stops at a thread boundary anyways). But of course that only
| scales to a couple hundred or thousand threads, so you probably
| have to sacrifice a lot of granularity.
|
| And any library that links to C/C++ code has the potential to
| bring the whole process down (unless you make your "lightweight
| processes" just "OS processes", but that just makes the scaling
| problems worse)
| jjtheblunt wrote:
| Lightweight processes is explicitly not os threads in at
| least two senses: smaller footprint in memory, and no system
| call for every context switch.
|
| It's explained in many documents about lightweight processes,
| of course for elixir/erlang/beam but also for Go and Crystal
| and even going back to Solaris Internals and modern Project
| Loom for upcoming JVM situations
| wongarsu wrote:
| Erlang's beam processes are undoubtedly awesome. But if we
| start with the premise of "achieving the same goals without
| Erlang" I think it's entirely valid to start with "what
| does our process primitive look like". With just 16GB of
| RAM my laptop is quite memory constrained, but according to
| task manager I'm still running 5200 threads accross 350
| processes right now. Many use cases that required light
| threads/processes 37 years ago or even 20 years ago would
| work with OS threads by now. Of course many others don't,
| which is where the Erlang popularity comes from.
| dmitriid wrote:
| It takes on the order of nanoseconds to start an Erlang
| process. They are also extremely lightweight memory-wise.
| And Erlang VM tries to keep context switching between CPU
| cores to a minumum. And all processes get more-or-less
| equal share, so it's hard to get a process consuming all
| of CPU and never yielding back to other processes.
|
| So firing off and monitoring processes becomes second
| nature easily. Rarely so in other languages
| brightball wrote:
| OS threads would defeat the purpose though. You can run
| millions of BEAM processes. Threads are a lot more expensive
| and still run the risk of taking over your system with an
| infinite loop.
| toast0 wrote:
| OS threads are actually premptive, so an infinite loop is
| really less of a deal than one in Erlang. Erlang is really
| not premptive, it's just that function calls are
| automatically (potential) yield points, and you can't loop
| without function calls because of the construction of the
| language; if somehow you did manage to get a real loop into
| a process, that scheduler would be stuck forever, and you'd
| end up with a broken BEAM when you triggered something that
| requires coordination between schedulers --- I forget what
| sorts of things do that, but code purging can't finish
| without each scheduler doing housekeeping, etc.
|
| Meanwhile, your OS thread will just keep eating CPU, but
| everything else is fine.
|
| You're right about the number of threads though. It takes a
| lot more memory to run an OS thread than a BEAM process,
| and I'm not sure OS schedulers will manage that many
| threads even if the memory is there (but I could be
| wrong... getting things to work nicely at 50k threads may
| be sufficient for millions)
| brightball wrote:
| Unless code is exiting the BEAM itself, an infinite loop
| that bypasses yield points shouldn't be possible.
|
| Doing this at such a granular level is also one of the
| reasons that you can run a database that thousands of
| processes are accessing, within the same runtime, without
| a performance impact to the overall system.
| JustSomeNobody wrote:
| I get that this is probably written for people familiar with
| Erlang, but a quick suggestion if I may:
|
| In the Behaviors section you talk about behaviors being similar
| to interfaces in Go and give a Go example. But then you switch
| examples for Erlang. Maybe show the Joe/Mike example written in
| Erlang and _then_ say, here 's a more complicated example (the
| key-value example) that really describes behaviors better.
| worthless-trash wrote:
| One thing I like about "behaviours" is that I can immediately
| recognise, discard boiler plate, skim quickly, and see the
| difference from expected pattern just by knowing the behaviour
| that the process implements.
| revskill wrote:
| What i learn from Erlang, is, naming is hard.
|
| What does `gen` mean in `gen_server` ?
|
| Thanks for replies for "Generic". So this article is not well
| organized in that case. With some simple naming explanation, it
| should be obvious to guess what the system does.
|
| Abbreviation doesn't save the writings.
|
| Some more examples, in Ruby, instead of calling `implement
| Module`, it uses `include Module` . I do think include is not as
| clear as implement.
| mtremsal wrote:
| generic, I believe. i.e. it's an interface that has to be
| implemented.
| sodapopcan wrote:
| Yep, "generic" it is.
| [deleted]
| dolmen wrote:
| generic?
| Lammy wrote:
| > Some more examples, in Ruby, instead of calling `implement
| Module`, it uses `include Module` . I do think include is not
| as clear as implement.
|
| Agreed, though I do think the `include` naming might just be a
| relic from a time before `prepend` and `extend` also existed in
| Ruby. I find it a lot more intuitive when I remember it as
| `append`ing to the ancestors, and the language even _sort of_
| calls it that internally with the naming mismatch between
| `include` and `append_features`: https://ruby-
| doc.org/3.2.0/Module.html#method-i-include
|
| Then one can see that's exactly what it does to a Module's
| ancestor chain: irb(main):001:0> RUBY_VERSION
| => "3.2.0" irb(main):002:0> lol = ::Module::new =>
| #<Module:0x00007f2969821740> irb(main):003:0>
| lol.ancestors => [#<Module:0x00007f2969821740>]
| irb(main):004:0> lol.singleton_class.ancestors =>
| [#<Class:#<Module:0x00007f2969821740>>, Module, Object,
| PP::ObjectMixin, Kernel, BasicObject] irb(main):005:0>
| rofl = ::Module::new => #<Module:0x00007f296982dfe0>
| irb(main):006:0> lmao = ::Module::new =>
| #<Module:0x00007f296982fd40> irb(main):007:0> omg =
| ::Module::new => #<Module:0x00007f2969822a00>
| irb(main):008:0> lol.include(rofl) =>
| #<Module:0x00007f2969821740> irb(main):009:0>
| lol.prepend(lmao) => #<Module:0x00007f2969821740>
| irb(main):010:0> lol.extend(omg) =>
| #<Module:0x00007f2969821740> irb(main):011:0>
| lol.ancestors => [#<Module:0x00007f296982fd40>,
| #<Module:0x00007f2969821740>, #<Module:0x00007f296982dfe0>]
| irb(main):012:0> lol.singleton_class.ancestors =>
| [#<Class:#<Module:0x00007f2969821740>>,
| #<Module:0x00007f2969822a00>, Module, Object, PP::ObjectMixin,
| Kernel, BasicObject]
| lopatin wrote:
| I always thought it was "generate" and that something about the
| implementation/immutableness required the server to be
| "generated" at runtime. I never did any actual Erlang
| programming, it's just how my brain plugged in the gaps when I
| heard about all the gen stuff.
| bkolobara wrote:
| I also dislike this name, as it was not clear what it does just
| by looking at the name.
|
| Recently, I was adding a similar abstraction to Rust on a
| project I'm working on and I called it `AbstractProcess`. Like,
| it is some kinda of template for a process. Still don't think
| it's clear what it does by just looking at the name. Does
| anyone have a better idea on how to name such a pattern?
| revskill wrote:
| Could also be called "ProcessTemplate" ?
| gen220 wrote:
| The suffix `Factory` often gets made fun of, but there are a
| few places where it makes sense. Sounds like this is one of
| them.
| imtringued wrote:
| Factory went out of favor. Nowadays people just add an s as
| a suffix in Java.
| mabbo wrote:
| If they called it 'generic_server' then people would make fun
| of it for being too verbose. If they call it 'gen_server' then
| everyone pretends they know what it means and stays quiet.
| edgyquant wrote:
| gen_server to me is "generate a server"
| layer8 wrote:
| I associate "gen" more with "generation", in the sense of
| "gen Z". In any case, it's confusing.
|
| With code completion you shouldn't have to resort to
| abbreviations (unless they are already part of the
| (business) domain language). This was different in the
| 1980s though.
| crad wrote:
| Erlang vs OTP
| practal wrote:
| Very interesting read. Also very timely for me (I am always
| amazed how HN often has posts that resonate strongly with what I
| am currently doing), as I am just now designing a programming
| language based on a generalisation of Algebra, which I call
| Abstraction Algebra. I think there is a strong connection to this
| post: Behaviours are just an abstraction algebra you program
| against. Switch out the algebra implementation against another
| one, and you can adapt the same program to different scenarios
| without changing the program itself.
| survirtual wrote:
| I always thought it would be neat to have a system that takes
| math symbols / formulas / functions etc in the notation of a
| mathematician, and automatically generates highly performant
| implementations -- sort of like Taichi but instead of using
| python, you just use mathematics.
|
| Seems to me all the tools are available to accomplish this
| effectively now days.
|
| Good luck on your project, in any case.
| practal wrote:
| Thank you!
|
| Yes, a large part of programming will be just a special case
| of doing mathematics.
| what-no-tests wrote:
| > I am just now designing a programming language based on a
| generalisation of Algebra, which I call Abstraction Algebra.
|
| I admire your ambition!
| practal wrote:
| To be honest, I am quite amazed by how nicely all the pieces
| are coming together now. It all just feels right, it feels
| like collecting fruit from under a huge apple tree.
| ComputerGuru wrote:
| I don't mean to disparage the article (quite the contrary, it's
| quite the eye opener if you're not accustomed to architecting
| your code in this fashion) but choosing go of all languages to
| mockup the ideas was a terrible choice. It's syntax and type
| system are really quite the red-headed stepchild and for anyone
| unfamiliar with its syntax, its incredibly hard to make heads or
| tails of.
|
| (And I say this as someone that's at least passably familiar with
| go and have worked on/contributed to golang projects in the
| past.)
| rahilb wrote:
| As far as I can tell the article contains only erlang snippets
| directly quoted from the paper reviewed.
| preseinger wrote:
| I think you may be alone in this assessment. Go is uniquely
| easy to understand, due to its small set of keywords and lack
| of esoteric e.g. sigils.
| weatherlight wrote:
| Yeah, I had no idea how to read the Go code.
|
| I'm also coming from a ruby/js/ocaml/elixir/erlang
| background.
| sb8244 wrote:
| > just like the concurrent code of gen_server
|
| I found this a really interesting read, but this stuck out
| because it doesn't jive with my mental model of gen_server.
|
| gen_server is fully serialized. Even the code underpinning it is
| not concurrent.
|
| Now I guess gen_server does expose some top level functions to
| simplify sending/receiving a message into the process, but the
| process itself is serial.
|
| And this is part of the genius of gen_server to me. You don't
| need to think about your state concurrently because it processes
| a single message at a time. Your system executes concurrently,
| but the individual components do not.
|
| Maybe that is what the post means and I misinterpreted it.
| btbuildem wrote:
| Yup, that's my understanding too -- what Erlang does extremely
| well is abstract away and isolate the concurrency mechanisms,
| allowing you to think of the details of (these concurrent)
| processes, linearly.
| sb8244 wrote:
| I remember when I was learning Elixir--I asked my co-worker /
| Elixir mentor how Elixir solves concurrent programming, data
| access, etc. His cheeky but accurate response was "How do you
| solve a problem that you don't have?"
|
| His point was that the paradigm is so different at the VM
| level, that many things become irrelevant to the
| conversation. That said, there still are concurrent
| programming challenges on the BEAM, but it's very minimal
| compared to languages where it's not baked in.
|
| Ruby Ractor is a good example of how a VM-backed concurrency
| mechanism will likely change how programs in that language
| can be built.
| wefarrell wrote:
| Yeah they mention that in 2 of the 6 points arguing in favor of
| behaviors:
|
| 2. The application programmer writes sequential code, all
| concurrency is hidden away in the behaviour;
|
| 4. Easier for new team members to get started: business logic
| is sequential, similar structure that they might have seen
| before elsewhere;
| sb8244 wrote:
| Thanks for that. My thought is probably just semantics then.
|
| I think it's even more nuanced, but doesn't really matter for
| this type of post.
|
| (I am actively writing a chapter exploring this topic, so is
| has been top of mind.)
| toast0 wrote:
| Adding on to sb8244's comments; writing sequential code is
| _very_ nice, but it 's not the gen_server behavior that gets
| you that. It's the lightweight processes.
|
| gen_server is useful, but it's not much more than receive in
| a tail recursive loop and conventions around messages, most
| specifically about if the sender expects to receive a reply
| or not. It's not magic, and it's written in clear Erlang that
| anyone with a week of Erlang fiddling can understand (which
| kind of is magic; almost everything in OTP is clearly
| readable)
|
| Concurrency comes from running many processes each with their
| own message queue.
| octacat wrote:
| gen_server is a serializer, it is the point of it to begin
| with. Could be called gen_serializer ;) Because we need to
| update the state inside of it without locks.
|
| It is usually fine, for parallelism just use a pool of
| gen_servers or many gen_server processes that would map to your
| data model well (example, one gen_server for each network
| socket).
|
| Though, these properties of gen_server come from the fact it is
| just a single erlang process.
| davidw wrote:
| > just use a pool of gen_servers or many gen_server processes
|
| I think architecting an Erlang system can be kind of tricky
| because you do have to think about all these processes sort
| of co-existing at the same time and how they interact. With
| one system I was involved with, we weren't really satisfied
| with how we'd divided things up among different behaviors,
| and did some refactoring. It wasn't difficult, but it did
| require a different way of thinking about things. When we had
| it all working, I was really satisfied with the results
| though. That thing was robust, and quite resilient.
|
| I miss working with Erlang. It's tricky to find people who
| are using it or Elixir in the sweet spot, IMO. Lots of "it
| looked cool and we wanted to play around with it" out there,
| as well as some people thinking it'll magically make their
| systems "internet scale".
|
| My best experiences with it have been semi-embedded systems
| where it's not doing too much distributed computing, but
| where "robust" and "predictable" are important qualities.
| toast0 wrote:
| > I think architecting an Erlang system can be kind of
| tricky because you do have to think about all these
| processes sort of co-existing at the same time and how they
| interact.
|
| I find it easier to think about each process in isolation
| --- what messages does it get, and what does it do with
| them; if it needs to send messages, who/where does it send
| them to and what will it get back, including errors or
| timeouts and not worrying about what the other process does
| in the moment.
|
| The overall behavior of a system built from communicating
| processes can get hard to predict though. You build up
| observations, intuition, and escape hatches over time.
| jFriedensreich wrote:
| It makes me sentimental to see some erlang code, truly miss
| writing erlang, as its probably the closest to how i think about
| systems. But it is so hard to justify using it when most of my
| problems can be solved with javascript edge workers. These days
| it's rare for me to even come across a backend heavy project, yet
| something that would fit perfect to erlang. Does anyone else fell
| this way or is it because of my drift in the industry? When was
| the last project you started and thought this is perfect to solve
| with erlang?
| noloblo wrote:
| We need backend heavy erlang prototype
| yurishimo wrote:
| Arguably that's what Elixir is. They're really trying to push
| Phoenix as a batteries included full stack solution. I know
| it's not exactly Erlang, but the interfaces for things like
| gen_server are nearly identical. It's effectively Erlang with
| a more modern syntax.
| throwawaymaths wrote:
| Its also got much better tooling, and a few features that
| don't exist in Erlang yet (Tasks - and no, tasks are not
| just proc_lib, they're actually special.
| peter_d_sherman wrote:
| >"The next interesting behavior is supervisor. Supervisors are
| processes which sole job is to make sure that other processes are
| healthy and doing their job. If a supervised process fails then
| the supervisor can restart it according to some predefined
| strategy."
|
| This is an interesting software pattern -- especially applied to
| having it entirely supported entirely inside of one programming
| language... We see this software pattern in such things as
| Windows Services (they restart automatically if they fail),
| Unix/Linux daemons, as one of the original purposes of the
| original Unix 'init' process, and in high-availability, mission-
| critical 24/7 systems (databases, etc.).
|
| Having all of that fault-tolerant infrastructure entirely inside
| of a programming language is indeed, somewhat novel.
|
| But let's work up what a given language must have as
| prerequisites to support this...
|
| First, we need the ability to run multiple processes inside of a
| language. Many languages can accomplish this with threads, but of
| the languages that do this, many need additional cumbersome
| programming to guarantee thread safety...
|
| So the language needs to support the notion of a threaded process
| inside of that code -- without having to code extra to support
| this threaded process.
|
| Next, the language needs some form of communication between
| supervisor (process A) and supervised (process B) code. Message
| passing is the solution -- but that requires each process to have
| its own message queue.
|
| And message passing requires interfaces...
|
| Here we're starting to sound like we're duplicating a mini
| Operating System inside a language(!) (should the language have
| pipes, too?) -- and/or that the complexity of most OS's would be
| deeply mitigated by designing them inside of a language that
| supported these constructs...
|
| Whatever the case, I think it's a fascinating software pattern.
|
| Yes, Go exists and supports features like this, Rust exists, and
| people are using it to write Operating Systems. And there are
| probably all other kinds of languages (both existing and existing
| in the future) that do/will support some form of these
| constructs...
|
| But there's another interesting, purely academic, purely
| theoretical question here...
|
| That is: _What is the simplest full-featured OS (processes, IPC,
| synchronization primitives, memory management, hardware resource
| management, syscalls, scheduling, etc.) that could be written in
| the smallest amount of lines of code if the language it was
| written in had intrinsic knowledge of all of those underlying
| constructs?_
|
| Anyway, a fascinating article about Erlang, and it definitely
| gave the impression that Erlang was a whole lot more than I
| thought it was... I will be checking out Erlang for future
| projects!
| brudgers wrote:
| I agree with the headline because uptime was the goal of Erlang.
|
| Lightweight processes using message passing is how Erlang stays
| up (and perhaps ironically, let-it-crash is how Erlang avoids
| going down).
|
| Lightweight processes and message passing are the architectural
| decisions that address the business problem Erlang was designed
| to solve.
| adamddev1 wrote:
| > In 1998 Ericsson decided to ban all use of Erlang.
|
| Does anyone know why?
| rr808 wrote:
| I work on a project that uses a proprietary language. Its very
| good but its miserable to be the only place where its used.
| Tools are sloppy and old, no external documentation or videos,
| can't hire anyone who knows it, and the people on the team are
| half committed because they have to be but also want to keep
| with industry trends for their next job.
| layer8 wrote:
| There might be a case for open-sourcing it.
| SanjayMehta wrote:
| Management decided it was easier to hire c++ programmers.
| rhodin wrote:
| Yes, they built a platform Cello [0] that would allow for
| that if I remember correctly
|
| [0] https://web.archive.org/web/20170829230730/https://www.er
| ics...
| doikor wrote:
| Easier to find C/C++ devs according to one friend who used to
| work there in early 2000s.
| [deleted]
| tiffanyh wrote:
| C++ was all the rage in late 90s / early 00s.
| bluGill wrote:
| Java was all the rage then. C with classes was dominate
| (not to be confused with C++), but java was the rage. Just
| like today Rust is all the rage, but it isn't as popular as
| C++ in the real world.
| FpUser wrote:
| I remember that. What puzzled me that it was the time
| when software written in Java ran up to 100 time slower
| than native processes. Now it is not nearly as bad.
|
| I had very interesting reaction from CTO of one Telecom
| when I showed them a prototype running on PC handling the
| same amount of transactions without breaking sweat as
| their Java backend equivalent that ran on beefy HP
| servers.
| ethbr0 wrote:
| Java is a language designed so that IBM can shovel
| interchangeable programmers into a furnace and produce
| something functional.
|
| Once you start with that goal, all the compromises and
| quirks make more sense.
| jimbokun wrote:
| Java is also a language with extensive library support,
| and some of the most sophisticated tooling and virtual
| machines with state virtual machine and garbage
| collection implementations of any language. The most
| recent versions of the language are much more programmer
| friendly than previous versions, greatly cutting the
| expressibility gap with other languages.
|
| Just stay away from Spring, and you can have a nice
| development experience.
| bluGill wrote:
| The topic here isn't modern Java, it is last 90s early
| 2000s java. Spring was either not invented yet, but the
| thought process behind it was already in place, or the
| new hotness everyone needed to know.
| simiones wrote:
| Pretty sure Sun wasn't working for IBM when they designed
| Java, were they?
| bluGill wrote:
| Now, but IBM went all in on it which was an important
| boost.
| hazn wrote:
| Do you mean big red and not big blue with that statement?
|
| Also, this reads as unnecessary snide. Having a language
| that makes it easy to onboard new people is a major
| strength. A lot of Golangs success is attributed to that
| strength of the language.
| ethbr0 wrote:
| IMHO, modern IBM (inasmuch as it exists) is the textbook
| target for Java, from a professional services standpoint.
|
| And it's snide and isn't.
|
| There's a lot to be said of being able to shovel
| programmers into a furnace and produce something
| functional. Decreasing risk to timeline and of failure
| makes large project PMs very happy. And generally makes
| stakeholders happy, because things don't outright fail as
| often.
| davidcuddeback wrote:
| > _Do you mean big red and not big blue with that
| statement?_
|
| I assume you're referring to Oracle here, but Oracle
| didn't invent Java either. Java came out of Sun
| Microsystems, whose color scheme seems to shift between
| blue and purple, so "big violet?" Just doesn't have the
| same ring to it.
| mikepurvis wrote:
| See, it's funny because as a _consumer_ of software, I
| 've overall been pretty pleased with solutions that come
| out of that process (think Jenkins, Artifactory, Elastic,
| Bazel).
|
| So it does seem capable of leading to reasonable results
| on the eventual timescale, despite various inconveniences
| and tradeoffs along the way. Of course, lots of terrible
| software has certainly been written in Java too, so it's
| not fair to compare Java's cream of the crop with some
| random PyPI package and draw conclusions.
| ethbr0 wrote:
| I think Java got a lot better when it stopped trying to
| bring along a OS GUI toolkit.
|
| Hence the vastly different experience between Java-
| exposed-as-app and Java-exposed-as-webapp.
| pjmlp wrote:
| Sun was the one creating and designing Java....
| lupire wrote:
| Microsoft was big into C++ then.
|
| C++ was terrible at cross-plaform (incompatible support
| for language futures across compiler), which was a deal
| breaker for many orgs.
| bluGill wrote:
| That is a cop out. You always need to train your people. I've
| been doing C++ for 20 years, and consider myself somewhat and
| expert, but if I join your team I will still need to learn a
| lot of things that are just how your team does it. If I also
| need to learn Erlang/java/go/ (pick any language I haven't
| used much) that only adds a short time.
|
| Even as complex as C++ is, I can train a great programmer C++
| a lot faster than I can train a future great junior C++
| programmer to be a great programmer. Yes you will encounter
| the rough edges of whatever language often in the first 5
| years, but a great programmer will be great in any language
| quickly. You need one expert in the language on the team for
| the weird complex stuff, but most code isn't that complex.
| [deleted]
| sillysaurusx wrote:
| I mean... sort of. It'd be like trying to write an ML stack
| in lisp circa 2023. Sure, you _can_ find devs willing to
| learn it, but it'll be at least two orders or magnitude
| easier to find python devs.
|
| I love lisp, but the thought of being forced to use it
| exclusively for ML makes me uneasy. And I've tried building
| lisp ML stacks. :)
| sitkack wrote:
| I'd probably use Hy and JAX, :)
|
| Should be beautiful.
| sillysaurusx wrote:
| Beautiful perhaps, but not productive for ML engineers
| who are joining the team and familiar with the standard
| stack.
|
| Sure, they _can_ be retrained, but I feel that there's a
| real cost to this, and it's too tempting to handwave it
| away as "they should just learn."
| stevan wrote:
| From Joe Armstrong's thesis (p. 6):
|
| > In February 1998 Erlang was banned for new product
| development within Ericsson--the main reason for the ban was
| that Ericsson wanted to be a consumer of sodware technologies
| rather than a producer.
|
| From Bjarne Dacker's thesis (2000, p. 37):
|
| > In February 1998, Erlang was banned within Ericsson Radio AB
| (ERA) for new product projects aimed for external customers
| because: > > "The selection of an implementation language
| implies a more long-term commitment than selection of
| processors and OS, due to the longer life cycle of implemented
| products. Use of a proprietary language, implies a continued
| effort to maintain and further develop the support and the
| development environment. It further implies that we cannot
| easily benefit from, and find synergy with, the evolution
| following the large scale deployment of globally used
| languages." [Ri98]
| arp242 wrote:
| Also, from Wikipedia:
|
| _" In February 1998, Ericsson Radio Systems banned the in-
| house use of Erlang for new products, citing a preference for
| non-proprietary languages. The ban caused Armstrong and
| others to make plans to leave Ericsson. In March 1998
| Ericsson announced the AXD301 switch, containing over a
| million lines of Erlang and reported to achieve a high
| availability of nine "9"s. In December 1998, the
| implementation of Erlang was open-sourced and most of the
| Erlang team resigned to form a new company Bluetail AB.
| Ericsson eventually relaxed the ban and re-hired Armstrong in
| 2004."_
|
| Not wanting to rely on a fairly esoteric in-house language
| makes some sense.
|
| Since then things have changed significantly of course.
| znpy wrote:
| > Not wanting to rely on a fairly esoteric in-house
| language makes some sense.
|
| Not necessarily... thst language clearly was a competitive
| advantage
| arp242 wrote:
| There's pros and cons to these kind of things. "Best tool
| for the job" reasoned purely from a technical point of
| view isn't necessarily the "best tool for the job" when
| everything is factored in.
|
| It's hard for me to judge one way or the other; I wasn't
| at Ericsson in 1998, or indeed, ever at Ericsson. I just
| figured that the language wasn't open source at the time
| and that they came back on their decision just a few year
| later were important bits missing from the previous
| comment.
| di4na wrote:
| There have been second hand report of the current
| leadership at Ericsson saying that opensourcing erlang
| was the worst decision they had taken. As it would have
| given way a massive competitive advantage.
| ilikehurdles wrote:
| Reminds of Paul Graham's essay on Lisp being a secret
| weapon for Viaweb, Beating The Averages[1].
|
| [1]: http://www.paulgraham.com/avg.html
| giancarlostoro wrote:
| I think the decision makes sense, there are things where
| paying someone else for the effort can save you on cost and
| on a maintenance nightmare and you can have your engineers
| focused on things that matter that already exist and building
| things that maybe no third party vendor gets it quite right,
| and that's okay to build in-house.
| leoc wrote:
| It wasn't an obviously crazy argument, especially at the
| time. The problem is that, in the _best_ case, Greenspun 's
| Tenth Law catches you. Putting things in-band doesn't make
| them go away.
| giancarlostoro wrote:
| Sidenote: Isn't there also a law / rule that says every
| concurrent system eventually ends up reinventing Erlang?
| ramchip wrote:
| Virding's First Rule of Programming:
|
| > Any sufficiently complicated concurrent program in
| another language contains an ad hoc informally-specified
| bug-ridden slow implementation of half of Erlang.
|
| http://rvirding.blogspot.com/2008/01/virdings-first-rule-
| of-...
| pupperino wrote:
| People really did love "synergy" in the 90s.
| christophilus wrote:
| > sodware
|
| This made me giggle.
| numbsafari wrote:
| When it comes to buy vs build, the grass _is_ always
| greener.
| another2another wrote:
| Then what you buy turns out to be not nearly as good as
| you thought, meanwhile lots of motivated open source
| developers are turning your homegrown build into
| something really game changing.
|
| Sod's Law.
| leoc wrote:
| What I recall from Erlang writings and videos is that,
| basically, Ericsson's C++ developers didn't want their cheese
| moved and successfully deployed the obvious (and not
| obviously unreasonable) arguments about how C++ was the
| industry standard, as above. I think Armstrong also admitted
| that the Erlang crew had a bit of a cocky attitude and wasn't
| great at winning others over.
| adamddev1 wrote:
| This reminds me a lot of Ron Garret's story of the
| difficulties of advocating for the use of Lisp against the
| industry standard C for space projects. Sounds like a lot
| of the exact same dynamics.
|
| https://www.corecursive.com/lisp-in-space-with-ron-garret/
| https://news.ycombinator.com/item?id=34524552
| AeroNotix wrote:
| > the Erlang crew had a bit of a cocky attitude
|
| This is also pretty prevalent in the Erlang users of today.
|
| I count myself among those, but I am hopefully not as cocky
| with it as I once was.
| jacquesm wrote:
| If you're cocky and you have the results to back it up
| then I'm ok with that.
| andrewflnr wrote:
| Morally? Yes. Strategically? Not optimal. :/
| samsquire wrote:
| I wrote a multithreaded message passing system in Java and I was
| trying to get async await to work
|
| The thing I want to define with behaviours is observable effects
| on objects.
|
| Such as interactions between tasks and objects between them. Not
| necessarily method calls but state interactions across multiple
| objects.
|
| Some objects should be colocated on threads and send behaviours
| to arbitrary groups of objects. Think of it as a collection that
| responds to events.
|
| I wrote a state machine serialization that looks like this. This
| defined a state progression between threads and async/await state
| machine.
|
| I think organised state machines would mean actor programming is
| so much easier to program. next_free_thread = 2
| task(A) thread(1) assignment(A, 1) = running_on(A, 1) | paused(A,
| 1) running_on(A, 1) thread(1)
| assignment(A, 1) thread_free(next_free_thread) =
| fork(A, B) | send_task_to_thread(B,
| next_free_thread) | running_on(B, 2)
| paused(B, 1)
| running_on(A, 1) | { yield(B, returnvalue) | paused(B, 2)
| } { await(A, B, returnvalue) | paused(A, 1) }
| | send_returnvalue(B, A, returnvalue)
|
| I think this serialization is truly powerful and easy to
| understand.
| notamy wrote:
| > Please don't share this repo or any of the linked to documents
| or repos just yet!
|
| Oh no :P
|
| Very interesting read though, thanks for sharing!
| boomskats wrote:
| When I started my first job at a grad scheme some 15 years ago,
| on the first day, we were asked to introduce ourselves to the
| group and say what name we'd prefer to be called by. This one
| guy says: "My name is XXX XXXXXXX. I don't care what you call
| me, just don't call me Ziggy... I HATE being called Ziggy".
|
| I heard Ziggy still works there.
| giancarlostoro wrote:
| Wonder if he did it intentionally or did not realize when he
| opened his mouth and said that.
| pnt12 wrote:
| This reminds of lord of the flies: I'm not a native English
| speaker, and only read it recently. When Piggy says he
| doesn't want to be called Piggy, I figured that's what he
| would be called the whole book. Alas, he was, and I cannot
| remember his real name.
| sriram_sun wrote:
| This is not the first time I'm reading about Ziggy!
| layer8 wrote:
| > I heard Ziggy still works there.
|
| He can't have hated it that much, then. ;)
| Pet_Ant wrote:
| I'm guessing that guys real name was "Zbigniew"
| pietroppeter wrote:
| hopefully stevan is same as stevana from github
| peoplefromibiza wrote:
| Erlang isn't, but the BEAM is.
| rhodin wrote:
| Yes came to say this. The Erlang VM is about lightweight
| processes and message passing.
| lippingoff wrote:
| This is fucking cool, thanks for sharing
| jb3689 wrote:
| To me the real amazing thing about Erlang is more the fact that
| these patterns based on queues and messages and supervisors scale
| so well. They are easy to implement on a small, single node
| application. You can compose them. You don't need to complete
| reauthor your calls to start distributing them and fanning out
| responsibility across your nodes because you've been doing RPCs
| from the start. A multi-node application is not that much more
| difficult to reasonable than a single-node application
| lippingoff wrote:
| This fucking rocks, great work and thanks for sharing.
| dominicl wrote:
| IMHO the differentiator is deeper yet everywhere engrained in
| OTP/Behaviours and the famous "Let it crash" slogan. It is the
| OS-Style process and resource isolation. That is something you
| can't port with a library into a language that doesn't have it
| built-in. Lightweight processes, are not the same if they can
| crash each other, or messup some shared objects/resources/...
|
| You can implement actor behaviours in go or node or even c, but
| without that lower level support it will never give you the
| stability guarantees that Erlang process isolation is giving.
|
| To draw a weird comparison Elixir (with Erlang process isolation)
| brings two world together. First it's a PHP/Ruby level of fire-
| and-forget productivity because each http request is handled in
| an independent isolated process, which if it crashes won't affect
| the system, but instead provide automatically a nicely debuggable
| crash log. And second it provides natively all distributed tools
| for long-lived systems. E.g. PubSub , Sessions and database
| connections don't have to be rebuilt like in ruby/PHP on a per
| request basis but can be their own long-lived processes.
|
| If there would be a library that could bring this easy to use
| process isolation+communication e.g. to C programming it would be
| a game changer. But the best you get in all other languages I'm
| aware of is to use actual process isolation (fork multiple
| node/ruby/go processes) and then use some kind of IPC manually or
| redis or k8s...
| xwowsersx wrote:
| > In 1998 Ericsson decided to ban all use of Erlang.
|
| What's the story there? Why did they decide to ban it?
|
| EDIT: doh, this has been answered elsewhere in this thread.
| throwawaymaths wrote:
| Wait till this person learns about links, monitors, call's
| default automatic backpressure mechanism, etc...
| mapcars wrote:
| As someone who worked with Erlang for a few of years I still
| wonder if using parallel processing in program top level is worth
| it or not. Because there are many ways to process data
| efficiently using queues, pipelines, etc and being clear on when
| it happens rather than a "wild west" of Erlang where you need to
| manage processes, links, restarts, and it's harder to focus on
| the business logic.
| Kototama wrote:
| What do you mean with "process data efficiently"? You speak
| about performance? But the main goal of the design of Erlang is
| fault tolerance, not performance. You can still use queues or
| whatever in Erlang if you want, but having a fault-tolerant
| system with process tree and process supervision in other
| languages is a different story.
| mapcars wrote:
| Right, I was mainly describing arguments for parallel
| processing which Erlang is also famous for.
|
| In terms of being fault-tolerant I think the modern approach
| with (micro)services is quite similar, one can have multiple
| services running and communicating using something like
| protobuf, having restart strategies, fallbacks and so on.
| From my experience Erlang doesn't offer any killer features
| in this case, does it?
| jimbokun wrote:
| Erlang pushes kubernetes down into the language and its
| libraries, instead of requiring Docker images and
| containers, etc.
|
| This means that the processes can be much, much more
| lightweight than kubernetes pods and containers. Cheaper to
| kill and restart. Can provide concurrency and fault
| tolerance at a far more granular level. Much simpler to
| write and deploy than Docker images. Etc. etc.
|
| At least that's my understanding (significant work
| experience with kubernetes, no real production experience
| with Erlang/Elixir).
| Thaxll wrote:
| Except you still need to run and deploy your Erlang app
| somewhere which might end up on Kubernetes.
|
| Kubernetes is just better than Erlang for process
| managment because it does more and is completely language
| agnostic, imagine you can't build some part of your
| system in Erlang but you need the same kind of
| functionality what do you do?
| OkayPhysicist wrote:
| As an dev working in Elixir, you're right on the money.
| IMO, the biggest benefit you gain from that granularity
| is that since processes cost next to nothing to spin up
| and down, you can (ab)use your fault tolerance mechanism
| for error handling, leading to the famous "Let it crash"
| philosophy.
|
| Restarting your docker image every time someone sends a
| malformed packet to your webserver is going to make for a
| trivial not even D-DOS attack. Killing the individual
| process that spins up to handle that particular
| packet/connection, though, is simply best practice.
| ElectricalUnion wrote:
| The killer feature is that the fault-tolerant behaviour is
| something either implicit, builtin or bundled [Erlang/OTP]
| with the language, instead of something that you need to
| either reinvent or bolt yourself to both your own code and
| your dependencies code as a afterthought when things start
| to fail.
| mapcars wrote:
| In Erlang the fault-tolerant behavior is not builtin
| either, only tools to make it. You still have to make the
| right supervision tree, dependencies between processes,
| links, making sure you handle the process termination
| messages correctly and many other details.
|
| In my experience in non-Erlang setups while doing
| requests to other services you have to check the response
| status and add some code handling it, so it's not really
| a complete afterthought. The only difference I see here
| is that Erlang handles failure in a real-time way, but it
| also can be done using some periodic task to query
| important services. And implementing in outside of Erlang
| gives more flexibility (think of Erlang cluster size and
| network limitations)
| hosh wrote:
| Our team had worked on an Elixir app for a couple years,
| before splitting off game logic into Dotnet. Scaling the
| dotnet server was a much different beast:
|
| - It wasn't designed to crash on failure. It uses thread
| pools with no supervision trees. We had to add in
| liveliness probes to check if it is alive. I've only had
| to use readiness checks for Elixir
|
| - No REPL. With a REPL in production, we can debug things
| live, even try patches to see if those work. Can't do
| that with Dotnet. That's also something that contributes
| to reliability
|
| Now, cluster size do matter. The way Erlang and BEAM was
| designed were for _vertical scaling_. You can minimize
| cluster size by biasing towards vertical scaling. That 's
| what we do on our systems. There's a way to do that with
| Kubernetes so that we scale vertically during our daily
| traffic cycle.
|
| At some point though, you start looking at partial
| clustering topology for BEAM, or use one of the many
| process registeries that are better suited for dynamic
| membership. (The one bitwalker wrote comes to mind).
| throwawaymaths wrote:
| That's wrong. Fault tolerance is basically the default.
| Yes you have to build a supervision tree, but unless
| you're writing a one-off script, you have to build it to
| _anyways_ to do anything.
| throwawaymaths wrote:
| >From my experience Erlang doesn't offer any killer
| features in this case, does it?
|
| Yes the killer feature in this case (relative to
| microservices) is a sane deployment, single codebase,
| unified language, being able to do integration tests
| without a full DevOps team, etc.
| stevan wrote:
| Do I understand you correctly in that you'd like more
| structure? E.g. that you can only deploy an `application` (=
| supervisor tree)?
| mapcars wrote:
| I'm talking about development, deployment should be automated
| anyways.
|
| As a developer I think it's easier to think in terms of how
| to send data to hadoop, sqs queue etc for processing and read
| results later than keep in mind the supervision tree,
| messages, mailbox size, linking and so on. And the
| "processing" side can be as well implemented in Erlang just I
| don't feel Erlang's features are needed in the "top level
| development" and create more problems and barriers.
| finder83 wrote:
| There's a component that seems to be missing here which is
| preemptive task scheduling. I've not seen another non-OS system
| do it like the BEAM VM (the VM behind Erlang), though there may
| be something I'm not aware of. It really prevents a whole class
| of concurrency issues where a hung process can freeze or slow
| down the entire system.
|
| For example, if you recreated gen_server in a cooperative
| concurrency environment, one gen_server could use up all of the
| CPU and have an impact on the performance of the rest of the
| system. Maybe the other threads (microthreads, not OS threads)
| would still respond in <500ms, but if every request takes 500ms
| when they normally take 15ms you could essentially have outage-
| like conditions, particularly with upstream timeouts.
|
| Instead, because BEAM is preemptive that one (or 10) hung
| gen_server doesn't hang up everything else on a node. Sure at
| some point performance will degrade, but that point is much
| further down the line than in cooperative concurrency models.
| There was a fantastic talk by Sasa Juric that demonstrates this
| in Erlang. [1] Otherwise you run a higher risk of even the
| supervisors being starved for CPU time, particularly if you are
| launching hundreds of processes that are all locking the CPU.
|
| It's really the combination of the behaviors (OTP), the
| scheduler, lightweight threads, message passing, and immutability
| to me that makes the Erlang (Elixir for me) concurrency model so
| appealing.
|
| Creating a language with the feel of a lisp, the environment of
| Smalltalk, and the concurrency of Erlang has been my dream for a
| long time.
|
| [1] https://www.youtube.com/watch?v=JvBT4XBdoUE
| samsquire wrote:
| I wrote a preemptive 1:M:N scheduler in C, Rust and Java.
|
| https://github.com/samsquire/preemptible-thread
|
| It is a 1:M:N scheduler where there is one scheduler thread, M
| kernel threads and N lightweight threads. I take advantage that
| loop indexes can be structures and can be modified by other
| threads. So we can set the thread's looping variable to the
| limit to end the current loop and pause it and then schedule
| another thread.
| stevan wrote:
| > There's a component that seems to be missing here which is
| preemptive task scheduling.
|
| For Erlang, yes. For implementing behaviours (the point of my
| post), I don't think so (I sketch a "single threaded" solution
| towards the end).
|
| I think one worker behaviour per CPU/core per stage in the
| processing pipeline is better than throwing thousands of
| processes at the problem and let the scheduler deal with it.
| This is what I got from Martin Thompson's talks (I linked to
| one of them in the "see also" section).
| waynesonfire wrote:
| Thanks for pointing this out. It is very important. The
| preemptive task scheduling is the secret sauces that makes
| Erlang unique. For example, if you look at how Akka implements
| the actor system, you'll see it riddled with the Future and/or
| async/await patterns which is due to the non-preemptive java
| scheduler. After playing with Erlang, this hack becomes a
| irritating.
| Fire-Dragon-DoL wrote:
| If I'm not wrong, Go added uncooperative task scheduling that
| behaves similarly?
| ksbrooksjr wrote:
| I'm not a Go expert but preemptive scheduling of goroutines
| is supposed to be one of the primary benefits of Go as a
| language [1]. Although I'm not sure if there's any runtime
| that adopts preemptive scheduling as fervently as the BEAM.
| Even the regular expression engine is preemptive in Erlang,
| which prevents regular expression denial of service attacks
| [2].
|
| [1] https://go.dev/src/runtime/preempt.go
|
| [2] https://www.erlang.org/doc/man/re.html#:~:text=run/3%20al
| way....
| ogogmad wrote:
| Regex DoSes can be made impossible without using any
| concurrency. See Go's and Rust's implementation. Also see
| https://swtch.com/~rsc/regexp/
| ksbrooksjr wrote:
| I wasn't trying to imply that concurrency is the only way
| to mitigate ReDos attacks, I was just pointing out that
| the BEAM applies preemption seemingly everywhere, which I
| found interesting.
| Fire-Dragon-DoL wrote:
| It wasn't non-cooperative pre-emptive scheduling for a
| while, which is why I kept pushing Elixir, until Go
| implemented it. Now Go has the same benefit as Elixir, but
| it's very, very fast.
| ksbrooksjr wrote:
| Yeah with preemptive scheduling, static typing, and
| better cpu utilization Go is a pretty compelling option
| for Erlang and Elixir users.
| cztomsik wrote:
| It's not immutable, there is no messaging, no hot-swap,
| no supervisors, it's not FP, and it doesn't have
| transparent support for clustering.
|
| Also, it's not memory-safe, 30ys old, and battle-tested
| pretty much everywhere.
| ksbrooksjr wrote:
| There are definitely trade offs when switching from
| Elixir or Erlang to Go. If you're a functional programmer
| who can't live without immutability, or you plan on
| running a cluster of machines that can communicate with
| each other and hot-swap code into the running system,
| then Elixir and Erlang are good choices.
|
| If you have some extremely CPU intensive code, or you
| like cross compiling to a lightweight binary, or you need
| static typing (without giving up preemptive scheduling),
| Go is a decent choice.
|
| Elixir and Erlang are not slow by any metric, but there
| are faster languages. Discord famously had to augment
| their Elixir code with Rust for example to scale to 11
| million users [1].
|
| [1] https://discord.com/blog/using-rust-to-scale-elixir-
| for-11-m...
| Fire-Dragon-DoL wrote:
| https://stackoverflow.com/a/73932230/312907 I found this
| answer.
|
| I can't find the original announcement, but I found the
| accepted Go proposal:
| https://github.com/golang/go/issues/24543
|
| I remember that being the straw that made me drop Elixir
| (which I love, to be clear). Go excels in many, many places
| where Elixir does, but it's way faster.
|
| I do think Elixir has immutability on their side, which is
| huge for new developers, but there are way less developers in
| Elixir than Go, so the end result doesn't change
| unfortunately.
| finder83 wrote:
| Interesting, at least when I was using it the goroutines
| themselves were cooperative, and of course the OS threads
| were preemptive when GO_MAXPROCS > 1. Not finding much with a
| search, there was a proposal to make them preemptive. Curious
| if others will chime in that it's now preemptive. Even so I
| like Elixir better as a language, but the processing speed of
| Go with a preemptive scheduler would be tempting for some use
| cases.
| travisgriggs wrote:
| > Creating a language with the feel of a lisp, the environment
| of Smalltalk, and the concurrency of Erlang has been my dream
| for a long time.
|
| Hey! Now there's two of us! It would be interesting to compare
| notes. :)
|
| After 20 years of really dedicated Smalltalk evangelism, I
| jumped out of that balloon a little over 10 years ago. Then I
| wandered in many strange lands embracing the polyglots "right
| tool for the right job" mantra. What I found was more like
| "here's a lot of really mediocre tools; rarely is it crystal
| clear which if many is right for the job."
|
| A year or so ago, I built out an API server in Elixir (no
| Phoenix) and I've really loved it. Great community. I love that
| it's built on basic fundamental principals, and not a bunch of
| edge cases. I've always wondered what some sort of mashup would
| look like. If I was independently wealthy I would tinker away
| at such a thing.
| finder83 wrote:
| "out of that balloon" -- I see what you did there. :)
|
| I've not really thought about it beyond that it would be
| nice. I really wanted to love Smalltalk, but Pharo at least
| didn't run well on Arch for me, and I didn't much love the
| web frameworks/concurrency story. I will say that Seaside is
| probably what got me interested in Liveview in the first
| place though. So many "if only"s...the developer experience
| is amazing in Smalltalk though.
|
| There's a lot that scares me about making languages and I've
| not studied it much. I've started reading through SICP though
| as a starting place. Writing a basic scheme interpreter (and
| future compiler/JIT compiler) as an image-based language
| (with change tracking) to make it interactive from the get-go
| seemed to be as far as I made it, maybe basing it on an
| existing Scheme, but tacking on preemptive scheduling sounds
| hard. But so is making a language and entire development
| environment. :)
|
| Shoot me an email though if you'd like to chat (in my
| profile, also username@gmail.com), not 100% sold if this is
| something I'd want to take on in the future or not, I really
| wish it already existed and someone better than me had
| already made it. :-D
| kaveh808 wrote:
| >> Creating a language with the feel of a lisp, the
| environment of Smalltalk, and the concurrency of Erlang has
| been my dream for a long time.
|
| >Hey! Now there's two of us! It would be interesting to
| compare notes. :)
|
| Please make this happen. :)
| jerf wrote:
| There is a very common pattern in the world where people conflate
| _goals_ with _results_. Of course, when I say that, it 's obvious
| that the two things aren't the same, but the unexamined
| assumption that they are the same sneaks in anyhow when you
| aren't looking, a cognitive shortcut hard to catch yourself
| making and harder yet to get in front of and deal with.
|
| In the light of this statement, the answer to what I think is the
| thesis question of that entire piece:
|
| "This begs the question: why aren't language and library
| designers stealing the structure behind Erlang's behaviours,
| rather than copying the ideas of lightweight processes and
| message passing?"
|
| Is that while Erlang has a lot of good _goals_ , the _results_ of
| how they got there are simply not the state of the art. Or, to
| put it another way, language designers are not copying Erlang,
| and they are _correct_ to not copy Erlang.
|
| I respect Erlang a lot. They were a good 10-15 years ahead of
| their time. However, you will note that if you add 10-15 years to
| the creation date of Erlang, you still end up in the past. If
| Erlang were to come out today, fresh, nobody had seen it before,
| into an otherwise identical programming language environment, I
| would say it makes several mistakes.
|
| One I've written about before is that Erlang is a non-Algol
| language for no reason:
| https://news.ycombinator.com/item?id=7277957 (Note the library
| mentioned in that post, suture, is now mature, and I use it all
| the time. It works for what I need.) But in the context of this
| post, that's less critical.
|
| The other major mistake I'd say it made if it came out in 2023 is
| that it is a totalizing environment. By that I mean that it has
| this built in implicit assumption that it is responsible for all
| the reliability in the system, and you don't get Erlang's
| features very conveniently if you don't use it as the complete
| control backplane for your entire system. You run an Erlang
| cluster, and it bundles all the message passing, restarting,
| reliability, cluster management, software deploy, and everything
| into one system.
|
| But for the world we live in today, that's a price not worth
| paying. We don't need the Erlang message bus to be the only
| message bus. The Erlang message bus is, frankly, not very good,
| and it's actively terrible if you want to use it for one non-
| Erlang process to communicate to another. We don't need the
| Erlang message bus. We have a dozen message busses, some in the
| cloud, some commercial, some open source, some that double as
| retention (Kafka), all of which scale better, none of which tie
| you to Erlang's funky data types.
|
| And then, within the matrix of those message busses, we don't
| need Erlang's restart capability. We have an abundance of ways to
| restart processes, from systemd, to kubernetes, to any number of
| other ways.
|
| We don't need Erlang clusters for redundancy any more. You just
| run multiple copies of a service against the message bus, on
| multiple systems for redundancy.
|
| We don't need Erlang's behaviors. We have interfaces, traits,
| object orientation, and even just pushing that entire problem up
| to the OS process level, or writing a cloud function, and any
| number of ways of achieving the same goal.
|
| Erlang's software deploy is interesting, but we have a lot of
| options for it. The whole attempt to do live updates is
| interesting, but it also imposed a lot of constraints that
| systems that don't have that need, which is the vast majority of
| them, don't need or want. This is perhaps the space where the
| state of the art isn't that far ahead of Erlang. It's still a
| mess, despite all the churn in this space. But even so, with all
| the options available, you can probably find something better for
| your system than the Erlang way of upgrading software, even if it
| isn't necessarily much easier.
|
| The cognitive hazard that Erlang presents the community in 2023
| is that it has some very good writing on the topic of reliability
| and its other goals, and then, naturally, one segues into the
| discussion of how Erlang solved the problem. And it was a very
| interesting solution for the time. I used Erlang for many, many
| years back when it was effectively the only solution to these
| problems.
|
| But it isn't the only solution anymore. The space has exploded
| with options. Unsurprisingly, the ones that a highly innovative
| pioneer tried out first are not the best, or the only. They chose
| well. Let me again emphasize my respect for the project. But it's
| not on the cutting edge anymore.
|
| Granted, the diversity of options does mean the real world has
| gotten quite chaotic, where you may have three message busses
| attaching systems implemented in a dozen different languages, but
| that's something you can take up with Conway's Law. Erlang
| couldn't work with Conway's Law without totally converting your
| entire company to it, which just isn't going to happen.
|
| The reason why language designers aren't rushing to copy Erlang
| is that what was excellent and amazing in 2000 (and, again let me
| underline, I mean that _very seriously_ , it was a cutting edge
| platform built with a lot of vision and moxie) is, in 2023,
| mediocre. Erlang is a mediocre language (Elixir is at least
| "good", Erlang is mediocre), attached to a mediocre message bus,
| with a type system that doesn't even reach mediocre, with a
| mediocre totalizing approach to system design where there's a
| very significant impedence mismatch between it and the rest of
| the world, with an at-par-at-best VM (I won't call that mediocre,
| but where it used to be head-and-shoulders above everything else
| in certain ways, it is now merely competitive), with mediocre
| standard libraries, and a mediocre product fit to its own stated
| goals. It just isn't the best any more.
|
| The state of the art right now is super chaotic. I can hardly get
| two systems deployed on the same infrastructure any more, because
| there's always some reason _something_ in that list has changed.
| But when the chaos settles and best practices emerge, something
| that I 'd say is at _least_ a good 5 years away, the result will
| clearly have Erlang inspiration in it for sure... but it won 't
| look a lot like Erlang on the surface.
|
| What is worth copying has largely been copied. It doesn't look
| exactly like Erlang, but this turns out to be a _good thing_.
| stevan wrote:
| > In the light of this statement, the answer to what I think is
| the thesis question of that entire piece:
|
| > "This begs the question: why aren't language and library
| designers stealing the structure behind Erlang's behaviours,
| rather than copying the ideas of lightweight processes and
| message passing?"
|
| > [...]
|
| > But for the world we live in today, that's a price not worth
| paying. We don't need the Erlang message bus to be the only
| message bus. The Erlang message bus is, frankly, not very good,
| and it's actively terrible if you want to use it for one non-
| Erlang process to communicate to another. We don't need the
| Erlang message bus. We have a dozen message busses, some in the
| cloud, some commercial, some open source, some that double as
| retention (Kafka), all of which scale better, none of which tie
| you to Erlang's funky data types.
|
| I asked why they were not stealing _the structure of behind
| Erlang 's behaviours_, I didn't suggest anyone should steal
| Erlang's message bus or anything else.
|
| > And then, within the matrix of those message busses, we don't
| need Erlang's restart capability. We have an abundance of ways
| to restart processes, from systemd, to kubernetes, to any
| number of other ways.
|
| I don't think restarting the process from systemd or kubernetes
| is comparable with a supervisor tree. First of all the tree
| gives you a way to structure and control the restarts, e.g.
| frequently failing processes should be further down the tree or
| they will cause their sister nodes to get restarted etc. The
| other obvious difference is speed.
|
| > We don't need Erlang's behaviors. We have interfaces, traits,
| [...]
|
| Yet I don't know of any other language which uses interfaces in
| a way which they achieve the benefits (listed in the article)
| that behaviours in Erlang (e.g. gen_server) give you, do you?
| jerf wrote:
| "I asked why they were not stealing the structure of behind
| Erlang's behaviours, I didn't suggest anyone should steal
| Erlang's message bus or anything else."
|
| To some extent I know... but to some extent the answer is
| these things are all tied together. Erlang is a really tight
| ball of solutions to its own problems at times. I don't mean
| that in a bad way, but it all works together. It needs
| "behaviors" because it didn't have any of the other things I
| mentioned.
|
| When I went to implement behaviors
| (https://www.jerf.org/iri/post/2930/ ), I discovered they
| just weren't worth copying into Go. You ask if any other
| language uses interfaces to achieve what Erlang does; my
| perspective is that I've seen people try to port "behaviors"
| into two or three other languages now, and they're always
| these foreign things that are very klunky, and solve problems
| better solved other ways.
|
| "I don't think restarting the process from systemd or
| kubernetes is comparable with a supervisor tree."
|
| It isn't, but the problem is...
|
| "First of all the tree gives you a way to structure and
| control the restarts, e.g. frequently failing processes
| should be further down the tree or they will cause their
| sister nodes to get restarted etc."
|
| I don't need that. I've been using supervisor trees for over
| a decade now, and they rarely, if ever, go down more than
| "application -> services". Maybe somebody out there has
| "trees" that go down six levels and have super complicated
| bespoke restart operations on each level and branch, but they
| must be the exception.
|
| To the extent that I have deep trees, they're for
| _composition_ , not because I need the complicated behaviors.
| A thing that used to be a single process service is now three
| processes, and to hide that, I make that thing a supervisor
| of its own so that the upper levels still just see an
| ".Add()" operation, instead of having to know about all the
| bits and pieces.
|
| "The other obvious difference is speed."
|
| Certainly, but those are only one of the options.
|
| "Yet I don't know of any other language which uses interfaces
| in a way which they achieve the benefits (listed in the
| article) that behaviours in Erlang (e.g. gen_server) give
| you, do you?"
|
| This is a case of what I'm talking about. Don't confuse
| Erlang's particular solution for being the only possible
| solution. Erlang's behaviors are basically the Template
| pattern
| (https://en.wikipedia.org/wiki/Template_method_pattern )
| written into the language rather than implemented through
| objects. If you look for the exact Erlang behaviors out in
| the wild, you won't hardly find anything. If you look for
| _things that solve the same problems_ , there's tons of them.
| A lambda function in AWS is a solution to that problem. The
| suture library I wrote is a different one. Java frameworks
| have their own solutions in all sorts of different ways.
|
| To put it another way, whereas in 1998 people having the
| problems Erlang solved was rare, today we _all_ have them. We
| _can 't_ be blundering around with no solutions since we are
| all too blinkered to use Erlang which just solves them all.
| That makes no sense. There are _far_ more distributed systems
| concerned with reliability out there now implemented in not-
| Erlang than in Erlang. We are not all just blundering along
| in a fog of confusion, unaware of the existence of
| architecture, modularity, and abstraction. If programmers
| have a flaw, it 's too much architecture rather than too
| little.
|
| Maybe that's one of the problems with the Erlang writing.
| It's all implicitly written from a perspective of the 90s,
| where this is all a surprise to people, and it kind of seeps
| in if you let it. But that's not where the world is right
| now. It is not news that we need to be reliable. It is not
| news that we want to run on multiple systems. I've got _non-
| technical managers_ asking me about this stuff at work
| whenever I propose a design. There 's been a _ton_ of work on
| all of these issues. It 's not all good, by any means! But
| there's now too many solutions moreso than not enough.
| stevan wrote:
| > To some extent I know... but to some extent the answer is
| these things are all tied together. Erlang is a really
| tight ball of solutions to its own problems at times. I
| don't mean that in a bad way, but it all works together. It
| needs "behaviors" because it didn't have any of the other
| things I mentioned.
|
| I can see why you would say that regarding the supervisor
| behaviour, but I don't see how your argument applies to the
| other five behaviours I wrote about. Let's keep it simple
| and focus on, say, `gen_server`?
|
| > This is a case of what I'm talking about. Don't confuse
| Erlang's particular solution for being the only possible
| solution.
|
| I'm not, in fact I mention that I've started working on a
| small prototype to explore implementing behaviours in a
| different way at the end of the post.
|
| > Erlang's behaviors are basically the Template pattern
| (https://en.wikipedia.org/wiki/Template_method_pattern )
| written into the language rather than implemented through
| objects. If you look for the exact Erlang behaviors out in
| the wild, you won't hardly find anything. If you look for
| things that solve the same problems, there's tons of them.
|
| My understanding of Joe's thesis is that we can compose a
| small set of behaviours into complex systems. He and OTP
| expose and highlight this _structure_. I don 't doubt that
| other's have also discovered the usefulness of this
| structure, but I don't see anyone else try to help bring
| that structure to the forefront so that we can improve the
| state of software.
|
| Your argument seems to be that this structure has now
| become implicit in the tools that have been developed since
| then, but I think again this misses the point: the
| structure is simple and should be made explicit not hidden
| away behind 500k lines of C (systemd) or almost 4M lines of
| Go (kubernetes).
| macintux wrote:
| Additionally, the fact that a very small set of standard
| behaviours has been re-used time and time again for big
| software projects means they're extremely well-tested,
| and well-documented.
| [deleted]
| bcrosby95 wrote:
| > Erlang couldn't work with Conway's Law without totally
| converting your entire company to it, which just isn't going to
| happen.
|
| I think this says more than you think it does.
|
| I've worked in software development professionally for 20
| years. I have never worked at a company that had more than 1
| backend language. A large percentage of developers work at
| small companies.
|
| Erlang/Elixir/BEAM isn't for the Googles of the world. And
| that's fine. Tech needs to stop its infatuation with these
| companies. Just because something is right for those companies,
| _doesn 't mean it's right for yours._ Ignoring this has done a
| lot of damage over the years. Production complexity is a
| killer.
|
| People love to talk about scaling (and here, I'm talking about
| scaling a company), but only ever mention one aspect: scaling
| up. Things also have to scale _down_. Technological choices
| tend to sacrifice one for the other.
|
| But BEAM languages have a wider scaling band than most other
| technological choices. And that is in large part because of
| everything it offers out of the box.
| weatherlight wrote:
| No one ever talks about scaling down systems, everything is
| always additive.
| petejodo wrote:
| > We don't need the Erlang message bus to be the only message
| bus. The Erlang message bus is, frankly, not very good
|
| Genuinely curious why it's not very good. Were you speaking
| solely from the perspective of non-Erlang processes? And also
| specifically regarding remote messages rather than local?
| jerf wrote:
| There's two ways a non-erlang process can speak Erlang terms.
|
| It can implement the Erlang node protocol and show up as a
| full Erlang node. Neat capability, but the impedence mismatch
| between systems is something fierce, because the protocol
| deeply assumes that you're basically Erlang, e.g., not just
| that processes have mailboxes but that mailboxes have the
| exact same semantics as Erlang, and you have to implement
| process linking with the exact same semantics, etc. It's
| difficult.
|
| Alternatively, you can write a proxy in Erlang where the
| first process speaks to an Erlang server to send a term to
| some other Erlang process that will then relay the message in
| whatever form. This will either be a custom protocol, in
| which case this is an awful lot of code to write for such a
| task, or a common messaging protocol in which case you don't
| need Erlang. That is, you can speak rabbitmq, but that
| doesn't need to be implemented in Erlang.
|
| The production system I maintained on Erlang did this a lot,
| because I was forced to have a lot of Perl code interacting
| with the Erlang system. Back in the day it was a fine choice;
| it was a time when you couldn't just pop on to google and
| turn up a dozen battle-ready message busses in two minutes.
| Now, though, it is far better for the Erlang code to just be
| another node on your common bus than for it to _be_ the
| message bus.
| saurik wrote:
| > And then, within the matrix of those message busses, we don't
| need Erlang's restart capability. We have an abundance of ways
| to restart processes, from systemd, to kubernetes, to any
| number of other ways.
|
| I agree with pretty much all of your comment (which clearly
| comes from a place of deep experience), but the thing that
| keeps bringing me back to the ideas of Erlang--potentially
| trying in vain to implement similar concepts in the languages I
| actually work in (including developing a way to manage fibers
| in C++ coroutines that work similar to Erlang processes so I
| could debug background behaviors)--is the idea that these
| restartable and isolated units simply aren't large enough to be
| managed by the operating system and systemd or kubernetes of
| all things: they are things like individual user connections.
| While there are plentiful easy ways to do shared-nothing
| concurrency in the world attached to virtually every software
| project and framework these days, they are all orders of
| magnitude more expensive than what Erlang was doing, even with
| its silly little kind of inefficient VM.
| llimllib wrote:
| It's worth noting that jerf implemented process trees
| including restarting services in golang - so I'm curious in
| what way he means that we don't need Erlang's restart
| capability when he added it to his own library.
| camgunz wrote:
| I read that list as "Erlang doesn't have exclusive rights
| to X"
| jerf wrote:
| We do not need _Erlang 's exact solution_.
|
| The suture interface for a service in the latest version
| is: type Service interface {
| Serve(ctx context.Context) error }
|
| Where's all the stuff for gen_server?
| https://www.erlang.org/doc/man/gen_server.html Where's the
| "start_link" versus "start_monitor" distinction? Where's
| "cast" versus "call"? Heck, where's "stop"?
|
| The answer is, those things are all Erlangisms. They make
| sense in Erlang. But in Go, the supervisor tree doesn't
| need to enforce any of those things. Cast or call or
| multicall or whatever else you like in the code that talks
| to those services. When I got down to it I couldn't even
| justify a "Start" or "Initialize" call; I just couldn't
| help but notice it was completely redundant and it could
| just be in the Serve function.
|
| What you need is that _a single crash does not take down
| your entire service_. It does not have to be in Erlang
| supervisor trees with Erlang gen_servers that have the
| exact features and behaviors as Erlang supervisors and the
| exact APIs. It doesn 't even have to be in an OS process
| the same way as Erlang; cloud lambda functions in many ways
| solve the same problems in a completely different way.
| Getting too stuck in Erlang's thoughtspace inhibits your
| ability to design solutions. There are many ways to make it
| so a single crash does not take down your entire service.
| Erlang's way is not the only way.
| sangnoir wrote:
| > Heck, where's "stop"?
|
| I'm intermediate (at best) in Go, but I've always found
| server teardown in Go a little clunky: most solution's
| I've seen are variations of sending a KILL signal to a
| channel that the server is listening on, which
| technically is still message-passing, though
| unstandardized, unlike stop/n
| jimbokun wrote:
| That sounds very interesting. What's the project that
| interface if from?
| llimllib wrote:
| https://pkg.go.dev/github.com/thejerf/suture/v4
| brightball wrote:
| > We don't need Erlang clusters for redundancy any more. You
| just run multiple copies of a service against the message bus,
| on multiple systems for redundancy.
|
| The thing about Erlang is that you never needed clusters at
| all. The redundancy was built into each instances with the
| runtime. When you build that way, everything naturally scales
| out horizontally with additional processors and/or physical
| nodes.
|
| You can't do that in any other language without building the
| entire system for it from the ground up.
|
| Using multiple systems for redundancy means counting on the
| entire system going down. The Erlang way isolates this impact
| to 1 of potentially millions of parallel actions on the system
| itself. Using other systems for redundancy, the other million
| actions in progress on this server go down with it. The
| difference in the level of redundancy is significant.
|
| But I do agree with you that we don't need it for most systems
| because most systems simply aren't that complex. The benefit of
| the BEAM comes from simplifying complexity, which tends to
| evolve over time. Elixir, Phoenix and LiveView will likely lead
| to earlier adoption of the BEAM in projects before the
| complexity ramps up which will show a long term benefit.
| jerf wrote:
| "The thing about Erlang is that you never needed clusters at
| all."
|
| IIRC, if you read the original thesis, the reason for
| clusters is just that there's always that chance an entire
| machine will go down, so if you want high reliability, you
| have no choice but to have a second one.
|
| The OP is correct in that the key to understanding every
| design decision in Erlang is to look at it through the lens
| of reliability. It also helps to think about it in terms of
| phone switches, where the time horizon for reliability is in
| milliseconds. I am responsible for many "reliable" systems
| that have a high need for reliability, but not _quite_ on
| that granularity. A few seconds pause, or the need for a
| client to potentially re-issue a request, is not as critical
| as missing milliseconds in a phone call.
| brightball wrote:
| That's true. You do always need to plan for machine
| redundancy but hopefully machines don't completely fail
| that often. I can't remember the last time I experienced an
| instance failure that wasn't a data center wide impact.
|
| It impacts how you architect certain solutions though. For
| example, if you've got users connected with websockets
| you're suddenly able to maintain their state right there
| with the connection.
|
| In a situation where you can't rely on state on the server
| itself, every websocket connection has to relay to some
| backend system like Redis/DB, etc since the state can't be
| counted on at the connection layer.
| OkayPhysicist wrote:
| The totalizing environment is a huge boon, rather than a
| downside. The utility of tooling is directly tied to how
| predictable the entire architecture of your application is. If,
| at the highest level, your projects always consist of a random
| hodgepodge of services written in a variety of flavor-of-the-
| week programming languages, 100% of your top-level tooling will
| need to be custom built for your project. In contrast, if all
| of your projects consist of "Erlang" or maybe "Erlang+a
| database", then suddenly you can create tooling that works on
| any project. For example, Observer lets you look at the entire
| process tree of your application, for any Erlang application.
| If your entire project is an Erlang application, you now have
| complete visibility into your entire application's current
| state. It's practically magic.
|
| That was my biggest point, but you posted a long comment with a
| few other disagreeable points, too:
|
| 1. re:Conway's Law : Just write the code in Erlang (read:
| Elixir in this day and age) to begin with. Rewriting large
| applications in new languages is rarely worth it anyways,
| that's not a failure of the better language you want to switch
| to, it's a failure of the poor choice of language you started
| with.
|
| 2. re: But it's not on the cutting edge anymore: And yet it is.
| Erlang and Elixir are the only languages in anything remotely
| resembling widespread use to not have an utterly pathological
| concurrency story. Async-Await is an awkward crutch to shoehorn
| concurrency into languages that were never designed to support
| it, Go and its goroutines entirely misses the point of the
| exercise and simultaneously encourages you to write mutable
| state and punishes you for doing so, and the Actor model
| libraries for other languages are just half-baked, bug-ridden
| implementations of a small fraction of Erlang.
| mononcqc wrote:
| I'm not quite sure how Erlang's world is totalizing. It has
| ways to ship things in a very integrated manner, but I have
| shipped and operated Erlang software that was containerized the
| same as everything else, in the same K8s cluster as the rest,
| with the same controls as everything else, with similar feature
| flags and telemetry as the rest, using the same Kafka streams
| with the same gRPC (or Thrift or Avro) messages as the rest,
| invisibly different from other applications in the cluster to
| the operator in how they were run aside from generating stack
| traces that look different when part of it crashes.
|
| That it _also_ ships with other ways of doing things in no way
| constrains or limits your decisions, and most modern Erlang (or
| Elixir) applications I have maintained ran the same way.
|
| You still get message passing (to internal processes),
| supervision (with shared-nothing and/or immutability mechanisms
| that are essential to useful supervision and fault isolation),
| the ability to restart within the host, but also from systemd
| or whatever else.
|
| None of these mechanisms are mutually exclusive so long as you
| build your application from the modern world rather than
| grabbing a book from 10-15 years ago explaining how to do
| things 10-15 years ago.
|
| And you don't _need_ any of what Erlang provides, the same way
| you don't _need_ containers (or k8s), the same way you don't
| _need_ OpenTelemetry, the same way you don't _need_ an
| absolutely powerful type system (as Go will demonstrate). But
| they are nice, and they are useful, and they can be a bad fit
| to some problems as well.
|
| Live deploys are one example of this. Most people never
| actually used the feature. Those who need it found ways (and I
| wrote one that fits in somewhat nicely with modern kubernetes
| deployments in https://ferd.ca/my-favorite-erlang-
| container.html) but in no way has anyone been forced to do it.
| In fact, the most common pattern is people wanting to
| eventually use that mechanism and finding out they had not
| structured their app properly to do it and needing to give it a
| facelift. Because it was never necessary nor totalizing.
|
| Erlang isn't the only solution anymore, that's true, and it's
| one of the things that makes its adoption less of an obvious
| thing in many corners of the industry. But none of the new
| solutions in the 2023 reality are also mutually exclusive to
| Erlang. They're all available to Erlang as well, and to Elixir.
|
| And while the type system is underpowered (and there are
| ongoing area of research there -- I think at least 3-4
| competing type systems are being developed and experimented
| with right now), that the syntax remains what it is, I still
| strongly believe that what people copied from Erlang were the
| easy bits that provide the less benefit.
|
| There is still nothing to this day, whether in Rust or Go or
| Java or Python or whatever, that lets you decompose and
| structure a system for its components to have the type of
| isolation they have, a clarity of dependency in blast radius
| and faults, nor the ability to introspect things at runtime
| interactively in production that Erlang (and by extension,
| languages like Elixir or Gleam) provide.
|
| I've used them, I worked in them, and it doesn't compare on
| that front. Regardless of if Erlang is worth deploying your
| software in production for, the approach it has becomes as
| illuminating as the stacks that try and push concepts such as
| lack of side-effects and purity and what they let you transform
| in how you think about problems and their solutions.
|
| That part hasn't been copied, and it's still relevant to this
| day in structuring robust systems.
| btbuildem wrote:
| In a way, this "totalizing" can be looked at as a pro rather
| than a con.
|
| Yes, today we have an entire collection of ecosystems of
| services that can provide the key functionalities as described
| above. Each of these technologies comes with it's own long tail
| of dependencies, security issues, maintenance effort, plain old
| computational overhead, etc.
|
| Meanwhile, this 30-year old technology provides matching
| functionality (yes, admittedly with syntax and object types
| that simultaneously induce vertigo and motion sickness), but
| all the bugs have long been eradicated or encased in amber, and
| pound-for-pound it will rip circles around an alternative
| solution that's dragging a Java VM or a megaton of node_modules
| along with it, wrapped in docker images and k8s yamls.
|
| I use yaws as my go-to webserver. It's nuke-proof. It's
| simple*, and it Just Works. Good luck finding a haxxor that can
| breach it. I believe that it's in large part due to the very
| simple conceptual building blocks it's constructed out of (the
| gen_ behaviours OP describes).
| [deleted]
| [deleted]
| lvass wrote:
| >But when the chaos settles and best practices emerge,
| something that I'd say is at least a good 5 years away
|
| That's incredibly optimistic. I am 30 and have basically no
| hope this will happen in my lifetime, and in the meanwhile,
| BEAM works great. I agree with all your points, except when you
| write 2023 as if we're doing things better now. Research in
| this area hasn't really bore many fruit over the last 30 years
| as you make it look like.
| josevalim wrote:
| Your comment seems to consider Erlang as a tool to build
| heterogeneous systems, where you have multiple distinct
| applications, written in the same or different technologies,
| talking to each other.
|
| In such cases, I would agree with part of your criticisms
| because it is indeed the wrong tool for the job. Erlang was not
| designed to solve this problem: the serialization format is
| centered around Erlang. The distribution messages reflect the
| semantics of processes, messaging, monitoring, etc. Inter
| communication is not the focus. Even on its early days, the
| distribution was used to provide tolerance against hardware
| failures by running two identical systems. So I find comparing
| Kafka and Erlang to be an apples to oranges scenario.
|
| In my opinion, Erlang shines for building homogeneous systems:
| multiple instances of the same application running in a
| cluster. Precisely because all I need is Erlang. It comes with
| a unified programming model for both local and distributed
| execution. Look at Phoenix uses it to provide features such as
| distributed pubsub and presence out-of-the-box, features which
| either require external tools - and additional complexity - or
| simply do not exist in other platforms. And the beauty in
| designing such solutions is that you start with the concurrent
| version and then naturally evolve into making it distributed
| (if necessary).
|
| I also find the comparison equally misses the mark between
| restarts/fault-tolerance and Kubernetes. Because, once again,
| they mostly work at different levels. The classical example is
| using supervisors to model database connections, something you
| simply cannot delegate to k8s. But a more recent example comes
| from working on Nx, which communicates to the GPU. You can
| stream data in and out of the GPU, but what happens when the
| code streaming data errors out? You need to develop a
| synchronization mechanism to make sure the GPU does not get
| stuck. And what happens if the synchronization mechanism fails?
| With Erlang I can model and test all of those failure scenarios
| quite elegantly. Perhaps there are better approaches lurking
| out there, but it certainly isn't k8s.
|
| When it comes to k8s, they mostly complement each other. Erlang
| tool for restarting _machines_ is basically non-existent (there
| is -heart or the stand-by system described by Joe) and k8s
| addresses that. Erlang doesn't have service discovery, k8s
| covers that gap. But, even then, there is no assumption you
| must use Erlang clustering. It is opt-in, you don't have to use
| it, and in the "worst case" scenario, you can deploy Erlang
| just as any other language.
| dmitriid wrote:
| > We don't need Erlang's behaviors. We have interfaces, traits,
| object orientation, and even just pushing that entire problem
| up to the OS process level, or writing a cloud function, and
| any number of ways of achieving the same goal.
|
| None of these achieve the same goal. Or they result in
| significantly more complicated and brittle systems. Or they
| only achieve that goal insofar as you need to glue several
| heterogenous systems together.
|
| > The reason why language designers aren't rushing to copy
| Erlang is that what was excellent and amazing in 2000 (and,
| again let me underline, I mean that very seriously, it was a
| cutting edge platform built with a lot of vision and moxie) is,
| in 2023, mediocre.
|
| The main reason is that it is borderline impossible to retrofit
| Erlang model onto an existing language. Adding concurrency
| alone may be a decade-long project (see OCaml). Adding all of
| the guarantees that Erlang VM provides... well.
|
| And on top of that too many people completely ignore anything
| in Erlang beyond "lightweight processes/actors".
|
| The fact that you can have an isolated process that you can
| monitor and observe, and have a guaranteed notification that it
| failed/succeeded without affecting the rest of the system is a)
| completely ignored and b) nearly impossible to retrofit onto
| existing systems.
|
| And there are exceedingly few new languages that even think
| about concurrency at all. And async/await is not even remotely
| state of the art (but people are busy grafting them onto all
| languages they can lay their hands on).
|
| State of the art still is mutexes and killing your entire
| program if something fails. Often both of those.
| natdempk wrote:
| Relevant, I've written a bit about the various actor models here
| including Erlang under process-based actors: http://dist-prog-
| book.com/chapter/3/message-passing.html
| styluss wrote:
| opening the root page http://dist-prog-book.com/ seems to load
| Tufte CSS documentation
| matthiasl wrote:
| Not important to the conclusion or reasoning... but Stevena's
| post says: "the whole team working on Erlang quit
| and started their own company."
|
| The same event as described in "A history of Erlang":
| "In December, most of the group that created Erlang resigned
| from Ericsson and started a new company called Bluetail AB."
| https://www.labouseur.com/courses/erlang/history-of-erlang-
| armstrong.pdf
|
| 'most of the group that _created_ Erlang' is not the same thing
| as 'the whole team _working_ on Erlang'. Or, quantifying it, at
| the end of the 'history' paper, there's a list of 45 people under
| 'implementation' and 'tools'. Around 35 were at Ericsson in 1998.
| Of those, nine or ten quit to form Bluetail, and another two or
| three left for Bluetail later on.
|
| (In 1998, there were two connected groups working on Erlang in
| the same building in Alvsjo, Stockholm. One was the computer
| science laboratory (CSLAB), where Erlang was _created_ , the
| other was "Open Systems", which had more of a development role. A
| significant part of CSLAB left. Almost all of 'Open Systems'
| stayed. Many that stayed were already doing a stellar job on
| Erlang and many still are.)
| eternalban wrote:
| J2EE could have had all that but it was missing an OTP. [Java got
| the competing 'application server' vendors competing themselves
| out of a massive market.] Components with well defined life
| cycles, request/reply and message processing, dynamic remote
| interface binding, isolated execution environment, pure single
| threaded application level code ('business logic'), declarative
| composition ... that's all in the specs.
|
| Erlang is mainly about OTP. OTP delivered an opionated take on
| distributed components -- think of _Erlang on_ OTP as _Ruby on_
| Rails -- and did it exceptionally well.
|
| But one could still do this with Java btw. The specs are still
| there and they are solid.
| didip wrote:
| These days, if you have Kubernetes, queues, and worker queues,
| does OTP still have an edge?
| bsaul wrote:
| I believe the only argument left is that erlang /OTP provide
| all of those in a single cohesive environment.
| [deleted]
| sb8244 wrote:
| Yes.
|
| Being able to do this at the programming language level is
| extremely powerful and creates an entirely different way of
| building applications. My go-to analogy is building a city
| (lots of individual, isolated, separate stack processes)
| instead of a big skyscraper (deep stack requests, concurrency
| difficult, error recovery manual).
|
| You can build that city in a limited way with k8s, but there's
| way more overhead along the way, to the point of it not being
| enjoyable for me.
| amelius wrote:
| Isn't Erlang's model a giant ball of microservices?
| lupire wrote:
| Nanoservices.
| valenterry wrote:
| > The application programmer writes sequential code, all
| concurrency is hidden away in the behaviour;
|
| Yeah, until the business problem itself involves inherent
| concurrency, which usually happens much faster than people think.
| Or until I, as the non-expert, want to dig in to make changes or
| debug a problem.
|
| This distinction into "expert" and "lowlife (SCNR) using the
| expert abstractions" is really one that doesn't hold in practice
| most of the time.
|
| I think it's much better to embrace that concurrency is a cross-
| cutting-concern and reality is that it can happen on _any_ level,
| so the language should better support reality.
| jeremyjh wrote:
| I think you're talking about two different things. When they
| say concurrency is hidden away, I think they mean the
| programmer isn't dealing directly with locks/mutexes. They are
| also lying, since async functionality like `handle_cast` is
| built into the `gen_server` behaviour and every caller is
| calling either `gen_server:cast` or `gen_server:call` depending
| on whether or not it is going to block on a response. Whenever
| you invoke async behaviour concurrency will invade all the
| domain logic and can't be ignored.
___________________________________________________________________
(page generated 2023-01-27 23:00 UTC)