[HN Gopher] On Building Glue Systems
___________________________________________________________________
On Building Glue Systems
Author : poga
Score : 69 points
Date : 2021-09-18 07:58 UTC (3 days ago)
(HTM) web link (devpoga.org)
(TXT) w3m dump (devpoga.org)
| john-doe wrote:
| > Since a glue system is about connecting many isolated pieces
| into a coordinated abomination (...)
|
| Sounds like modern web development.
| Joker_vD wrote:
| Or shell scripting, legacy or modern, doesn't matter.
| krossitalk wrote:
| This starts out strong and then trails off without a real
| conclusion or opinion on glue systems. A bad build system or lack
| of maintenance can happen to any project.
| xwowsersx wrote:
| Agreed. I'm kind of surprised to see this post so high up on
| the front page here.
| [deleted]
| zaphar wrote:
| I don't understand what he means with the build systems comment?
| What does that have to do with Glue systems? I could speculate
| that he wishes the build systems accounted for all the supposedly
| siloed dependent systems in some way but it's not clear what
| exactly his complaint here is.
| shireboy wrote:
| Not sure if this is exactly what the article refers to, but at
| one of my customers I helped create an app literally called Glue.
| It has things like a task scheduler for scheduling http posts to
| developers apps, or monitoring email boxes and posting to web
| hooks when email comes in. It _does_ need maintenance but I think
| it beats putting that logic in each app.
| nicktorba wrote:
| There seem to be a lot of components like this that need to be
| developed.
|
| For example, there are plenty of tools to create ML model
| endpoints (send data, receive prediction) but many times, those
| models are not customer/user facing. They will be hit by some
| other program/system (often on a regularly timed schedule).
|
| One solution is to embed the model in the application that
| needs the predictions, but this can get hairy when it comes to
| updates/maintanence/etc.
|
| Over the next few years, we will see some generalized "model
| invocation" components that integrate with common data sources
| for these use cases.
|
| You can see an inkling of this with the seldon-core kafka
| integration: https://docs.seldon.io/projects/seldon-
| core/en/stable/stream...
| jusonchan81 wrote:
| We have been using Netflix Conductor and it has a ton of features
| that helps with glueing systems.
| pphysch wrote:
| The top two comments as of now, paraphrasing:
|
| 1) "Complex systems should be designed like video games"
|
| 2) "Complex systems should be designed around a huge event bus
| SaaS"
|
| Video games/engines are architected around responsiveness (low
| latency) & content creation.
|
| Sending every little event to a remote SaaS for processing is
| sure to add tremendous latency.
|
| To be fair, many video games due make heavy use of event
| processing/message passing. But the architecture of a video game
| event system & SaaS event bus are quite different for
| performance, security, reliability reasons.
|
| Complex distributed systems are, well, complex, and can't be
| reduced to a simple formula or architecture. Better to spend time
| accurately understanding the problem that is being solved.
| sigmonsays wrote:
| is kubernetes considered a glue system?
| bovermyer wrote:
| Not really, in my opinion. It's a platform.
|
| I think a glue system is meant to connect disparate pieces that
| otherwise could not interact with each other.
| chubot wrote:
| I'd definitely say so. Combined with Docker, its job is to
| "tame" / coordinate / connect extremely heterogeneous software,
| much of which wasn't designed to run on clusters.
|
| I think of shell as the language for glue (and we need a better
| one). and I explored this relationship in a followup to a
| popular post [1]:
|
| http://www.oilshell.org/blog/2021/07/cloud-review.html
|
| [1] https://news.ycombinator.com/item?id=27903720
| dehrmann wrote:
| It's large and designed to the point that no, it isn't, but
| interestingly, it usually replaces a lot of glue systems.
| coding123 wrote:
| Seems like people think so, but I would say no, it's just a
| platform that runs VMs. It doesn't really care about the data
| running through services other than making sure it makes it to
| the destination. That being said, it's quickly becoming easier
| to plug and play traditional message busses or queuing systems
| by deploying a container to it. So it helps simplify the
| addition of some kind of glue system.
| PaulHoule wrote:
| The article makes the interesting claim that Clojure, Elixir and
| similar languages are good for "glue" but doesn't make any effort
| to develop the argument.
|
| For a long-time we've heard the opinion that complex systems are
| best written in a combination of a "systems" language (say C++ or
| Java) and a "scripting" language (say Tcl, Lua or Clojure.)
| That's how video games are written, for instance.
|
| Clojure and Elixir have the particular angle of being suitable
| for concurrency, which more conventional scripting languages
| (e.g. Python, Ruby, PHP, ...) fall down at thanks to the various
| global interpreter locks.
| marcinzm wrote:
| >global interpreter locks.
|
| Does this really matter for a glue language in a distributed
| system? Async for non-blocking IO and then just run multiple
| processes. Allows for things like killing a process and letting
| the OS cleanup. For example, in Java you're not supposed to
| just kill a thread.
| PaulHoule wrote:
| It's an interesting question.
|
| I've written a lot of "glue" in Python using asyncio,
| websockets, AQMP and similar things for my IoT system at
| home. The coding is fun and the performance seems excellent,
| but that's because the workload is low. If I was at the point
| where I needed more than one core worth of performance I
| wouldn't be so happy.
|
| One case where I am a little frustrated with Python now is a
| "batch job" that processes and resizes images. There are
| numerous half-baked ways to farm out the work to separate
| processes. These would be "good enough" but they are still
| half baked, and the frameworks that give a little structure
| to the process have one thing in common: they don't run on
| Windows.
|
| In a pure "glue" case there is also an Amdahl's law kind of
| situation: it's not crazy to have a 32-core or more system
| today and a single-threaded "glue" system that dispatches
| tasks will have a hard time keeping those cores busy if it is
| spending even 2% of the CPU effort that it takes to do the
| real work.
| marcinzm wrote:
| Your example doesn't makes sense to me. If there was no GIL
| then you'd still need a dispatch thread. And if that thread
| took 2% of the same compute then you're out of luck. The
| lack of a GIL doesn't magically distribute compute across
| many CPUs. You could off course make a minimal dispatch
| thread and then offload the 2% to other threads but you can
| do that with a minimal dispatch process as well.
| lazide wrote:
| Have you tried? I've had good luck using Perl (years ago)
| under very heavy load and it worked very well.
| rented_mule wrote:
| I've tried repeatedly over the last 25 years, in Python,
| with a lot of success for large amounts of computation.
| Architecture and algorithms matter _so_ much more than
| limitations like the GIL, slow function calls, etc. And
| in my experience, Python gives me the time and mental
| bandwidth to focus on architecture and algorithms. It 's
| hard work, but in my experience it's not as hard as
| getting the same level of success without a very high
| level language gluing things together.
|
| One memorable situation was sitting in a sales meeting in
| ~2003, showing the speed of one such system. We let the
| potential client customize some input data, then ran a
| computation (the point was to show that something that
| used to be a batch process could now be interactive). It
| took ~30 seconds on our laptop running on battery power.
| On the spot, their CTO accused us of lying because it
| took days of dedicated mainframe time for them to run the
| same computation. Our sales person later asked if we
| could introduce configurable delays that we'd only turn
| on during sales presentations so that it would be easier
| to believe and less threatening. Our original
| implementation was 1000s of lines of Python and would
| take weeks to run. Over time we optimized that down to
| about half a day of run time. Then we figured out that
| the thing taking all the time was equivalent to finding
| shortest paths in graphs over and over again. So we found
| an open source C extension with a nicely optimized
| implementation of Dijkstra's algorithm and our Python
| code reduced to 100s of lines and was literally
| unbelievably fast.
|
| Today I'm working on a simulation system for use by
| people who are experts in game theory. The thing being
| simulated is an algorithm in an online system (that
| almost everyone reading this in the US has used) that
| does server side optimization of the selection and
| placement of content during web / mobile app page loads.
| We want to share code between online and simulation to
| avoid semantic differences. The online system is in
| Python. Python is what our math folks know. The algorithm
| currently runs tens of millions of times per day online
| with billions of inputs, all on a small cluster of
| machines. Online we need low latency, O(1 ms) per run.
| Offline we need as much computational bandwidth as we can
| get to keep math folks going - improvements to this
| algorithm can mean big revenue wins. We haven't shipped
| yet, but the current version replays hundreds of millions
| of runs (10s of billions of inputs) per minute in a large
| Spark cluster (developers only ever see Python, not the
| PySpark API). The online backend runs the exact same
| Python code in O(100 us) with a Python backend (only one
| non-builtin extension is used online - Pyrsistent, which
| gives us efficient immutable data structures - the online
| backend uses FP concepts to avoid complexity).
|
| In my experience, running up against things like the GIL
| for doing large amounts of computation means I'm stuck in
| a local optimum for speed. Backing up and thinking about
| my architecture / algorithms gives me a real chance of
| making improvements that show that the GIL wasn't my core
| issue. Eliminating the GIL might get me an order of
| magnitude speed improvement (often at the expense of more
| complicated code). Finding better architecture /
| algorithms often gets me several orders of magnitude of
| speed improvement and simplifies my code at the same
| time. A very high level language that is good at gluing
| things together gives me more time / options for doing
| that.
| PaulHoule wrote:
| I've written high performance code in Java and often been
| able to quickly recruit many cores just by adding an
| Executor. (as opposed to "try to recruit many cores and
| recruit a few cores" with parallelStream(), Scala's
| actors and the many snake oil parallelism libraries --
| there are more of those than there are cores on a big
| machine.)
| Lammy wrote:
| > which more conventional scripting languages (e.g. Python,
| Ruby, PHP, ...) fall down at thanks to the various global
| interpreter locks
|
| For what it's worth, each instance of the new "Ractor"
| concurrency primitive in Ruby 3 has its own interpreter lock:
| https://docs.ruby-lang.org/en/master/doc/ractor_md.html
| dimitar wrote:
| Don't know about Elixir, but Clojure has a strong focus on
| dealing with external system the following way:
|
| - convert the data output/need as input in core data structures
| - Don't use classes or custom types to encapsulate data and
| make it accessible using a custom interface - instead
| manipulate the data structures instead using the large standard
| library. - convert the data back into the required format for
| storage/output/etc.
|
| Very ETL-like, that is needed as different systems that need to
| be glued together may be written in different languages and
| most likely will export serialised data or write in a database.
| Certainly binary formats are rare nowadays.
|
| Abstractions like the one for sequences allow reuse of the same
| function for different data structures (you don't have to
| convert things into lists for processing). It's possible to
| have other, non-core data structures implement the same
| abstractions, so that the existing library can be used.
|
| ORMs are not desirable with that approach, for example. The
| query result tables are usually converted into things like
| Clojure hash-maps instead being created into unique objects
| with different ways of accessing the encapsulated data.
|
| Then you have functional programming, which while nice by
| itself also allows using your cores efficiently. Less of a need
| for additional "infrastructure" glue that way. Asynchrony is
| also provided using libraries like core.async or agents.
| tonyhb wrote:
| Been thinking about glue systems a bunch - the platform I'm
| working on (Inngest [0]) solves this kind of problem.
|
| I think "glue systems" can be generalized by saying: systems that
| consolidate events, and react to events in real-time. Most
| everything that happens in your system is an event, and things
| need to react to this. For example:
|
| - A glue system will almost certainly plumb stripe events info a
| billing system, plus other systems for things like failed
| payments.
|
| This system shouldn't go down (HA). It should be able to react to
| changes easily - as biz ops change frequently. It should be
| auditable, easy to debug, and easy to deploy.
|
| The first pass at this in startups is usually... just doing stuff
| in the API request, or building webhook endpoints that manage
| this for you within the API. Then, you might add Sidekiq, or
| SQS/Lambda. Then, you might build out event-driven architecture
| using SQS/Kafka. It's kind of a pain, and still it's not easy to
| grok, debug, or architect.
|
| Inngest handles all of this for you, though. It consolidates
| events from every system (internal and external, via APIs,
| webhooks, oauth, etc.) and then allows you to run DAG-based
| workflows in real-time whenever events are received. With full
| logging, debugging (step-over debugging), retries, user-auditing,
| changelogs, version control, etc.
|
| I think that event-driven systems and glue-based systems overall
| haven't had much love in the developer UX, but I'm hopeful we can
| change that. If you're interested in using us or working with us
| on building the platform, ping me :)
|
| [0] https://www.inngest.com
| lukeramsden wrote:
| Inngest looks very cool - is there any chance of this being an
| "open-source SaaS" / open-core type thing like, for example,
| TimescaleDB or Temporal? Would love to contribute to something
| like this, but not as paid employment.
| tonyhb wrote:
| Yeah, soon! I need to redo some of the DAG-based workflow
| logic to rip out our internal scheduler (we have two:
| internal and nomad via firecracker/gvisor).
|
| Once that's done and I've refactored some stuff I'm planning
| on opening up the core runner. Our FoundationDB persistence
| we're working on may likely remain closed - we will provide
| an interface which allows you to persist stuff and a psql
| adapter :)
|
| That'll likely happen early next year.
| tut-urut-utut wrote:
| So many words to just say ESB. I know it's not fancy and
| falling out of fashion, but what you describe here is what
| different ESB/MIddleware systems have been doing for ages.
| tonyhb wrote:
| Yeah, agree, there's a large sector of overlap. Basically
| that, but modern, and built for engineers with a polished
| experience.
| fortydegrees wrote:
| Love your landing page. I know exactly what you do and how you
| can be of value for me without even scrolling. Well done!
___________________________________________________________________
(page generated 2021-09-21 23:02 UTC)