[HN Gopher] A simple core.async job system in Clojure
___________________________________________________________________
A simple core.async job system in Clojure
Author : janetacarr
Score : 106 points
Date : 2024-05-25 02:19 UTC (20 hours ago)
(HTM) web link (blog.janetacarr.com)
(TXT) w3m dump (blog.janetacarr.com)
| samsquire wrote:
| Thank you for this post Janet.
|
| I think my technical perspectives have moved in a similar
| direction: keep things extremely simple with minimum moving
| parts.
|
| I maintained (automated upgrades) a RabbitMQ cluster and while it
| is powerful software it is operationally expensive. For a side
| project you probably just batch process in a cron.
|
| If I were to take the approach in this blog post I would want
| everyone on the team to be extremely familiar with the model of
| task running: stuck jobs, timeouts, duplicate jobs, client
| disconnects and retries, stuck "poison" jobs seem like issues you
| might face.
| janetacarr wrote:
| I'm glad you liked it!
|
| I'm using a PaaS, so I didn't want to pay the extra money for a
| cron job. Maybe not a wise a choice, but here we are.
|
| Totally agree about the issues I might face. I'm glad that the
| Clojure REPL is a thing, so I can test out all of a job's
| functionality before sending it off to async land.
| lkrubner wrote:
| I appreciate this emphasis on simplicity and I recommend it to
| others. I've seen trends in the other direction which I think are
| dangerous. I notice that as some companies adopted a
| microservices approach there was a tendency to allow in more
| technologies than necessary.
|
| "Premature polyglot programming" is a disease that afflicts
| certain startups. While any sufficiently big company will be
| polyglot, a small startup needs to stay focused on a limited tech
| stack, for as long as possible. After all, each new technology
| requires someone on staff who has the skill for that technology,
| and when your startup is small, your talent pool will also be
| small, and so it becomes common that you only have 1 person on
| the team who knows how to run some particular technology (Kafka,
| RabbitMQ, Redshift, DynamoDB, MongoDB, Tornado, etc).
|
| So you end up with a lot of single point of failures, where the
| "single point of failure" is the single engineer who knows how to
| run the technology that you've made critical to the company. Be
| wary. Avoid this if possible.
|
| I notice, especially, as the tech industry developed better tools
| for managing complex devops situations (Docker, Kubernetes,
| Terraform) there was a tendency from some engineers to think
| "Nowadays it is easy to run 10 different technologies, therefore
| we should run 10 different technologies." Be wary of this.
|
| Janet Carr's emphasis on simplicity is something we should all
| imitate.
| janetacarr wrote:
| Wow, thanks for the kind words.
|
| I've been bitten by the ops hellscape of microservices and
| various tech many times throughout my career, and it's
| definitely shaped how I think about designing and building
| software now.
| signal11 wrote:
| I agree, and a lot of this applies to large companies as well.
|
| Of course larger orgs will have higher tolerance for a wider
| set of technologies, but this tolerance is not infinite, and
| choosing tech because that one team was a fan creates staffing
| problems over time, especially as the original devs move on.
| Eventually you get "estate sprawl" that's difficult to manage.
|
| Choosing to minimise dependencies is definitely a (good)
| choice. The key is the right tech for the task at hand. Equally
| org-wide "use Oracle for everything" mandates aren't helpful.
|
| Balancing these at a large org definitely needs good
| engineering leadership!
| BaculumMeumEst wrote:
| You can use this same line of reasoning to avoid using Clojure
| entirely.
| janetacarr wrote:
| Yeah but that's not nearly as fun!
| BaculumMeumEst wrote:
| Very true.
| Waterluvian wrote:
| This is what I think about most when looking at languages. "How
| painful would it be if I implemented the minimum now and wanted
| to add more capability later?"
|
| Some languages make it very painful to not have a mostly
| complete design upfront and then you're trapped adding
| complexity just in case.
| smrtinsert wrote:
| 100% agreed. Unfortunately overdoing it on complexity early on
| seems to lead to a loss of the faith in engineering decision
| making even after overengineering leadership is removed.
| tombert wrote:
| core.async was actually the "killer app" that sold Clojure to me.
| I've always liked Go's CSP-style concurrency, but I wanted a
| proper functional language to go with it. It turns out that
| Channels/Queues are just an insanely good abstraction for getting
| threads to talk to each other. The closest general-purpose
| library I've found that helps bring that to the rest of the world
| has been ZeroMQ, which is great but not as easy and nice to use
| as core.async.
|
| That said, I really don't think RabbitMQ is so bad. The default
| docker configuration for it is fine for most cases you're likely
| to come across for all but the biggest applications, and I do
| think that having the ability to have jobs restart/requeued when
| a worker crashes out of the box is worth the tradeoffs for
| slightly increased complexity. I usually just use a single docker
| compose and glue everything together in one big ol' YAML.
|
| Still, there's obvious value in avoiding dependencies, so it of
| course depends on the size and scope of your project. If you
| think you're going to end up distributing this over 100 nodes,
| something like RabbitMQ or ActiveMQ might be worth playing with,
| but if you're just doing relatively small (at you appear to be in
| this project), it's probably the correct choice to mimic whatever
| behavior you need with core.async.
| elwell wrote:
| > core.async was actually the "killer app"
|
| For Clojure _Script_ , it was React. Immutable state /
| functional programming fits so nicely, and hiccup syntax (JSX
| as simple vectors) is the perfect declarative (yet
| transformable) DOM structure.
| tombert wrote:
| Oh absolutely, no argument here at all.
|
| I don't do a lot of frontend, but for anything that does
| involve frontend I kind of refuse to use anything but re-
| frame. It took a bit of dogma-acceptance, but once I did I
| _really_ liked it for all the reasons you listed, and in
| particular that there 's no special JSX crap, and instead
| just using vanilla vectors and symbols. This has the
| interesting advantage where you can generate your HTML logic
| without actually having to import the re-frame/re-agent
| logic, which can be useful if you want to decouple logic from
| rendering.
|
| I haven't done any benchmarks, but I haven't noticed re-frame
| being slower than vanilla React, it seems to be a comparable
| speed, but as stated I don't do much frontend.
| yayitswei wrote:
| Want to put in a plug for Electric Clojure. More composable
| than React (mix server and client code in the same function!)
| and more performant. I've had a lot of fun using it over the
| past year, and it's IMHO the most transformative tech I've
| seen in web dev.
|
| Wrt the original article, effect systems like missionary
| (which electric is based on) can be used to complement or
| replace queues, often resulting in simpler code.
| mark_l_watson wrote:
| I appreciate Janet's aversion to adding RabbitMQ to the system. I
| had to use RabbitMQ with Common Lisp a year ago and in was a
| pain.
|
| It looks like Clojure has better RabbitMQ client options than
| Common Lisp, but still, very cool to build something on
| core.async and keep things cleaner and simpler.
| jwr wrote:
| Yes! core.async is a great tool and I've used it to solve a
| number of problems effectively. I'm very happy with what I get:
| reliable, predictable systems.
|
| A real "whoa" moment comes when you realize you can also use
| core.async in ClojureScript :-)
___________________________________________________________________
(page generated 2024-05-25 23:01 UTC)