[HN Gopher] Delayed Job vs. Sidekiq
___________________________________________________________________
Delayed Job vs. Sidekiq
Author : unripe_syntax
Score : 105 points
Date : 2022-02-15 12:55 UTC (1 days ago)
(HTM) web link (blog.appsignal.com)
(TXT) w3m dump (blog.appsignal.com)
| renenw wrote:
| edlebert wrote:
| If all you need is lightweight jobs like transactional email, I
| highly recommend suckerpunch.
| gw67 wrote:
| Interesting content but AppSignal lacks lot of features. I've
| tried to use it once for a Rails app and I haven't foud not
| usefult insights. Would be interesting: - Monitoring performance
| and suggest how to fine tune the app (# Threads, #Words, RAM, CPU
| etc..)
|
| - Identify memory leaks
|
| - Identity memory leaks on Postgres
|
| Why spending so much time writing content when you could have
| focused on your product instead?
| prescriptivist wrote:
| Do anyone (NewRelic or DataDog) actually do that? I have found
| New Relic focuses on large scale observability (distributed
| tracing) not actually understanding low level implementation
| specifics and building tuning profiles. NR doesn't even do
| request level object allocation tracking like AppSignal.
| toomanybeersies wrote:
| DataDog offer memory profiling for Java, I think, but not for
| Ruby.
| toomanybeersies wrote:
| Memory profiling is hard, especially for Ruby.
|
| To echo prescriptivist's comment, neither DataDog nor New
| Relic's profilers offer memory profiling for Ruby applications.
| lambdadmitry wrote:
| I can only wish there were more options like that in Python
| space. Celery is just a horrible tire fire and everything else
| seems to be either pretty minimalist, somewhat immature, or both.
| adrianthedev wrote:
| We chose `delayed_job` at Avo for it's simplicity and ease of
| use.
|
| https://avo.cool
| WJW wrote:
| One thing I would love to see is a Fiber-based job runner for
| mailers and other non-cpu-intensive jobs that spend most of their
| execution time waiting (say, doing API calls). It seems like that
| would be one of the absolute sweet spots for the new Fiber
| scheduler stuff in Ruby 3.0. Preferably it'd be sidekiq
| compatible but you could have the fiber runner listening to one
| queue and the normal sidekiq runner(s) to another queue.
|
| I've been toying around with this idea for a while and the main
| blocker is the incompatibility of Rails (mostly AR) with some of
| the Fiber stuff. That is slowly getting cleared out though.
| weatherlite wrote:
| I'd like to see it as well. However It's not immediately
| evident to me by how much this will improve things; afaik you
| will hit a bottleneck that is in the number of available db
| connection (since each fiber, just like threads, has to borrow
| a db connection from the pool).
|
| Fibers have a lot of potential but not in the one request per
| thread/fiber model we have now. Or perhaps I'm thinking about
| this wrong?
| WJW wrote:
| I have in one of my applications a background job that
| basically does the following:
|
| - Some quick database lookups (~ 10 ms total)
|
| - An API call (depends on the time of day, but between
| 500-800 ms)
|
| - Another quick database insert (~ 20 ms usually)
|
| The job only needs a db connection for about 5% of the total
| runtime of the job and can give the connection back to the
| pool while the API call is running. Many mailer jobs in the
| system have a similar db-to-api-call ratio.
|
| (Also, I am of the opinion that all but the very biggest
| sites should not have any problems with db connections. Get a
| database that handles tons of connections gracefully like
| MySQL or use something like pgbouncer when using Postgres.
| Having low connection limits should not be a problem in 2022)
| berkes wrote:
| I agree with most of the findings in that article, but I miss one
| very important one: consistency.
|
| Jobs in your queue in the database can be committed in the same
| transaction that makes the mutation. With redis you have no such
| thing. Leading to (been there, it hurt) potentially missing jobs
| or queuing jobs that have no associated mutation in the db.
|
| Furthermore, a big downside of redis is that it's promise of
| consistency is low. Mostly by design: redis is not meant to be
| the primary, one and only store of truth. Commonly, people store
| stuff in redis that can be re-built from database or some
| eventlog or so. But not sidekiq. If your redis starts failing
| (been there, it hurts), e.g. though OOM errors, you are loosing
| data. Jobs, with their parameters, are dropped, often
| irrecoverably. And with Rails' common set-up, this means you are
| irrecoverably loosing data: there simply is no way to find what
| emails you did not send, nor any way to re-queue them. There is
| no way to re-enqueue those "generateReport" jobs using the data
| at that point in time and so on.
|
| Basically: Redis is quite certainly not the best tool to store
| your job-queues with associated data, in. Sidekiq is the less
| sturdy approach of both.
| thejosh wrote:
| With listen and notify in Postgres, it's a fantastic choice for
| a job queue if you're not crazy scale (which is most apps,
| honestly).
|
| Especially with transactions as mentioned, with elixir and ecto
| and Oban, you can ensure your jobs are inserted properly. And
| it's easily testable.
| jdc0589 wrote:
| I have strong opinions on this, but IMO if you are thinking
| about using PG pub/sub, you'd almost always be better served
| by just running a worker that reads from a logical
| replication slot. Little bit heavier lift, but there's plenty
| of reference code out there, and you get consistency +
| delivery guarantees.
| prescriptivist wrote:
| Our solution was to build middleware/interfaces that only push
| jobs after a transaction has committed. There is still room for
| some inconsistency -- a failure to enqueue a job doesn't
| rollback a committed transaction, for instance, but in practice
| that is rare.
|
| Agreed on the Redis health being critical and if it's unhealthy
| you can't trust anything. We over-provision compute and memory
| and enable persistence as a result. We are also mindful of our
| payload sizes, which we also compress. Additionally any jobs
| scheduled beyond a certain time threshold are committed to the
| DB, not Redis, and we have a periodic job that adds them back
| to Redis as we approach their scheduled time to run. This is
| good for both durability and keeping the surface area of Redis
| work smaller.
|
| It's not perfect and there is probably a day where we switch it
| out but for now we have developed a deep understanding of
| Sidekiq and Redis and get to leverage the larger Sidekiq
| ecosystem. This has worked well for us to the tune of about 35
| million jobs a day.
| blueplanet200 wrote:
| > There is still room for some inconsistency -- a failure to
| enqueue a job doesn't rollback a committed transaction, for
| instance, but in practice that is rare.
|
| If it happens though is it a big deal? I've seen Sidekiq jobs
| used to move money, for instance. This can result is Bad
| Times. Right tool for the job, if consistency is a
| requirement Sidekiq I think gets incorrectly applied in a lot
| of applications.
| prescriptivist wrote:
| I don't move money, but if I did I would do it like we do
| for most objects in our system (like interacting with S3
| objects for example), I'd represent that transaction (in
| the money sense) in Postgres with a state and the job would
| be expected to do work and update the state. The
| transacting job itself could be queued immediately for
| convenience but ultimately there should be a periodic job
| that attempts to transition states of any outstanding
| transactions and the job that does that work should be able
| ensure uniqueness (in the runtime sense) and idempotency.
|
| In this sense the job acts on data, it doesn't hold data,
| and you can build in backstops in the event of any kind of
| failure, not just Redis. You'd want to build this into your
| architecture regardless of what your background job tooling
| and infrastructure you use.
| blueplanet200 wrote:
| > but ultimately there should be a periodic job that
| attempts to transition states of any outstanding
| transactions and the job that does that work should be
| able ensure uniqueness (in the runtime sense) and
| idempotency.
|
| You've re-implemented a job queue in the database.
| Fire-Dragon-DoL wrote:
| We worked on the same problem, and what you described
| works, however it also happen to be exactly what delayed
| job does.
|
| Which bring me to the next point: Sidekiq should never be
| the default choice. It brings distributed systems
| complexity to an audience that's trained to work on a
| monolithic database.
|
| Delayed job or similar should be the default, with
| sidekiq relegated tp jobs ready to be lost
| scosman wrote:
| Transactional consistency is nice. Maybe worth it for some low-
| volume jobs.
|
| The downside that comes with it is using your primary DB as
| your queue is that the queues in an app tend to absorb all the
| exceptional conditions (queue grows to millions during worker
| outage, backfill which needs to process millions of items).
| Using your primary DB means all that disk usage, memory usage,
| and IO hit your primary DB in exceptional ways and impacts your
| app in more ways than just queue delays. It also scales much
| less linearly than other queue options.
|
| A nice middle ground is to store all your data and state in
| your DB transactionally, and only queue tiny lightweight jobs
| with database primary keys (database: sentWelcomeMailDate=nil,
| queue: sentWelcomeMailForUser:1234).
|
| This is another case for neither DJ or Sidekiq. A fully hosted
| option like SQS addresses the durability concerns of Sidekiq,
| and the scaling concerns of both. I've queued 1B+ jobs in SQS
| and watched the workers slowly burn through them over the
| course of a month.
|
| DJ and Sidekiq can go quite far. I've used both and patched
| some perf issues on DJ. The lesson learned was queues at scale
| are hard distributed systems problems - you probably don't want
| to have to know how they work, or host the distributed system
| yourself.
| berkes wrote:
| > Maybe worth it for some low-volume jobs. > ... > The lesson
| learned was queues at scale are hard distributed systems
| problems
|
| I'd go so far to say that when you need such scale, Rails or
| Job Queues like Rails' are not the solution. Event-based
| architectures (event-sourced, event-buses, actors etc) are
| far better suited.
|
| Rather than trying to solve all the problems that those
| architectures were explicitly designed for in
| redis/sidekiq/rails, just move to those architectures.
|
| You won't have the nice "rails generate CreditCard" anymore
| /s, but you'll gain a lot of consistency, simplicity,
| performance and architectural sanity.
| cardy31 wrote:
| You can run Redis in a persistent setup. So each Redis instance
| has a replica and both master and replica write out to disk.
| You still don't get transactions, but depending on your scale a
| transaction for every job you enqueue may be impossible.
|
| If you're small enough that you don't knock over your DB by
| putting jobs in it then the transactions are nice!
| jmileham wrote:
| As you say, looks like redis-raft is capable of solving
| durability and consistency in a replicated environment as of
| 2020, which is welcome news:
| https://jepsen.io/analyses/redis-raft-1b3fbf6
|
| At Betterment we use our OSS mostly-compatible fork of
| Delayed::Job referenced elsewhere in the comments to enqueue
| and work millions of jobs a day and sleep much better at
| night with the at-least-once delivery semantics if-and-only-
| if the related transaction commits which you can't get
| without some form of integration with your primary database.
| [deleted]
| dorianmariefr wrote:
| Delayed Job seems to have been abandoned, a better alternative
| that is a fork with many improvements is
| https://github.com/betterment/delayed
| javawizard wrote:
| Another fork is [0], which adds debouncing, serialization,
| recurring jobs, and other useful things.
|
| (Disclaimer: I used to work at Instructure.)
|
| [0] https://github.com/instructure/inst-jobs
| xal wrote:
| Yes sorry, please use betterment's fork. I got busy with other
| projects since writing delayed_job and had to prioritize.
| dorianmariefr wrote:
| like running one of the top 100 companies in the world :)
| thanks for everything
| jrochkind1 wrote:
| Consider adding this to the top of the README of delayed_job?
| jhas78asd wrote:
| I helped maintain delayed_job_web as a UI. It has fallen
| behind substantially now though. If folks are looking for a
| UI - it would be great to update it for modern versions of
| Rails. https://github.com/ejschmitt/delayed_job_web
| xal wrote:
| very cool! thank you
| Fire-Dragon-DoL wrote:
| https://github.com/que-rb/que
|
| This one seems to be the most performant. By a lot too, from my
| understanding (haven't ran any benchmark myself, but the readme
| shows some good postgres knowledge)
| werdnapk wrote:
| One big difference if you're changing forks is to take note of
| "at least once delivery":
| https://github.com/betterment/delayed#at-least-once-delivery
|
| If you're not aware of that, simply switching repos may cause
| some surprises.
| jmileham wrote:
| Just to note that at least once delivery is the best case
| scenario if you configure Delayed::Job correctly, we just
| made it impossible to configure otherwise in our fork. The
| only alternative is at-most-once delivery, which puts you in
| Sidekiq territory with the potential for silent job loss. The
| semantics of exactly once delivery can only be achieved by
| domain-aware code, and the way to do that with any form of
| background job system is building jobs that are internally
| idempotent.
| byroot wrote:
| Seems maintained to me:
|
| https://rubygems.org/gems/delayed_job
|
| https://github.com/collectiveidea/delayed_job
| dorianmariefr wrote:
| > 4.1.10 - January 18, 2022 (40 KB) > 4.1.9 -
| December 09, 2020 (40 KB)
|
| Ok last I checked it was before January and I wanted to
| upgrade to rails 7 and couldn't so ended up moving to Sidekiq
| ghiculescu wrote:
| Rails 7 support only just got added
| https://github.com/collectiveidea/delayed_job/pull/1161
| dorianmariefr wrote:
| Sets a bad precedent, I'm on rails's main branch and this
| was the big dependency holding me back for a while.
| Sidekiq works with rails's main though
| deedubaya wrote:
| I've been using sidekiq (paid, free) for years, the next app
| (mostly rails apps, but not exclusively) I implement background
| processing probably will not include it.
|
| Reasons to avoid Sidekiq not mentioned:
|
| - Sidekiq does not implement all of or discourages use of
| activejob abstractions (retry_on, globalid for example)
|
| - Paid support can be curt, hostile, or down-right "works on my
| machine, sorry!"
|
| - Sidekiq's better performance has never materially mattered for
| my applications; they're almost always waiting on IO from the db
| or some net connection.
| mperham wrote:
| I'm sorry if any of our previous interactions were less than
| professional. I often answer support queries from my phone when
| out and about, so tossing off a curt, one sentence reply will
| happen.
|
| I generally get multiple app issues per day so I can't afford
| to reproduce every one: just yesterday I asked a customer to
| provide a reproduction, they worked on it and found the issue
| was in another gem their app used. At my scale with ~2000
| customers, this happens almost every day.
| yesdocs wrote:
| I am constantly reminded how lame ruby and the rails ecosystem
| is. Consistency was solved long ago in the .Net ecosystem and
| using standard tools would not lead the author down a dark road
| of 'it just works' but it really doesn't. All this 'you could use
| xxxx but I haven't setup my instance that way' is just fan boys
| trying to pull the wool over your eyes to hide the fact that the
| tools are really crap. Consistency is paramount in the enterprise
| and anything less is just a lie that the system works. It
| doesn't.
| clajiness wrote:
| OK, ha
| noodle wrote:
| Most of these problems are solved in Rails, too. If you want
| consistency, there are options for it. This is an article about
| two non-enterprise (aka, non-paid) options.
| berkes wrote:
| Negative tone aside, indeed: the problems that Rails "solves"
| here have long been solved in computing. Not just .net. In Ruby
| too, just not in Rails.
|
| Event-based architectures such as event-sourcing, -listeners,
| -subscribers, message-buses, actor pattern. All of them solve
| the problems that Rails tries to solve with delayed jobs. But
| does it with different tradeoffs.
|
| Rails is first and foremost an MVC setup with a heavy and tight
| coupling to the database. That is a poor fit for apps or
| domains with lots of sequential logic, messaging, services or
| orchestration. Rails is perfect for reasonable simple "CRUD",
| its less perfect for "CRUD+business-logic" and a very poor fit
| for complex domain models with (eventual) consistency and/or
| message passing.
|
| For the middle-scenario, Id -personally- forego Rails, and
| choose or build an archictecture that focuses more on logic
| than on "getting stuff in and out of a database". But I see why
| people will choose Rails there too, and then bolt the logic and
| consistency on top of it.
| toomanybeersies wrote:
| As a counterpoint, I used to work on an automated testing
| tool for websites and mobile apps, which was built entirely
| using Rails (with Resque for background jobs).
|
| We had customers setting up tests which could cover thousands
| of pages and run for hours (or even days).
|
| Sure, Rails probably wasn't the ideal tool for the job, but
| it worked fine.
| xiljin wrote:
| I've been using Sidekiq for years, rock solid and no complaints.
|
| However, this recent addition to the space seems to be gaining
| traction and appears to have an excellent feature set -
|
| https://github.com/bensheldon/good_job
| waffle_maniac wrote:
| Without the pro version it's not reliable:
|
| > If a Sidekiq process crashes while processing a job, that job
| is lost. [0]
|
| We use Sidekiq Pro at work but I find it unfortunate that
| reliability is a pro feature. It's really hard to tell a
| product owner: Yeah, we might lose a few jobs once in a while.
|
| [0] https://sidekiq.org/products/pro.html
| jdc0589 wrote:
| agree. its not a lot of money, or it didn't used to be last
| time I looked. If your org relies heavily on it just pay
| waffle_maniac wrote:
| We do pay for it.
|
| Perhaps you could address my point? Open source background
| job software that isn't reliable by default? That makes
| sense?
| aaronbrethorst wrote:
| I just looked this up. It's $995/year, or approximately 4%
| of the fully loaded cost of an FTE software engineer at
| most companies. i.e. about 3 weeks of a developer's time.
| waffle_maniac wrote:
| I understand paying extra for niche features. Isn't
| reliability a core feature of background job software?
| [deleted]
| aaronbrethorst wrote:
| Seems like a good reason to pay Mike.
| grk wrote:
| The problem with postgres-backed queues is they can't be used
| with connection pooling, so your workers are unnecessarily
| using up DB connections.
| waffle_maniac wrote:
| I thought that was an ActiveRecord problem. We use MYSQL and
| have a lot of long-lived connections.
| grk wrote:
| Not sure how it works with MySQL, but with postgres the
| most common pooling options use transactional mode, which
| doesn't support LISTEN/NOTIFY.
| Lio wrote:
| Yeah I'm a big fan of Good Job too.
|
| As long as you stick to pure ActiveJob features it really
| shouldn't be too difficult to scale from in memory based queue
| to PostgresSQL backed Good Job and then to Redis backed Sidekiq
| if and only if you need to.
| stephenhuey wrote:
| Yes, I wanted a PG-based one and reached for Good Job on a
| new project since it was recently created and with Active Job
| in mind from the start. Check out the intro:
|
| https://island94.org/2020/07/introducing-goodjob-1-0
| michaelbuckbee wrote:
| One aspect of this article that's touched on but not really
| explored in the article is the greater Sidekiq ecosystem. Beyond
| just the core Sidekiq application working well there are a host
| of well maintained open source add-ons to Sidekiq that are
| tremendously helpful.
|
| Statistics https://github.com/davydovanton/sidekiq-statistic
|
| Failure Handling https://github.com/mhfs/sidekiq-failures
|
| Unique Jobs https://github.com/mhenrixon/sidekiq-unique-jobs
|
| All of which also extend the web UI for Sidekiq which is
| incredibly useful for both debugging and having a handle on
| what's with your queues.
|
| Finally, if you're going to be using Sidekiq in any serious way
| I'd recommend Nate Berkopec's "Sidekiq in Practice" -
| https://nateberk.gumroad.com/l/sidekiqinpractice
|
| Beyond being an incredibly useful resource on its own - you get
| access to a very active private Slack that is filled with other
| very helpful developers who are using Sidekiq.
| acejam wrote:
| This article matches up with most of my own findings as well.
| I've been using Sidekiq for almost 10 years now. I currently use
| it with a project that just hit over 1 billion jobs.
|
| To me, Sidekiq is the perfect example of keeping something
| simple. A basic Redis + Sentinel deployment with persistence
| enabled and multiple read replicas has allowed me to achieve
| this.
|
| Scaling Sidekiq workers is also incredibly easy. I have dedicated
| job queues for various categories, and I simply create a new
| Kubernetes deployment for each category. Each deployment is set
| to only process a specific queue. This allows me to throttle how
| quickly each queue gets processed, simply based on replica/pod
| count.
| rikkipitt wrote:
| I've recently discovered jemalloc, specifically when used with
| Heroku.
|
| "Using jemalloc instead of regular malloc helps too. The exact
| way to do this depends on the platform you use, but it is pretty
| simple on Heroku. Just set heroku-buildpack-jemalloc as the first
| buildpack (ahead of the heroku/ruby buildpack)."
|
| FYI, remember to set JEMALLOC_ENABLED=true in your env to
| actually turn it on.
|
| https://github.com/gaffneyc/heroku-buildpack-jemalloc
| mawaldne wrote:
| Been pretty happy with delayed jobs for a long time. I've used it
| a number of times at jobs for small to medium businesses and it's
| alway worked well. And it's nice you don't have to setup any
| additional infra. Just use your database.
|
| No disrespect to sidekiq. I see the use case for it and it seems
| like it is better supported.
___________________________________________________________________
(page generated 2022-02-16 23:02 UTC)