[HN Gopher] Amazon SQS is 15 Years Old
___________________________________________________________________
Amazon SQS is 15 Years Old
Author : jeffbarr
Score : 60 points
Date : 2021-07-13 13:03 UTC (9 hours ago)
(HTM) web link (aws.amazon.com)
(TXT) w3m dump (aws.amazon.com)
| yummybear wrote:
| As a side note - For someone not totally an expert in AWS it's
| sometimes difficult to choose between SQS, SNS, EventBridge and
| the list of other related technologies.
| commandlinefan wrote:
| We had to do some queuing for replication and priced SQS vs.
| Kinesis and Kinesis was orders of magnitude cheaper and did
| pretty much the same thing.
| ransom1538 wrote:
| SQS - pull SNS - push
|
| Sure you can mix and match and get all confused. But I would
| stick to that.
| orf wrote:
| Not really, more like SQS - 1 to 1, SNS - 1 to many
| [deleted]
| [deleted]
| stephenhuey wrote:
| Even though I first created images for EC2 in 2008, I've still
| never used SQS. I have a lot of experience with Google PubSub and
| this blog post is making me wonder if I should try leveraging SQS
| in production. I'm sure it's very mature, but I'm curious about
| how it feels from a developer perspective with a heavy production
| load.
| yzmtf2008 wrote:
| IMO, SQS is one of the best products that AWS offers. Airbnb
| built a queueing + scheduling system on SQS to great success
| [0] (disclaimer, I wrote this post). There are many others,
| e.g., Slack, that are building on top of SQS.
|
| [0]: https://medium.com/airbnb-engineering/dynein-building-a-
| dist...
| Supermancho wrote:
| > SQS is one of the best products that AWS offers.
|
| I would never trust anyone who says this, regardless of who
| did what with it.
|
| SQS Standard (which was the only for a long time) has
| historically been a nightmare for almost any use case
| including job queueing. I love creating a whole separate
| cache to ensure when SQS delivers message multiple times,
| that I don't act on them. That's just ONE issue that I don't
| need to have. When you get into FIFO to avoid that, now your
| costing is ridiculous at scale AND the (relatively small)
| number of issues with FIFO that are still outstanding .
|
| eg https://tomgregory.com/3-surprising-facts-about-aws-sqs-
| fifo...
| heartofgold wrote:
| SQS FIFO is like 25-40% more expensive, but you're getting
| added benefit and additional guarantees in exchange.
|
| For the most expensive tier: 40 cents per million API
| requests for plain SQS vs 50 cents per million API requests
| for FIFO SQS.
|
| Like many AWS Services, it started off pretty basic or
| limited, but gradually and incrementally improved over
| time.
|
| I started off pretty skeptical, but over time I've grown to
| be impressed with SQS and SQS FIFO.
| emidln wrote:
| It's really not hard to pick a natural key to use for
| idempotency in your persistence layer. If you don't want at
| least once delivery, why are you picking SQS instead of
| some random ephemeral queue like NATS?
|
| I second the parent post. I've billions of messages through
| SQS at a previous job and I can remember having issues with
| SQS availability exactly once (due to a systemic failure
| involving DyanmoDB where cascading failures took down
| almost everything (and all of our integrations) that would
| have hosed us completely if not for our durable-
| queue/s3-journal[0] at the edge). SQS is simple to use,
| does what it says, and has very good SDK support. A++++
| Would build a business on top of again!
|
| [0] https://github.com/Factual/s3-journal
| arduinomancer wrote:
| I don't really understand this point.
|
| I've found its trivial to just have a unique ID in the
| message and just check if the ID already exists in a jobs
| table to avoid double processing.
|
| I pretty much always have some metadata table needed for
| the job anyway, so its not like I'm building something
| extra.
| gunapologist99 wrote:
| jobs table? why not just use the database directly then?
| chris_j wrote:
| Use the database directly? I hope you're not suggesting
| sharing a database between services...
| gunapologist99 wrote:
| Sharing an SQS queue between services is ok then? Even
| the most basic example for SQS queues from the AWS SA
| training course recommends using multiple queues for
| different classes of video transcoding. All of those
| benefits evaporate once you share a single queue; if
| you're just reading that queue to create a job in a
| database, then the queue provides no benefit.
| time0ut wrote:
| This is a surprising take. Accounting for duplicate events
| is such a common pattern in distributed systems that it
| never occurred to me to think of it as an issue. What is
| your preferred approach?
| yzmtf2008 wrote:
| I'm no longer with Airbnb / the project, but we did in fact
| use SQS Standard exclusively for job queueing (I don't
| think we ever did explicitly add any support for FIFO). Our
| SLA was at-least-once (as most job queues are), so this
| didn't matter as much. In practice it also doesn't happen
| that often.
| topspin wrote:
| The one legitimate "problem" in that list that is #3, the
| hard limit on FIFO messages (20,000). The rest are expected
| behavior for a 'deliver exactly once and in order' queue.
| The use cases offered (client fails to process and
| therefore consume the next message, blocking subsequent
| messages) and somehow (?) expecting the queue to facilitate
| reprocessing previous consumed messages are both misuses of
| such a queue.
|
| SQS (standard and FIFO) has another problem that precludes
| using it in one case I have; the 256KB message size limit.
| Amazon has a Java-only workaround (Amazon SQS Extended
| Client Library for Java) that will spool large messages
| to/from S3 -- so clearly my case isn't unreasonable -- but
| there is nothing for the general case.
| [deleted]
| ElViajero wrote:
| Simple Queue Service (SQS)
|
| It's just polite to write the meaning of acronyms.
| tforcram wrote:
| I interned on the SQS team back in 2006 and subsequently joined
| the team full time for a few years including briefly managing it.
| I have some great memories of that time. Keeping things 'simple'
| while scaling as AWS grew was a great introduction to the
| industry.
| kchoudhu wrote:
| Most consistently excellent AWS product: it's a simple thing that
| does a simple job with minimal hassle. I suspect it will save me
| the trouble of ever having to run messaging infrastructure again.
___________________________________________________________________
(page generated 2021-07-13 23:02 UTC)