[HN Gopher] How to analyse and aggregate data from DynamoDB (2020)
       ___________________________________________________________________
        
       How to analyse and aggregate data from DynamoDB (2020)
        
       Author : mooreds
       Score  : 13 points
       Date   : 2021-12-23 16:00 UTC (7 hours ago)
        
 (HTM) web link (bahr.dev)
 (TXT) w3m dump (bahr.dev)
        
       | melbourne_mat wrote:
       | The alternative - which I did 2 weeks ago at work - is to use
       | Hive through AWS EMR. Then it's just SQL all the way instead of
       | reinventing the wheel like this article recommends.
        
       | sakopov wrote:
       | DynamoDB streams are pretty awesome and my company uses it for
       | all kinds of neat aggregation scenarios. The one major thing to
       | consider with streams is data loss and how it can impact your use
       | case. Any non-intermittent issues in the lambda will cause stream
       | processing to stall out and I believe DynamoDB will begin
       | shedding events after 24 hours by default, so you will experience
       | data loss unless you take certain actions to prevent it. I would
       | highly recommend using IteratorAge metric in your lambda to track
       | times when your lambda is falling behind in processing of stream
       | events. From my personal experience, it is sometimes a better
       | option to write stream events to an SQS queue instead of
       | processing directly from the stream, as it offers better
       | durability and better dead-lettering/recovery support, and then
       | do aggregation tasks on the queue. Just my 0.02.
        
       ___________________________________________________________________
       (page generated 2021-12-23 23:02 UTC)