[HN Gopher] Husky, Datadog's Third-Generation Event Store
       ___________________________________________________________________
        
       Husky, Datadog's Third-Generation Event Store
        
       Author : louis-paul
       Score  : 5 points
       Date   : 2022-05-17 22:04 UTC (57 minutes ago)
        
 (HTM) web link (www.datadoghq.com)
 (TXT) w3m dump (www.datadoghq.com)
        
       | rektide wrote:
       | Lovely read. Condensing some, there's three node types in the
       | system, writers, compactors, and readers.
       | 
       | > _Writers read from Kafka, (briefly) buffer events in memory,
       | upload events to blob storage in our custom file format, and then
       | commit the presence of these new files to our metadata store....
       | Compactors scan the metadata store for small files generated by
       | the Writers and previous compactions, and compact them into
       | larger files.... The Reader (leaf) nodes run queries over
       | individual files in blob storage and return partial aggregates,
       | which are re-aggregated by the distributed query engine._
       | 
       | And then the meta-data supporting the system:
       | 
       | > _Husky 's metadata store has multiple responsibilities, but its
       | most important one is to serve as the strongly consistent source
       | of truth for the set of files currently visible to each customer.
       | We'll delve into the details of our metadata store more in future
       | blog posts, but it is a thin abstraction around FoundationDB,
       | which we selected because it was one of the few open source OLTP
       | database systems that met our requirements_
       | 
       | There's some nice scalability/isolation benefits in this all.
       | Having reader nodes reading from network storage has created a
       | lot of flexibility & ability to shift work around on demand.
        
       ___________________________________________________________________
       (page generated 2022-05-17 23:02 UTC)