[HN Gopher] Datomic Implicit Partitions
       ___________________________________________________________________
        
       Datomic Implicit Partitions
        
       Author : xmlblog
       Score  : 14 points
       Date   : 2023-04-21 17:50 UTC (5 hours ago)
        
 (HTM) web link (blog.datomic.com)
 (TXT) w3m dump (blog.datomic.com)
        
       | hlship wrote:
       | Like any database, Datomic often needs to perform queries by
       | reading data from its indexes. In fact, in Datomic, all data is
       | stored in indexes - there aren't rows with indexes pointing to
       | the rows, just indexes.
       | 
       | Datomic uses any of a number of stores, typically something like
       | DynamoDB, to persist this index data.
       | 
       | When performing a query, Datomic peers read directly from the
       | store (or from a cache) and read an entire block of index data as
       | a single unit - each such block effectively contains many
       | entity/attribute/value/transaction Datoms sorted by entity,
       | attribute, or value, depending on which index is being scanned.
       | 
       | In other words, when reading all the attributes for entity X,
       | Datomic will read from storage the block containing X, but also
       | must inadvertently read other data that precedes or follows X.
       | 
       | The partitioning business is to ensure that related entities are
       | stored close to each other; literally, a block of bits within the
       | 64 bit entity id is set to the partition value. This ensures that
       | the Datoms for related entities are stored close to each other,
       | which in turn, ensures that the query can be satisfied using
       | fewer of these index blocks, read from cache or storage, than if
       | the entire entity id was arbitrary. Essentially, if entity X and
       | entity Y are related, they can share a partition, and most likely
       | will be stored together in a single index block.
       | 
       | This co-location of data can make a big difference in very large
       | Datomic databases.
        
       ___________________________________________________________________
       (page generated 2023-04-21 23:01 UTC)