[HN Gopher] The Nimble File Format by Meta
       ___________________________________________________________________
        
       The Nimble File Format by Meta
        
       Author : zzulus
       Score  : 42 points
       Date   : 2024-04-25 21:52 UTC (1 hours ago)
        
 (HTM) web link (github.com)
 (TXT) w3m dump (github.com)
        
       | winwang wrote:
       | Although "wide" data is touted as a optimization guideline for
       | Nimble, how well does it fare against "normal"(?) data, i.e. with
       | just a few to tens of columns?
       | 
       | Also, are there any preliminary benchmarks?
        
         | quadrature wrote:
         | There are some numbers in the youtube presentation
         | https://www.youtube.com/watch?v=bISBNVtXZ6M
         | 
         | It seems to be optimized towards ML where sequential scan is
         | the access pattern. so it wouldn't be suitable for analytical
         | workloads yet, though they are planning on working on that.
        
       | jgarzik wrote:
       | Where is the file format specification?
       | 
       | I would prefer to write a parser with zero dependencies.
        
         | SonOfLilit wrote:
         | They strongly discourage you.
         | 
         | Unified: More than a specification, Nimble is a product. We
         | strongly discourage developers to (re-)implement Nimble's spec
         | to prevent environmental fragmentation issues observed with
         | similar projects in the past. We encourage developers to
         | leverage the single unified Nimble library, and create high-
         | quality bindings to other languages as needed.
        
           | jgarzik wrote:
           | Sigh. I get more than enough of that in the blockchain world.
           | 
           | Call me a greybeard; I want multiple implementations and a
           | spec.
        
       | 1-6 wrote:
       | I wonder how well it functions as a blobstore.
        
       | 1-6 wrote:
       | I thought the future would have been unstructured.
        
       | CharlesW wrote:
       | Video link and more comments from two weeks ago:
       | https://news.ycombinator.com/item?id=39995112
        
       | albertzeyer wrote:
       | https://xkcd.com/927/ ?
       | 
       | We still use HDF
       | (https://en.wikipedia.org/wiki/Hierarchical_Data_Format).
       | 
       | But I wonder, if I would choose a new file format today, what to
       | choose? Nimble is maybe too new and there is too less experience
       | with it (outside Meta).
       | 
       | Is there anywhere a good overview of all available options, and
       | some fair comparison? Some that I found, but older:
       | 
       | https://www.hopsworks.ai/post/guide-to-file-formats-for-mach...
        
       | zX41ZdbW wrote:
       | Interesting if there are benefits over MergeTree (ClickHouse's
       | data format).
        
       | levzettelin wrote:
       | What are the differences versus Arrow/Parquet?
        
         | nmstoker wrote:
         | Yes, I'd be curious to know how much better it is than them -
         | from my limited understanding, they also share many of the
         | advantages that Nimble boasts of, thus I can appreciate they'd
         | both be better than legacy formats but it's not clear how close
         | these two are.
        
       ___________________________________________________________________
       (page generated 2024-04-25 23:01 UTC)