[HN Gopher] Show HN: Lance - Alternative to Parquet for ML data
       ___________________________________________________________________
        
       Show HN: Lance - Alternative to Parquet for ML data
        
       Author : chop
       Score  : 28 points
       Date   : 2023-05-31 21:16 UTC (1 hours ago)
        
 (HTM) web link (github.com)
 (TXT) w3m dump (github.com)
        
       | mulmen wrote:
       | I'm confused. Parquet is a file format. The reference
       | implementation is in Java but rust implementations exist. Is this
       | faster because rust or because of the file format? Could this
       | format offer benefits in a Java environment?
        
         | eddyxu wrote:
         | Hey, co-author of Lance here. Lance is faster in random access
         | because the layout / encodings were designed to be fast in both
         | scan and random access case. We borrowed many ideas from
         | Google's Procella paper, and Arrow's in-memory layout. Also we
         | added a bunch of I/O exec plan optimizations with the
         | assumption that it has large-blob columns (i.e., image, lidar
         | point cloud) during scanning, which do not exist in traditional
         | OLAP systems, because their workloads are different than ML
         | training.
         | 
         | Re-implementing Lance in Java should have very similar I/O
         | characteristics. There are actually some efforts to support
         | Lance in JVM / Spark data sources.
        
           | mulmen wrote:
           | Thanks for the clarification! That's very exciting. A JVM
           | implementation that can drop in to Spark/JVM would be great
           | because there's so much inertia built up around the Apache
           | ecosystem.
        
           | alphaursaemin wrote:
           | Reminded me of deeplake. What is the comparative analysis?
        
             | eddyxu wrote:
             | We have not done benchmarks against deeplake yet. Deeplake
             | has some interesting concepts in their design, I'd be very
             | interested to do a benchmark soon.
        
       | gjreda wrote:
       | I recently prototyped out a "chat over PDF documents" project.[1]
       | I opted to use LanceDB for vector (embeddings) storage and
       | retrieval and found it really nice to use.
       | 
       | I'm working on using it in a large project now.
       | 
       | [1] - https://github.com/gjreda/scratch-pdf-bot
        
       | nemo44x wrote:
       | It looks nice but why is the implementation language mentioned?
       | Why does it matter? If anything it's confusing as some one might
       | assume it's only for Rust programs.
        
         | dang wrote:
         | Ok, we've taken Rust out of the title now.
         | 
         | Let's talk about the specifics of the project now please.
        
       | alphaursaemin wrote:
       | getting bunch of 404s on the docs. for example https://eto-
       | ai.github.io/lance/format.html (But this works:
       | https://lancedb.github.io/lance/*)
       | 
       | Did you guys just pivot from eto-ai to lancedb?
        
         | chop wrote:
         | Thanks for the call out, its been updated now! We haven't
         | pivoted, just updated the Github organization recently.
        
       ___________________________________________________________________
       (page generated 2023-05-31 23:01 UTC)