[HN Gopher] Show HN: Vortex - a high-performance columnar file f...
___________________________________________________________________
Show HN: Vortex - a high-performance columnar file format
Author : gatesn
Score : 115 points
Date : 2024-10-14 17:34 UTC (5 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| gkapur wrote:
| Not an expert in the space at all and it does seem like people
| are exploring new file and table formats so that is really cool!
|
| How does this compare to Lance
| (https://lancedb.github.io/lance/)?
|
| What do you think the key applied use case for Vortex is?
| the_mitsuhiko wrote:
| > One of the unique attributes of the (in-progress) Vortex file
| format is that it encodes the physical layout of the data within
| the file's footer. This allows the file format to be effectively
| self-describing and to evolve without breaking changes to the
| file format specification.
|
| That is quite interesting. One challenge in general with parqet
| and arrow in the otel / observability ecosystem is that the shape
| of data is not quite known with spans. There are arbitrary
| attributes on them, and they can change. To the best of my
| knowledge no particularly great solution exists today for
| encoding this. I wonder to which degree this system could be
| "abused" for that.
| gigatexal wrote:
| As someone who works in data schema on read formats like
| parquet are amazing. I hate having to guess schemas with CSVs.
| physicsguy wrote:
| Pandera is quite nice for at least forcing validation in
| Pandas for this
| cle wrote:
| Isn't this what the Arrow IPC File format does too? Is there
| something unique about this?
| _willmanning wrote:
| Compression! Vortex can easily be 10x smaller than the
| equivalent Arrow representation (and decompresses very
| quickly into Arrow)
| robert3005 wrote:
| The thing we are trying to achieve is to be able to experiment
| and tune the way data is groupped on disk. Parquet has one way
| of laying data out, csv is another (though it's a text format
| so a bit moot), ORC is another, Lance has yet another different
| method. The file format itself stores how it's physically laid
| out on disk so you can tune and tweak physical layouts to match
| the specific storage needs of your system (this is the toolkit
| part where you can take vortex and use it to implement your own
| file format). Having said that we will have an implementation
| of file format that follows particular layout.
| jnordwick wrote:
| If it's in the footer, then I'm pending to the columns out of
| the question it seems without moving the footer.
| ericyd wrote:
| Thank God this file format is written in Rust, otherwise I'd be
| extremely skeptical.
| neeh0 wrote:
| It gave me a moment of pause why Rust is part of the equation,
| but I concluded I'm too dumb
| beAbU wrote:
| For a while "written in Rust" was sort of a "trust me, bro"
| label. The hivemind asserted that something written in rust
| must be automatically good and safe, because rust is good and
| safe.
|
| Thank god everyone wisened up. The tool maketh not the
| craftsman. These days the "written in rust" tag is met with
| knee jerk skepticism, as-if the hive mind over corrected.
| smartmic wrote:
| It's funny how ,,written in Rust" has become a running gag here
| on HN - but only if mentioned already in the title...
| keybored wrote:
| Is this a pun or something?
| ericyd wrote:
| I was being sarcastic, yes. Also the title used to include
| "written in Rust"
| Havoc wrote:
| Can one edit it in place?
|
| That's the main thing currently irritating me about parquet
| aduffy wrote:
| You're unlikely to find this with any analytic file format
| (including Vortex). The main reason is that OLAP systems
| generally assume an immutable distributed object/block layer
| (S3, HDFS, ABFS, etc.).
|
| It's then generally up to a higher-level component called a
| table format to handle the idea of edits. See for example how
| Apache Iceberg handles deletes
| https://iceberg.apache.org/spec/#row-level-deletes
| slotrans wrote:
| This is true, and in principle a good thing, but in the time
| since Parquet and ORC were created GDPR and CCPA are things
| that have come to exist. Any format we build in that space,
| today, needs to support in-place record-level deletion.
| aduffy wrote:
| Yea so the thing you do for this is called "compaction",
| where you effectively merge the original + edits/deletes
| into a new immutable file. You then change your table
| metadata pointer to point at the new compacted file, and
| delete the old files from S3.
|
| Due to the way S3 and the ilk are structured as globally
| replicated KV stores, you're not likely to get in-place
| edits anytime soon, and until the cost structure
| incentivizes otherwise you're going to continue to see data
| systems that preference immutable cloud storage.
| gazpacho wrote:
| Very cool! Any plans to offer more direct integrations with
| DataFusion, e.g. a `VertexReaderFactory`, hooks for pushdowns,
| etc?
| aduffy wrote:
| We have a TableProvider for use with Datafusion, checkout this
| crate and its examples:
| https://github.com/spiraldb/vortex/tree/develop/vortex-dataf...
| gazpacho wrote:
| Thanks!
| jagged-chisel wrote:
| "Vortex is a toolkit for working with compressed Apache Arrow
| arrays in-memory, on-disk, and over-the-wire."
|
| So it's a toolkit written in Rust. It is not a file format.
| _willmanning wrote:
| Perhaps that verbiage is just confusing. "On-disk" sort of
| implies "file format" but could be more explicit.
|
| That said, the immediate next line in the README perhaps
| clarifies a bit?
|
| "Vortex is designed to be to columnar file formats what Apache
| DataFusion is to query engines (or, analogously, what LLVM +
| Clang are to compilers): a highly extensible & extremely fast
| framework for building a modern columnar file format, with a
| state-of-the-art, "batteries included" reference
| implementation."
| jagged-chisel wrote:
| "Vortex is [...] a highly extensible & extremely fast
| framework for building a modern columnar file format."
|
| It's a framework for building file formats. This does not
| indicate that Vortex is, itself, a file format.
___________________________________________________________________
(page generated 2024-10-14 23:00 UTC)