https://github.com/spiraldb/vortex Skip to content Navigation Menu Toggle navigation Sign in * Product + GitHub Copilot Write better code with AI + Security Find and fix vulnerabilities + Actions Automate any workflow + Codespaces Instant dev environments + Issues Plan and track work + Code Review Manage code changes + Discussions Collaborate outside of code + Code Search Find more, search less Explore + All features + Documentation + GitHub Skills + Blog * Solutions By size + Enterprise + Teams + Startups By industry + Healthcare + Financial services + Manufacturing By use case + CI/CD & Automation + DevOps + DevSecOps * Resources Topics + AI + DevOps + Security + Software Development + View all Explore + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Enterprise + Enterprise platform AI-powered developer platform Available add-ons + Advanced Security Enterprise-grade security features + GitHub Copilot Enterprise-grade AI features + Premium Support Enterprise-grade 24/7 support * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up Reseting focus You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} spiraldb / vortex Public * Notifications You must be signed in to change notification settings * Fork 13 * Star 372 A toolkit for working with compressed Arrow in-memory, on-disk, and over-the-wire. "The LLVM of file formats" License Apache-2.0 license 372 stars 13 forks Branches Tags Activity Star Notifications You must be signed in to change notification settings * Code * Issues 61 * Pull requests 13 * Discussions * Actions * Security * Insights Additional navigation options * Code * Issues * Pull requests * Discussions * Actions * Security * Insights spiraldb/vortex This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. develop BranchesTags Go to file Code Folders and files Last Last Name Name commit commit message date Latest commit History 803 Commits .cargo .cargo .github .github bench-vortex bench-vortex docs docs encodings encodings fuzz fuzz pyvortex pyvortex vortex-array vortex-array vortex-buffer vortex-buffer vortex-datafusion vortex-datafusion vortex-datetime-dtype vortex-datetime-dtype vortex-dtype vortex-dtype vortex-error vortex-error vortex-expr vortex-expr vortex-flatbuffers vortex-flatbuffers vortex-proto vortex-proto vortex-sampling-compressor vortex-sampling-compressor vortex-scalar vortex-scalar vortex-schema vortex-schema vortex-serde vortex-serde xtask xtask .gitignore .gitignore CHANGELOG.md CHANGELOG.md CONTRIBUTING.md CONTRIBUTING.md COPYRIGHT COPYRIGHT Cargo.lock Cargo.lock Cargo.toml Cargo.toml LICENSE LICENSE README.md README.md clippy.toml clippy.toml deny.toml deny.toml pyproject.toml pyproject.toml release-plz.toml release-plz.toml renovate.json renovate.json requirements-dev.lock requirements-dev.lock requirements.lock requirements.lock rust-toolchain.toml rust-toolchain.toml rustfmt.toml rustfmt.toml View all files Repository files navigation * README * Apache-2.0 license Vortex Build Status Crates.io Documentation PyPI - Python Version Vortex is a toolkit for working with compressed Apache Arrow arrays in-memory, on-disk, and over-the-wire. Vortex is designed to be to columnar file formats what Apache DataFusion is to query engines (or, analogously, what LLVM + Clang are to compilers): a highly extensible & extremely fast framework for building a modern columnar file format, with a state-of-the-art, "batteries included" reference implementation. Vortex is an aspiring successor to Apache Parquet, with dramatically faster random access reads (100-200x faster) and scans (2-10x faster), while preserving approximately the same compression ratio and write throughput. It will also support very wide tables (at least 10s of thousands of columns) and (eventually) on-device decompression on GPUs. Caution This library is still under rapid development and is a work in progress! Some key features are not yet implemented, both the API and the serialized format are likely to change in breaking ways, and we cannot yet guarantee correctness in all cases. The major features of Vortex are: * Logical Types - a schema definition that makes no assertions about physical layout. * Zero-Copy to Arrow - "canonicalized" (i.e., fully decompressed) Vortex arrays can be zero-copy converted to/from Apache Arrow arrays. * Extensible Encodings - a pluggable set of physical layouts. In addition to the builtin set of Arrow-compatible encodings, the Vortex repository includes a number of state-of-the-art encodings (e.g., FastLanes, ALP, FSST, etc.) that are implemented as extensions. While arbitrary encodings can be implemented as extensions, we have intentionally chosen a small set of encodings that are highly data-parallel, which in turn allows for efficient vectorized decoding, random access reads, and (in the future) decompression on GPUs. * Cascading Compression - data can be recursively compressed with multiple nested encodings. * Pluggable Compression Strategies - the built-in Compressor is based on BtrBlocks, but other strategies can trivially be used instead. * Compute - basic compute kernels that can operate over encoded data (e.g., for filter pushdown). * Statistics - each array carries around lazily computed summary statistics, optionally populated at read-time. These are available to compute kernels as well as to the compressor. * Serialization - Zero-copy serialization of arrays, both for IPC and for file formats. * Columnar File Format (in progress) - A modern file format that uses the Vortex serde library to store compressed array data. Optimized for random access reads and extremely fast scans; an aspiring successor to Apache Parquet. Overview: Logical vs Physical One of the core design principles in Vortex is strict separation of logical and physical concerns. For example, a Vortex array is defined by a logical data type (i.e., the type of scalar elements) as well as a physical encoding (the type of the array itself). Vortex ships with several built-in encodings, as well as several extension encodings. The built-in encodings are primarily designed to model the Apache Arrow in-memory format, enabling us to construct Vortex arrays with zero-copy from Arrow arrays. There are also several built-in encodings (e.g., sparse and chunked) that are useful building blocks for other encodings. The included extension encodings are mostly designed to model compressed in-memory arrays, such as run-length or dictionary encoding. Analogously, vortex-serde is designed to handle the low-level physical details of reading and writing Vortex arrays. Choices about which encodings to use or how to logically chunk data are left up to the Compressor implementation. One of the unique attributes of the (in-progress) Vortex file format is that it encodes the physical layout of the data within the file's footer. This allows the file format to be effectively self-describing and to evolve without breaking changes to the file format specification. In fact, the format is designed to support forward compatibility by optionally embedding WASM decoders directly into the files themselves. This should help avoid the rapid calcification that has plagued other columnar file formats. Components Logical Types The Vortex type-system is still in flux. The current set of logical types is: * Null * Bool * Integer(8, 16, 32, 64) * Float(16, b16, 32, 64) * Binary * UTF8 * Struct * List (partially implemented) * Date/Time/DateTime/Duration (implemented as an extension type) * Decimal: TODO * FixedList: TODO * Tensor: TODO * Union: TODO Canonical/Flat Encodings Vortex includes a base set of "flat" encodings that are designed to be zero-copy with Apache Arrow. These are the canonical representations of each of the logical data types. The canonical encodings currently supported are: * Null * Bool * Primitive (Integer, Float) * Struct * VarBin (Binary, UTF8) * VarBinView (Binary, UTF8) * Extension * ...with more to come Compressed Encodings Vortex includes a set of highly data-parallel, vectorized encodings. These encodings each correspond to a compressed in-memory array implementation, allowing us to defer decompression. Currently, these are: * Adaptive Lossless Floating Point (ALP) * BitPacked (FastLanes) * Constant * Chunked * Delta (FastLanes) * Dictionary * Fast Static Symbol Table (FSST) * Frame-of-Reference * Run-end Encoding * RoaringUInt * RoaringBool * Sparse * ZigZag * ...with more to come Compression Vortex's default compression strategy is based on the BtrBlocks paper. Roughly, for each chunk of data, a sample of at least ~1% of the data is taken. Compression is then attempted ( recursively) with a set of lightweight encodings. The best-performing combination of encodings is then chosen to encode the entire chunk. This sounds like it would be very expensive, but given basic statistics about a chunk, it is possible to cheaply prune many encodings and ensure the search space does not explode in size. Compute Vortex provides the ability for each encoding to specialize the implementation of a compute function to avoid decompressing where possible. For example, filtering a dictionary-encoded UTF8 array can be more cheaply performed by filtering the dictionary first. Note--as mentioned above--that Vortex does not intend to become a full-fledged compute engine, but rather to implement basic compute operations as may be required for efficient scanning & pushdown. Statistics Vortex arrays carry lazily-computed summary statistics. Unlike other array libraries, these statistics can be populated from disk formats such as Parquet and preserved all the way into a compute engine. Statistics are available to compute kernels as well as to the compressor. The current statistics are: * BitWidthFreq * TrailingZeroFreq * IsConstant * IsSorted * IsStrictSorted * Max * Min * RunCount * TrueCount * NullCount Serialization / Deserialization (Serde) The goals of the vortex-serde implementation are: * Support scanning (column projection + row filter) with zero-copy and zero heap allocation. * Support random access in constant or near-constant time. * Forward statistical information (such as sortedness) to consumers. * Provide IPC format for sending arrays between processes. * Provide an extensible, best-in-class file format for storing columnar data on disk or in object storage. TODO: insert diagram here Integration with Apache Arrow Apache Arrow is the de facto standard for interoperating on columnar array data. Naturally, Vortex is designed to be maximally compatible with Apache Arrow. All Arrow arrays can be converted into Vortex arrays with zero-copy, and a Vortex array constructed from an Arrow array can be converted back to Arrow, again with zero-copy. It is important to note that Vortex and Arrow have different--albeit complementary--goals. Vortex explicitly separates logical types from physical encodings, distinguishing it from Arrow. This allows Vortex to model more complex arrays while still exposing a logical interface. For example, Vortex can model a UTF8 ChunkedArray where the first chunk is run-length encoded and the second chunk is dictionary encoded. In Arrow, RunLengthArray and DictionaryArray are separate incompatible types, and so cannot be combined in this way. Usage For best performance we recommend using MiMalloc as the application's allocator. #[global_allocator] static GLOBAL_ALLOC: MiMalloc = MiMalloc; Contributing Please see CONTRIBUTING.md. Setup In order to build vortex, you may also need to install the flatbuffer compiler (flatc): Mac brew install flatbuffers This repo uses rye to manage the combined Rust/Python monorepo build. First, make sure to run: # Install Rye from https://rye-up.com, and setup the virtualenv rye sync License Licensed under the Apache License, Version 2.0 (the "License"). Governance Vortex is and will remain an open-source project. Our intent is to model its governance structure after the Substrait project, which in turn is based on the model of the Apache Software Foundation. Expect more details on this in Q4 2024. Acknowledgments This project is inspired by and--in some cases--directly based upon the existing, excellent work of many researchers and OSS developers. In particular, the following academic papers greatly influenced the development: * Maximilian Kuschewski, David Sauerwein, Adnan Alhomssi, and Viktor Leis. 2023. BtrBlocks: Efficient Columnar Compression for Data Lakes. Proc. ACM Manag. Data 1, 2, Article 118 (June 2023), 14 pages. https://doi.org/10.1145/3589263 * Azim Afroozeh and Peter Boncz. The FastLanes Compression Layout: Decoding >100 Billion Integers per Second with Scalar Code. PVLDB, 16(9): 2132 - 2144, 2023. * Peter Boncz, Thomas Neumann, and Viktor Leis. FSST: Fast Random Access String Compression. PVLDB, 13(11): 2649-2661, 2020. * Azim Afroozeh, Leonardo X. Kuffo, and Peter Boncz. 2023. ALP: Adaptive Lossless floating-Point Compression. Proc. ACM Manag. Data 1, 4 (SIGMOD), Article 230 (December 2023), 26 pages. https: //doi.org/10.1145/3626717 Additionally, we benefited greatly from: * the existence, ideas, & implementation of Apache Arrow. * likewise for the excellent Apache DataFusion project. * the parquet2 project by Jorge Leitao. * the public discussions around choices of compression codecs, as well as the C++ implementations thereof, from duckdb. * the Velox and Nimble projects, and discussions with their maintainers. Thanks to all of the aforementioned for sharing their work and knowledge with the world! About A toolkit for working with compressed Arrow in-memory, on-disk, and over-the-wire. "The LLVM of file formats" Topics python rust compression arrow array Resources Readme License Apache-2.0 license Activity Custom properties Stars 372 stars Watchers 8 watching Forks 13 forks Report repository Releases 9 0.12.0 Latest Oct 3, 2024 + 8 releases Contributors 12 * @gatesn * @robert3005 * @renovate[bot] * @lwwmanning * @AdamGS * @danking * @a10y * @delta003 * @jdcasale * @github-actions[bot] * @joseph-isaacs * @doki23 Languages * Rust 99.4% * Other 0.6% Footer (c) 2024 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact * Manage cookies * Do not share my personal information You can't perform that action at this time.