https://github.com/facebookexternal/nimble Skip to content Navigation Menu Toggle navigation Sign in * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} facebookexternal / nimble Public * Notifications * Fork 0 * Star 40 * New file format for storage of large columnar datasets. License Apache-2.0 license 40 stars 0 forks Branches Tags Activity Star Notifications * Code * Issues 0 * Pull requests 0 * Actions * Projects 0 * Security * Insights Additional navigation options * Code * Issues * Pull requests * Actions * Projects * Security * Insights facebookexternal/nimble This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main BranchesTags Go to file Code Folders and files Name Name Last commit Last commit message date Latest commit History 1 Commits .github/workflows .github/workflows CMake CMake dwio/nimble dwio/nimble scripts scripts velox @ 2da7641 velox @ 2da7641 .clang-format .clang-format .gitmodules .gitmodules CMakeLists.txt CMakeLists.txt CODE_OF_CONDUCT.md CODE_OF_CONDUCT.md LICENSE LICENSE Makefile Makefile README.md README.md license.header license.header View all files Repository files navigation * README * Code of conduct * Apache-2.0 license The Nimble File Format Nimble (formerly known as "Alpha") is a new columnar file format for large datasets created by Meta. Nimble is meant to be a replacement for file formats such as Apache Parquet and ORC. Watch this talk to learn more about Nimble's internals. Nimble has the following design principles: * Wide: Nimble is better suited for workloads that are wide in nature, such as tables with thousands of columns (or streams) which are commonly found in feature engineering workloads and training tables for machine learning. * Extensible: Since the state-of-the-art in data encoding evolves faster than the file layout itself, Nimble decouples stream encoding from the underlying physical layout. Nimble allows encodings to be extended by library users and recursively applied (cascading). * Parallel: Nimble is meant to fully leverage highly parallel hardware by providing encodings which are SIMD and GPU friendly. Although this is not implemented yet, we intend to expose metadata to allow developers to better plan decoding trees and schedule kernels without requiring the data streams themselves. * Unified: More than a specification, Nimble is a product. We strongly discourage developers to (re-)implement Nimble's spec to prevent environmental fragmentation issues observed with similar projects in the past. We encourage developers to leverage the single unified Nimble library, and create high-quality bindings to other languages as needed. Nimble has the following features: * Lighter metadata organization to efficiently support thousands to tens of thousands of columns and streams. * Use Flatbuffers instead of thrift/protobuf to more efficiently access large metadata sections. * Use block encoding instead of stream encoding to provide predictable memory usage while decoding/reading. * Supports many encodings out-of-the-box, and additional encodings can be added as needed. * Supports cascading (recursive/composite) encoding of streams. * Supports pluggable encoding selection policies. * Provide extensibility APIs where encodings and other aspects of the file can be extended. * Clear separation between logical and physical encoded types. * And more. Nimble is a work in progress, and many of these features above are still under design and/or active development. As such, Nimble does not provide stability or versioning guarantees (yet). They will be eventually provided with a future stable release. Use it at your own risk. Build Nimble's CMake build system is self-sufficient and able to either locate its main dependencies or compile them locally. In order to compile it, one can simply: $ git clone git@github.com:facebookexternal/nimble.git $ cd nimble $ make To override the default behavior and force the build system to, for example, build a dependency locally (bundle it), one can: $ folly_SOURCE=BUNDLED make Nimble builds have been tested using clang 15 and 16. It should automatically compile the following dependencies: gtest, glog, folly, abseil, and velox. You may need to first install the following system dependencies for these to compile (example from Ubuntu 22.04): $ sudo apt install -y \ flatbuffers-compiler \ libflatbuffers-dev \ libgflags-dev \ libunwind-dev \ libgoogle-glog-dev \ libdouble-conversion-dev \ libevent-dev \ liblzo2-dev \ libelf-dev \ libdwarf-dev \ libsnappy-dev \ bison \ flex \ libfl-dev Although Nimble's codebase is today closely coupled with velox, we intend to decouple them in the future. License Nimble is licensed under the Apache 2.0 License. A copy of the license can be found here. About New file format for storage of large columnar datasets. Resources Readme License Apache-2.0 license Code of conduct Code of conduct Activity Custom properties Stars 40 stars Watchers 13 watching Forks 0 forks Report repository Releases No releases published Packages 0 No packages published Contributors 5 * @pedroerp * @helfman * @Yuhta * @facebook-github-bot * @xiaoxmeng Languages * C++ 94.3% * Python 2.9% * CMake 2.6% * Other 0.2% Footer (c) 2024 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact * Manage cookies * Do not share my personal information You can't perform that action at this time.