https://github.com/ballista-compute/ballista Skip to content Sign up * Why GitHub? Features - + Code review + Project management + Integrations + Actions + Packages + Security + Team management + Hosting + Mobile + Customer stories - + Security - * Team * Enterprise * Explore + Explore GitHub - Learn & contribute + Topics + Collections + Trending + Learning Lab + Open source guides Connect with others + Events + Community forum + GitHub Education + GitHub Stars program * Marketplace * Pricing Plans - + Compare plans + Contact Sales + Nonprofit - + Education - [ ] [search-key] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this organization All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} ballista-compute / ballista * Watch 49 * Star 1.2k * Fork 76 Distributed compute platform implemented in Rust, using Apache Arrow memory model. ballistacompute.org Apache-2.0 License 1.2k stars 76 forks Star Watch * Code * Issues 54 * Pull requests 3 * Actions * Projects 1 * Wiki * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Wiki * Security * Insights main 4 branches 13 tags Go to file Code Clone HTTPS GitHub CLI [https://github.com/b] Use Git or checkout with SVN using the web URL. [gh repo clone ballis] Work fast with our official CLI. Learn more. * Open with GitHub Desktop * Download ZIP Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Go back Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Go back Launching Xcode If nothing happens, download Xcode and try again. Go back Launching Visual Studio If nothing happens, download the GitHub extension for Visual Studio and try again. Go back Latest commit @andygrove andygrove Implement more of BallistaContext (#384) ... 7a5f6ac Jan 18, 2021 Implement more of BallistaContext (#384) 7a5f6ac Git stats * 460 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time .github/workflows Added rust code formatting check to CI. Closes #364 (#367) Jan 17, 2021 benchmarks bump version to 0.4.0-SNAPSHOT Aug 11, 2020 dev Enable more code from 0.3.0 (#377) Jan 17, 2021 docker Port Flight service from old code (#352) Jan 12, 2021 docs add protobuf installation in rust env guide (#360) Jan 16, 2021 integration-tests Update again for latest DataFusion code (#326) Aug 20, 2020 jvm rewrite SqlTokenizer.kt (#324) Sep 8, 2020 kubernetes bump version to 0.4.0-SNAPSHOT Aug 11, 2020 proto Branch 0.4 rebased (#323) Aug 19, 2020 rust Implement more of BallistaContext (#384) Jan 18, 2021 spark Implement all TPC-H queries in Spark (#336) Dec 29, 2020 .dockerignore Getting closer to a working example Mar 20, 2020 .gitignore random release prep work (#221) Jul 16, 2020 CONTRIBUTING.md Add project goals to CONTRIBUTING (#345) Jan 12, 2021 LICENSE add license Jul 7, 2019 README.md TPC-H example compiles again (#383) Jan 18, 2021 View code README.md Ballista: Distributed Compute Platform License Crates.io Discord chat Overview Ballista is a proof-of-concept distributed compute platform primarily implemented in Rust, using Apache Arrow as the memory model. It is built on an architecture that allows other programming languages to be supported as first-class citizens without paying a penalty for serialization costs. Status With the release of Apache Arrow 3.0.0 there are many breaking changes in the Rust implementation and as a result it has been necessary to comment out much of the code in this repository and gradually get each Rust module working again with the 3.0.0 release. For the latest stable version of Ballista, see branch-0.3. Technologies The foundational technologies in Ballista are: * Apache Arrow memory model and compute kernels for efficient processing of data. * Apache Arrow Flight Protocol for efficient data transfer between processes. * Google Protocol Buffers for serializing query plans. * Docker for packaging up executors along with user-defined code. Ballista can be deployed in Kubernetes, or as a standalone cluster using etcd for discovery. Architecture The following diagram highlights some of the integrations that will be possible with this unique architecture. Note that not all components shown here are available yet. Ballista Architecture Diagram How does this compare to Apache Spark? Although Ballista is largely inspired by Apache Spark, there are some key differences. * The choice of Rust as the main execution language means that memory usage is deterministic and avoids the overhead of GC pauses. * Ballista is designed from the ground up to use columnar data, enabling a number of efficiencies such as vectorized processing (SIMD and GPU) and efficient compression. Although Spark does have some columnar support, it is still largely row-based today. * The combination of Rust and Arrow provides excellent memory efficiency and memory usage can be 5x - 10x lower than Apache Spark in some cases, which means that more processing can fit on a single node, reducing the overhead of distributed compute. * The use of Apache Arrow as the memory model and network protocol means that data can be exchanged between executors in any programming language with minimal serialization overhead. Examples The following examples should help illustrate the current capabilities of Ballista * TPC-H Benchmark Releases Ballista releases are now available on crates.io, Maven Central and Docker Hub. Please refer to the user guide for instructions on using a released version of Ballista. Documentation The user guide is hosted at https://ballistacompute.org, along with the blog where news and release notes are posted. Developer documentation can be found in the docs directory. Contributing See CONTRIBUTING.md for information on contributing to this project. About Distributed compute platform implemented in Rust, using Apache Arrow memory model. ballistacompute.org Topics kotlin java kubernetes rust scala spark jvm arrow distributed dataframe datafusion Resources Readme License Apache-2.0 License Releases 13 v0.3.0 Latest Aug 9, 2020 + 12 releases Packages 2 Contributors 21 * @andygrove * @houqp * @jorgecarleitao * @sd2k * @edrevo * @pjmore * @KenSuenobu * @Dyqer * @max-sixty * @laplab * @meijies + 10 contributors Languages * Rust 51.0% * Kotlin 30.5% * Java 10.1% * Scala 6.8% * Other 1.6% * (c) 2021 GitHub, Inc. * Terms * Privacy * Security * Status * Help * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.