https://github.com/feldera/feldera Skip to content Navigation Menu Toggle navigation Sign in * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + GitHub Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions By size + Enterprise + Teams + Startups By industry + Healthcare + Financial services + Manufacturing By use case + CI/CD & Automation + DevOps + DevSecOps * Resources Topics + AI + DevOps + Security + Software Development + View all Explore + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Enterprise + Enterprise platform AI-powered developer platform Available add-ons + Advanced Security Enterprise-grade security features + GitHub Copilot Enterprise-grade AI features + Premium Support Enterprise-grade 24/7 support * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up Reseting focus You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} feldera / feldera Public * Notifications You must be signed in to change notification settings * Fork 39 * Star 539 The Feldera Incremental Computation Engine feldera.com License View license 539 stars 39 forks Branches Tags Activity Star Notifications You must be signed in to change notification settings * Code * Issues 241 * Pull requests 10 * Discussions * Actions * Projects 0 * Wiki * Security * Insights Additional navigation options * Code * Issues * Pull requests * Discussions * Actions * Projects * Wiki * Security * Insights feldera/feldera This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main BranchesTags Go to file Code Folders and files Last commit Last Name Name message commit date Latest commit History 3,373 Commits .devcontainer .devcontainer .github .github .vscode .vscode benchmark benchmark crates crates demo demo deploy deploy docs docs python python scripts scripts sql-to-dbsp-compiler sql-to-dbsp-compiler web-console web-console .arg .arg .dockerignore .dockerignore .earthlyignore .earthlyignore .gitattributes .gitattributes .gitignore .gitignore .gitmodules .gitmodules .pre-commit-config.yaml .pre-commit-config.yaml CODE_OF_CONDUCT.md CODE_OF_CONDUCT.md CONTRIBUTING.md CONTRIBUTING.md Cargo.lock Cargo.lock Cargo.toml Cargo.toml Earthfile Earthfile LICENSE LICENSE README.md README.md architecture.svg architecture.svg bun.lockb bun.lockb openapi.json openapi.json View all files Repository files navigation * README * Code of conduct * License Feldera [6874747073] [badge] Nightly [badge] [6874747073] Discord Sandbox Feldera is a fast query engine for incremental computation. Feldera has the unique ability to evaluate arbitrary SQL programs incrementally, making it more powerful, expressive and performant than existing alternatives like batch engines, warehouses, stream processors or streaming databases. --------------------------------------------------------------------- Incremental Computation Engine Our approach to incremental computation is simple. A Feldera pipeline is a set of SQL tables and views. Views can be deeply nested. Users start, stop or pause pipelines to manage and advance a computation. Pipelines continuously process changes, which are any number of inserts, updates or deletes to a set of tables. When the pipeline receives changes, Feldera incrementally updates all the views by only looking at the changes and it completely avoids recomputing over older data. While a pipeline is running, users can inspect the results of the views at any time. Our approach to incremental computation makes Feldera incredibly fast (millions of events per second on a laptop). It also enables unified offline and online compute over both live and historical data. Feldera users have built batch and real-time feature engineering pipelines, ETL pipelines, various forms of incremental and periodic analytical jobs over batch data, and more. Our defining Features 1. Full SQL support and more. Our engine is the only one in existence that can evaluate full SQL syntax and semantics completely incrementally. This includes joins and aggregates, group by, correlated subqueries, window functions, complex data types, time series operators, UDFs, and recursive queries. Pipelines can process deeply nested hierarchies of views. 2. Fast out-of-the-box performance. Feldera users have reported getting complex use cases implemented in 30 minutes or less, and hitting millions of events per second in performance on a laptop without any tuning. 3. Datasets larger than RAM. Feldera is designed to handle datasets that exceed the available RAM by spilling efficiently to disk, taking advantage of recent advances in NVMe storage. 4. Strong guarantees on consistency and freshness. Feldera is strongly consistent. It also guarantees that the state of the views always corresponds to what you'd get if you ran the queries in a batch system for the same input. 5. Connectors for your favorite data sources and destinations. Feldera connects to myriad batch and streaming data sources, like Kafka, HTTP, CDC streams, S3, Data Lakes, Warehouses and more. If you need a connector that we don't yet support, let us know. Architecture The following diagram shows Feldera's architecture Feldera Platform Architecture [?][?] Quick start with Docker First, make sure you have Docker Compose installed. Next, run the following command to download a Docker Compose file, and use it to bring up a Feldera Platform deployment suitable for demos, development and testing: curl -L https://github.com/feldera/feldera/releases/latest/download/docker-compose.yml | \ docker compose -f - --profile demo up It can take some time for the container images to be downloaded. About ten seconds after that, the Feldera web console will become available. Visit http://localhost:8080 on your browser to bring it up. We suggest going through our tutorial next. Our Getting Started guide has more detailed instructions on running the demo. [?][?] Running Feldera from sources To run Feldera from sources, first install all the required dependencies. This includes the Rust toolchain (at least 1.75), Java (at least JDK 19), Maven and Typescript. After that, the first step is to build the SQL compiler: cd sql-to-dbsp-compiler ./build.sh Next, from the repository root, run the pipeline-manager: cargo run --bin=pipeline-manager --features pg-embed As with the Docker instructions above, you can now visit http:// localhost:8080 on your browser to see the Feldera WebConsole. Documentation To learn more about Feldera Platform, we recommend going through the documentation. * Getting started * UI tour * Tutorials * Demo * SQL reference * API reference Benchmarks Feldera is generally faster and uses less memory than systems like stream processors. Our Benchmarks are performed by our CI on every commit that goes in main. If you want to see all the results, please visit benchmarks.feldera.io. [68747470733a2f2f7777772e66656c646572612e636f6d2f5f6e6578742f696d616] Contributing The software in this repository is governed by an open-source license. We welcome contributions. Here are some guidelines. Theory Feldera Platform achieves its objectives by building on a solid mathematical foundation. The formal model that underpins our system, called DBSP, is described in the accompanying paper: * Budiu, Chajed, McSherry, Ryzhyk, Tannen. DBSP: Automatic Incremental View Maintenance for Rich Query Languages, Conference on Very Large Databases, August 2023, Vancouver, Canada * Here is a presentation about DBSP at the 2023 Apache Calcite Meetup. The model provides two things: 1. Semantics. DBSP defines a formal language of streaming operators and queries built out of these operators, and precisely specifies how these queries must transform input streams to output streams. 2. Algorithm. DBSP also gives an algorithm that takes an arbitrary query and generates an incremental dataflow program that implements this query correctly (in accordance with its formal semantics) and efficiently. Efficiency here means, in a nutshell, that the cost of processing a set of input events is proportional to the size of the input rather than the entire state of the database. About The Feldera Incremental Computation Engine feldera.com Topics rust streaming sql database kafka analytics realtime data-analysis continuous data-pipeline materialized-view incremental-computation Resources Readme License View license Code of conduct Code of conduct Activity Custom properties Stars 539 stars Watchers 12 watching Forks 39 forks Report repository Releases 36 v0.26.0 Latest Sep 17, 2024 + 35 releases Packages 0 Contributors 21 * @gz * @blp * @lalithsuresh * @ryzhyk * @mihaibudiu * @Kixiron * @Karakatiza666 * @abhizer * @snkas * @absoludity * @aehmttw * @dependabot[bot] * @rivudhk * @bin-wang + 7 contributors Languages * Rust 57.7% * Java 34.7% * TypeScript 3.7% * Python 1.9% * Svelte 1.0% * Shell 0.5% * Other 0.5% Footer (c) 2024 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact * Manage cookies * Do not share my personal information You can't perform that action at this time.