https://github.com/probably-nothing-labs/denormalized

Skip to content

Navigation Menu

Toggle navigation
 
Sign in

  * Product
      +  
        Actions
        Automate any workflow
      +  
        Packages
        Host and manage packages
      +  
        Security
        Find and fix vulnerabilities
      +  
        Codespaces
        Instant dev environments
      +  
        GitHub Copilot
        Write better code with AI
      +  
        Code review
        Manage code changes
      +  
        Issues
        Plan and track work
      +  
        Discussions
        Collaborate outside of code
    Explore
      + All features
      + Documentation
      + GitHub Skills
      + Blog
  * Solutions
    By size
      + Enterprise
      + Teams
      + Startups
    By industry
      + Healthcare
      + Financial services
      + Manufacturing
    By use case
      + CI/CD & Automation
      + DevOps
      + DevSecOps
  * Resources
    Topics
      + AI
      + DevOps
      + Security
      + Software Development
    Explore
      + Learning Pathways
      + White papers, Ebooks, Webinars
      + Customer Stories
      + Partners
  * Open Source
      +  
        GitHub Sponsors
        Fund open source developers
      +  
        The ReadME Project
        GitHub community articles
    Repositories
      + Topics
      + Trending
      + Collections
  * Enterprise
      +  
        Enterprise platform
        AI-powered developer platform
    Available add-ons
      +  
        Advanced Security
        Enterprise-grade security features
      +  
        GitHub Copilot
        Enterprise-grade AI features
      +  
        Premium Support
        Enterprise-grade 24/7 support
  * Pricing

Search or jump to...

Search code, repositories, users, issues, pull requests...

Search
[                    ]
Clear

Search syntax tips

Provide feedback

We read every piece of feedback, and take your input very seriously.

[                    ] [ ] Include my email address so I can be
contacted
Cancel Submit feedback

Saved searches

Use saved searches to filter your results more quickly

Name [                    ] 
Query [                    ]

To see all available qualifiers, see our documentation.

Cancel Create saved search
Sign in
Sign up Reseting focus
You signed in with another tab or window. Reload to refresh your
session. You signed out in another tab or window. Reload to refresh
your session. You switched accounts on another tab or window. Reload
to refresh your session. Dismiss alert
{{ message }}
probably-nothing-labs / denormalized Public

  * Notifications You must be signed in to change notification
    settings
  * Fork 2
  * Star 98

Embeddable stream processing engine based on Apache DataFusion

www.denormalized.io

License

Apache-2.0 license
98 stars 2 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

  * Code
  * Issues 0
  * Pull requests 1
  * Actions
  * Security
  * Insights

Additional navigation options

  * Code
  * Issues
  * Pull requests
  * Actions
  * Security
  * Insights

probably-nothing-labs/denormalized

This commit does not belong to any branch on this repository, and may
belong to a fork outside of the repository.
 main
BranchesTags
  
Go to file
Code

Folders and files

      Name              Name          Last commit       Last commit
                                        message            date
Latest commit

 

History

114 Commits
 
.github/workflows .github/                             
                  workflows
crates/core       crates/core                          
docs              docs                                 
examples          examples                             
.gitignore        .gitignore                           
Cargo.lock        Cargo.lock                           
Cargo.toml        Cargo.toml                           
LICENSE           LICENSE                              
README.md         README.md                            
clippy.toml       clippy.toml                          
View all files

Repository files navigation

  * README
  * Apache-2.0 license

Denormalized Logo

 

Denormalized is a fast embeddable stream processing engine built on
Apache DataFusion. It currently supports kafka as a real-time source
and a sink, windowed aggregations, and stream joins.

This repo is still a work-in-progress and we are actively seeking
design partners. If you have have a specific use-case you'd like to
discuss please drop us a line via a github issue or email
hello@denormalized.io.

Quickstart

 

Prerequisites

 

  * Docker
  * Rust/Cargo installed

Running an example

 

 1. Start kafka in docker docker run -p 9092:9092 --name kafka apache
    /kafka
 2. Start emitting some sample data: cargo run --example
    emit_measurements
 3. Run a simple streaming aggregation on the data using
    denormalized: cargo run --example emit_measurements

More examples

 

A more powerful example can be seen in our kafka ridesharing example

Roadmap

 

  * [*] Stream aggregation
  * [*] Stream joins
  * [ ] Checkpointing / restoration
  * [ ] Session windows
  * [ ] Stateful UDF API
  * [ ] DuckDB support
  * [ ] Reading/writing from Postgres
  * [ ] Python bindings
  * [ ] Typescript bindings
  * [ ] UI

Credits

 

Denormalized is built and maintained by Denormalized in San
Francisco. Please drop in a line to hello@denormalized.io or simply
open up a GitHub Issue!

About

Embeddable stream processing engine based on Apache DataFusion

www.denormalized.io

Resources

Readme

License

Apache-2.0 license
Activity
Custom properties

Stars

98 stars

Watchers

1 watching

Forks

2 forks
Report repository

Releases

No releases published

Packages 0

No packages published

Contributors 2

  * @emgeee emgeee Matt Green
  * @ameyc ameyc Amey Chaugule

Languages

  * Rust 100.0%

Footer

 (c) 2024 GitHub, Inc.

Footer navigation

  * Terms
  * Privacy
  * Security
  * Status
  * Docs
  * Contact
  * Manage cookies
  * Do not share my personal information

You can't perform that action at this time.