https://github.com/turbolytics/sql-flow Skip to content Navigation Menu Toggle navigation Sign in * Product + GitHub Copilot Write better code with AI + Security Find and fix vulnerabilities + Actions Automate any workflow + Codespaces Instant dev environments + Issues Plan and track work + Code Review Manage code changes + Discussions Collaborate outside of code + Code Search Find more, search less Explore + All features + Documentation + GitHub Skills + Blog * Solutions By company size + Enterprises + Small and medium teams + Startups By use case + DevSecOps + DevOps + CI/CD + View all use cases By industry + Healthcare + Financial services + Manufacturing + Government + View all industries View all solutions * Resources Topics + AI + DevOps + Security + Software Development + View all Explore + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners + Executive Insights * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Enterprise + Enterprise platform AI-powered developer platform Available add-ons + Advanced Security Enterprise-grade security features + GitHub Copilot Enterprise-grade AI features + Premium Support Enterprise-grade 24/7 support * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up Reseting focus You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} turbolytics / sql-flow Public * Notifications You must be signed in to change notification settings * Fork 7 * Star 113 DuckDB for streaming data License MIT license 113 stars 7 forks Branches Tags Activity Star Notifications You must be signed in to change notification settings * Code * Issues 9 * Pull requests 0 * Actions * Projects 0 * Wiki * Security * Insights Additional navigation options * Code * Issues * Pull requests * Actions * Projects * Wiki * Security * Insights turbolytics/sql-flow main BranchesTags [ ] Go to file Code Folders and files Name Name Last commit Last commit message date Latest commit History 35 Commits .github/workflows .github/workflows adrs adrs benchmark benchmark cmd cmd dev dev sqlflow sqlflow tests tests .dockerignore .dockerignore .gitignore .gitignore Dockerfile Dockerfile LICENSE LICENSE Makefile Makefile README.md README.md requirements.dev.txt requirements.dev.txt requirements.txt requirements.txt setup.py setup.py View all files Repository files navigation * README * MIT license SQLFlow: DuckDB for Streaming Data. SQLFlow enables SQL-based stream-processing, powered by DuckDB. SQLFlow embeds duckdb, supporting kafka stream processing logic using pure sql. SQLFlow executes SQL against streaming data, such as Kafka or webhooks. Think of SQLFlow as a way to run sql against a continuous stream of data. The data outputs can be shipped to sinks, such as Kafka. Screenshot 2024-12-31 at 7 22 55 AM SQLFlow Use-Cases * Streaming Data Transformations: Clean data and types and publish the new data (example config). * Stream Enrichment: Add data an input stream and publish the new data (example config). * Data aggregation: Aggregate input data batches to decrease data volume (example config). * Tumbling Window Aggregation: Bucket data into arbitrary time windows (such as "hour" or "10 minutes") (example config). * Running SQL against the Bluesky Firehose: Execute SQL against any webhook source, such as the Bluesky firehose (example config) SQLFlow Features * Streaming SQL: Execute SQL against a variety of input streams, including kafka and websockets (bluesky firehose). * Custom Serialization and Encoding: Support for various formats, such as JSON and Parquet. * High Throughput: Optimized to handle tens of thousands of messages per second using DuckDB, librdkafka, and confluent python * Tumbling Window Aggregations: Perform aggregations over fixed intervals and output data once the interval is complete, enabling rollups like hourly or 10-minute summaries. * Static Table Joins: Join streaming data with static datasets, such as CSVs, using SQLFlow. SQLFlow Roadmap * Sources + [*] Kafka Consumer using consumer groups + [*] Websocket input (for consuming bluesky firehose) + [ ] HTTP (for webhooks) * Sinks + [*] Kafka Producer + [*] Stdout + [ ] Postgres + [ ] Local Disk + [ ] S3 * Serialization + [*] JSON Input + [*] JSON Output + [ ] Parquet Output * Handlers + [*] Memory Persistence + [ ] Disk Persistence * [*] CSV Static Files for joinging static data during processing * [*] Tumbling Window Aggregations * [ ] Observability Metrics Getting Started Docker Docker is the easiest way to get started. * Pull the sql-flow docker image docker pull turbolytics/sql-flow:latest * Validate config by invoking it on test data docker run -v $(PWD)/dev:/tmp/conf -v /tmp/sqlflow:/tmp/sqlflow turbolytics/sql-flow:latest dev invoke /tmp/conf/config/examples/basic.agg.yml /tmp/conf/fixtures/simple.json ['{"city":"New York","city_count":28672}', '{"city":"Baltimore","city_count":28672}'] * Start kafka locally using docker docker-compose -f dev/kafka-single.yml up -d * Publish test messages to kafka python3 cmd/publish-test-data.py --num-messages=10000 --topic="topic-local-docker" * Start kafka consumer from inside docker-compose container docker exec -it kafka1 kafka-console-consumer --bootstrap-server=kafka1:9092 --topic=output-local-docker * Start SQLFlow in docker docker run -v $(PWD)/dev:/tmp/conf -v /tmp/sqlflow:/tmp/sqlflow turbolytics/sql-flow:latest run /tmp/conf/config/local.docker.yml * Verify output in the kafka consumer ... ... {"city":"San Francisco504","city_count":1} {"city":"San Francisco735","city_count":1} {"city":"San Francisco533","city_count":1} {"city":"San Francisco556","city_count":1} The dev invoke command enables testing a SQLFlow pipeline configuration on a batch of test data. This enables fast feedback local development before launching a SQLFlow consumer that reads from kafka. Configuration The heart of SQLFlow is the pipeline configuration file. Each configuration file specifies: * Kafka configuration * Pipeline configuration + Source: Input configuration + Handler: SQL transformation + Sink: Output configuration Screenshot 2024-12-30 at 9 08 57 AM Every instance of SQLFlow needs a pipeline configuration file. Consuming Bluesky Firehose SQLFlow supports DuckDB over websocket. Running SQL against the Bluesky firehose is a simple configuration file: bluesky firehose config Invoke sql-flow using the configuration listed above: output Checkout the configuration files here Recipes Coming Soon, until then checkout: * Benchmark configurations * Unit Test configurations Running multiple SQLFlow instances on the same filesystem Verifying a configuration locally Development * Install python deps pip install -r requirements.txt pip install -r requirements.dev.txt C_INCLUDE_PATH=/opt/homebrew/Cellar/librdkafka/2.3.0/include LIBRARY_PATH=/opt/homebrew/Cellar/librdkafka/2.3.0/lib pip install confluent-kafka * Run tests make test-unit Benchmarks The following table shows the performance of different test scenarios: Name Throughput Max RSS Peak Memory Memory Usage Simple Aggregation 45,000 msgs / 230 MiB 130 MiB Memory sec Simple Aggregation Disk 36,000 msgs / 256 MiB 102 MiB sec Enrichment 13,000 msgs / 368 MiB 124 MiB sec CSV Disk Join 11,500 msgs / 312 MiB 152 MiB sec CSV Memory Join 33,200 msgs / 300 MiB 107 MiB sec In Memory Tumbling 44,000 msgs / 198 MiB 96 MiB Window sec More information about benchmarks are available in the wiki. Contact Us Like SQLFlow? Use SQLFlow? Feature Requests? Please let us know! danny@turbolytics.io About DuckDB for streaming data Resources Readme License MIT license Activity Stars 113 stars Watchers 5 watching Forks 7 forks Report repository Releases 3 In Memory Tumbling Window Support Latest Dec 26, 2024 + 2 releases Packages 0 No packages published Languages * Python 92.4% * Shell 6.0% * Other 1.6% Footer (c) 2025 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact * Manage cookies * Do not share my personal information You can't perform that action at this time.