https://github.com/spiceai/spiceai Skip to content Toggle navigation Sign in * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} spiceai / spiceai Public * Notifications * Fork 44 * Star 912 * A unified SQL query interface and portable runtime to locally materialize, accelerate, and query data tables sourced from any database, data warehouse, or data lake. docs.spiceai.org License Apache-2.0 license 912 stars 44 forks Branches Tags Activity Star Notifications * Code * Issues 15 * Pull requests 2 * Actions * Projects 0 * Security * Insights Additional navigation options * Code * Issues * Pull requests * Actions * Projects * Security * Insights spiceai/spiceai This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. trunk BranchesTags Go to file Code Folders and files Name Name Last commit Last commit message date Latest commit History 687 Commits .github .github .vscode .vscode bin bin crates crates docs docs examples examples install install monitoring monitoring test/tpch test/tpch tools tools .gitignore .gitignore CODEOWNERS CODEOWNERS CODE_OF_CONDUCT.md CODE_OF_CONDUCT.md CONTRIBUTING.md CONTRIBUTING.md Cargo.lock Cargo.lock Cargo.toml Cargo.toml Dockerfile Dockerfile LICENSE LICENSE Makefile Makefile README.md README.md SECURITY.md SECURITY.md acknowledgements.md acknowledgements.md clippy.toml clippy.toml go.mod go.mod go.sum go.sum version.txt version.txt View all files Repository files navigation * README * Code of conduct * Apache-2.0 license * Security Spice.ai OSS CodeQL License: Apache-2.0 Discord Follow on Twitter What is Spice? Spice is a small, portable runtime that provides developers with a unified SQL query interface to locally materialize, accelerate, and query data tables sourced from any database, data warehouse, or data lake. Spice makes it easy to build data-driven and data-intensive applications by streamlining the use of data and machine learning (ML) in software. The Spice runtime is written in Rust and leverages industry leading technologies like Apache DataFusion, Apache Arrow, Apache Arrow Flight, and DuckDB. Spice.ai Why Spice? Spice makes querying data by SQL across one or more data sources simple and fast. Easily co-locate a managed working set of your data with your application or ML, locally accelerated in-memory with Arrow, with SQLite/DuckDB, or with an attached database like PostgreSQL for high-performance, low-latency queries. Before Spice Before Spice With Spice With Spice Example Use-Cases 1. Faster applications and frontends. Accelerate and co-locate datasets with applications and frontends, to serve more concurrent queries and users with faster page loads and data updates. 2. Faster dashboards, analytics, and BI. Faster, more responsive dashboards without massive compute costs. 3. Faster data pipelines, machine learning training and inferencing. Co-locate datasets in pipelines where the data is needed to minimize data-movement and improve query performance. 4. Easily query many data sources. Federated SQL query across databases, data warehouses, and data lakes using Data Connectors. Supported Data Connectors Currently supported data connectors for upstream datasets. More coming soon. Name Description Status Protocol/Format Refresh Modes databricks Databricks Alpha Delta Lake full postgres PostgreSQL Alpha full spiceai Spice.ai Alpha Arrow Flight append, full s3 S3 Alpha Parquet full dremio Dremio Alpha Arrow Flight SQL full snowflake Snowflake Coming soon! Arrow Flight SQL full bigquery BigQuery Coming soon! Arrow Flight SQL full mysql MySQL Coming soon! full Supported Data Stores Currently supported data stores for local materialization/ acceleration. More coming soon. Name Description Status Engine Modes arrow In-Memory Arrow Records Alpha memory duckdb Embedded DuckDB Alpha memory, file sqlite Embedded SQLite Alpha memory, file postgres Attached PostgreSQL Alpha Intelligent Applications Spice enables developers to build both data and AI-driven applications by co-locating data and ML models with applications. Read more about the vision to enable the development of intelligent AI-driven applications. [?][?] DEVELOPER PREVIEW Spice is under active alpha stage development and is not intended to be used in production until its 1.0-stable release. If you are interested in running Spice in production, please get in touch so we can support you (See Connect with us below). Quickstart quickstart.mp4 macOS, Linux and WSL Step 1. Install the Spice CLI: curl https://install.spiceai.org | /bin/bash Or using brew: brew install spiceai/spiceai/spice Step 2. Initialize a new Spice app with the spice init command: spice init spice_qs A spicepod.yaml file is created in the spice_qs directory. Change to that directory: cd spice_qs Step 3. Start the Spice runtime: spice run Example output will be shown as follows: Spice.ai runtime starting... Using latest 'local' runtime version. 2024-02-21T06:11:56.381793Z INFO runtime::http: Spice Runtime HTTP listening on 127.0.0.1:3000 2024-02-21T06:11:56.381853Z INFO runtime::flight: Spice Runtime Flight listening on 127.0.0.1:50051 2024-02-21T06:11:56.382038Z INFO runtime::opentelemetry: Spice Runtime OpenTelemetry listening on 127.0.0.1:50052 The runtime is now started and ready for queries. Step 4. In a new terminal window, add the spiceai/quickstart Spicepod. A Spicepod is a package of configuration defining datasets and ML models. spice add spiceai/quickstart The spicepod.yaml file will be updated with the spiceai/quickstart dependency. version: v1beta1 kind: Spicepod name: PROJECT_NAME dependencies: - spiceai/quickstart The spiceai/quickstart Spicepod will add a taxi_trips data table to the runtime which is now available to query by SQL. 2024-02-22T05:53:48.222952Z INFO runtime: Loaded dataset: taxi_trips 2024-02-22T05:53:48.223101Z INFO runtime::dataconnector: Refreshing data for taxi_trips Step 5. Start the Spice SQL REPL: spice sql The SQL REPL inferface will be shown: Welcome to the interactive Spice.ai SQL Query Utility! Type 'help' for help. show tables; -- list available tables sql> Enter show tables; to display the available tables for query: sql> show tables +------------+ | table_name | +------------+ | taxi_trips | +------------+ Query took: 0.007505084 seconds. 1/1 rows displayed. Enter a query to display the longest taxi trips: sql> SELECT trip_distance_mi, total_amount FROM taxi_trips ORDER BY trip_distance_mi DESC LIMIT 10; Output: +------------------+--------------+ | trip_distance_mi | total_amount | +------------------+--------------+ | 191.9 | 3.0 | | 189.2 | 63.0 | | 163.8 | 93.64 | | 122.4 | 160.0 | | 104.0 | 3.0 | | 69.7 | 213.58 | | 64.8 | 280.83 | | 60.0 | 350.12 | | 53.9 | 0.0 | | 53.3 | 5.33 | +------------------+--------------+ Query took: 0.002458976 seconds Next Steps You can use any number of predefined datasets available from Spice.ai in the Spice runtime. A list of publically available datasets from Spice.ai can be found here: https://docs.spice.ai/building-blocks/datasets. In order to access public datasets from Spice, you will first need to create an account with Spice.ai by selecting the free tier membership. Navigate to spice.ai and create a new account by clicking on Try for Free. spiceai_try_for_free-1 After creating an account, you will need to create an app in order to create to an API key. create_app-1 You will now be able to access datasets from Spice.ai. For this demonstration, we will be using the Spice.ai/eth.recent_blocks dataset. Step 1. Log in and authenticate from the command line using the spice login command. A pop up browser window will prompt you to authenticate: spice login Step 2. Initialize a new project and start the runtime: # Initialize a new Spice app spice init spice_app # Change to app directory cd spice_app # Start the runtime spice run Step 3. Configure the dataset: In a new terminal window, configure a new dataset using the spice dataset configure command: spice dataset configure You will be prompted to enter a name. Enter a name that represents the contents of the dataset dataset name: (spice_app) eth_recent_blocks Enter the description of the dataset: description: eth recent logs Enter the location of the dataset: from: spice.ai/eth.recent_blocks Select y when prompted whether to accelerate the data: Locally accelerate (y/n)? y You should see the following output from your runtime terminal: 2024-02-21T22:49:10.038461Z INFO runtime: Loaded dataset: eth_recent_blocks Step 4. In a new terminal window, use the Spice SQL REPL to query the dataset spice sql SELECT number, size, gas_used from eth_recent_blocks LIMIT 10; The output displays the results of the query along with the query execution time: +----------+--------+----------+ | number | size | gas_used | +----------+--------+----------+ | 19281345 | 400378 | 16150051 | | 19281344 | 200501 | 16480224 | | 19281343 | 97758 | 12605531 | | 19281342 | 89629 | 12035385 | | 19281341 | 133649 | 13335719 | | 19281340 | 307584 | 18389159 | | 19281339 | 89233 | 13391332 | | 19281338 | 75250 | 12806684 | | 19281337 | 100721 | 11823522 | | 19281336 | 150137 | 13418403 | +----------+--------+----------+ Query took: 0.004057791 seconds You can experiment with the time it takes to generate queries when using non-accelerated datasets. You can change the acceleration setting from true to false in the datasets.yaml file. Upcoming Features See the Roadmap to v1.0-stable for upcoming features. Connect with us We greatly appreciate and value your support! You can help Spice in a number of ways: * Build an app with Spice.ai and send us feedback and suggestions at hey@spice.ai or on Discord, X, or LinkedIn. * File an issue if you see something not quite working correctly. * Join our team (We're hiring!) * Contribute code or documentation to the project (see CONTRIBUTING.md). [?] star this repo! Thank you for your support! For a more comprehensive guide, see the full online documentation. About A unified SQL query interface and portable runtime to locally materialize, accelerate, and query data tables sourced from any database, data warehouse, or data lake. docs.spiceai.org Topics infrastructure data machine-learning sql time-series artificial-intelligence developers Resources Readme License Apache-2.0 license Code of conduct Code of conduct Security policy Security policy Activity Custom properties Stars 912 stars Watchers 19 watching Forks 44 forks Report repository Releases 39 Spice.ai v0.10-alpha Latest Mar 28, 2024 + 38 releases Packages 2 Contributors 19 * * * * * * * * * * * * * * + 5 contributors Languages * Rust 77.4% * Go 19.2% * Shell 2.0% * Makefile 1.2% * Dockerfile 0.2% Footer (c) 2024 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact * Manage cookies * Do not share my personal information You can't perform that action at this time.