https://github.com/streamdal/streamdal Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} streamdal / streamdal Public * Notifications * Fork 1 * Star 168 Deployment for Streamdal server www.streamdal.com License Apache-2.0 license 168 stars 1 fork Activity Star Notifications * Code * Issues 19 * Pull requests 0 * Actions * Projects 1 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights streamdal/streamdal This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 6 branches 0 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/s] Use Git or checkout with SVN using the web URL. [gh repo clone stream] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit GitHub Actions Update Helm chart ... c77684e Oct 31, 2023 Update Helm chart c77684e Git stats * 178 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time .github/workflows install.sh; restructure; improved main README October 29, 2023 00:23 assets streamdal logo October 30, 2023 17:37 install Update Helm chart October 31, 2023 13:38 .gitignore add node sdk examples October 30, 2023 14:21 LICENSE add license October 21, 2023 12:27 README.md capabilities doc fix October 30, 2023 23:40 install.sh use https for git October 30, 2023 21:54 View code [ ] Live Demo DEMO.STREAMDAL.COM Getting Started Install Instrument How Does It Work? Community Resources Getting Help Manifesto Roadmap Contributing License README.md [streamdal-][streamdal-] GitHub Discord This is the main repo for the Streamdal project. Streamdal is a combination of open source services and SDKs that enable real-time: * Data observability (think tail -f for your data!) * Data governance & enforcement * Data quality monitoring * Data transformation + obfuscation + masking + stripping .. and a bunch of other things. [This is what it looks like:] [img] Overview * Demo * Getting Started * How Does It Work? * Community * Resources Live Demo You don't have to install the server, the console or instrument any of your apps to see Streamdal in action. We've got a live demo :) DEMO.STREAMDAL.COM The demo will showcase: 1. Tail + Select any Component and start Tail request to watch live output 2. Welcome Service/Emailer + View how attached Mask dot com emails pipeline obfuscates recipient field if field contains .com 3. Signup Service/Recorder + Observe attached Scrub ICMP pipeline changing object.ipv4_address if object.protocol equals "ICMP" 4. Observe Throughput + Watch the read/write-per-second throughput for any component 5. Data Graph + Use the service map to see all services, operations and components and how they relate to each other + The data graph is the "node map" of all services and components 6. Schema Inference + View the inferred JSON Schema for all consumers and producers Each one of these actions is occurring in real-time! [You can read more about how this is achieved in the "how does it work?" docs.] Getting Started Getting started consists of two steps: 1. Installing the server, console and their dependencies 2. Instrumenting your code with one of our SDKs Install The easiest way to get Streamdal running is via curl | bash: curl -sSL https://sh.streamdal.com | bash 1. The install script will verify that you have git, docker and docker-compose installed 2. The install script will clone this repo to ~/streamdal 3. The install script will bring up all components via docker-compose Once done: Openhttp://localhost:8080 in your browser! You should be presented with a beautiful (but empty) UI! To populate it, we will need to instrument some code. Onto the next section! [For alternative installation methods, check the install dir.] Instrument Once you've installed the server and console, you can instrument your code using one of our SDKs. To see an example of a complete instrumentation, take a look at the Go demo client that is bundled with the server. Go package main import ( "context" "github.com/streamdal/go-sdk" ) func main() { sc, _ := streamdal.New(&streamdal.Config{ ServerURL: "streamdal-server.svc.cluster.local:8082", ServerToken: "1234", ServiceName: "billing-svc", ShutdownCtx: context.Background(), }) resp, _ := sc.Process(context.Background(), &streamdal.ProcessRequest{ OperationType: streamdal.OperationTypeConsumer, OperationName: "new-order-topic", ComponentName: "kafka", Data: []byte(`{"object": {"field": true}}`), }) } Python from streamdal import (OPERATION_TYPE_CONSUMER, ProcessRequest, StreamdalClient, StreamdalConfig) client = StreamdalClient( cfg=StreamdalConfig( service_name="order-ingest", streamdal_url="streamdal-server.svc.cluster.local:8082", streamdal_token="1234", ) ) res = client.process( ProcessRequest( operation_type=OPERATION_TYPE_CONSUMER, operation_name="new-order-topic", component_name="kafka", data=b'{"object": {"field": true}}', ) ) Node.js import { OperationType, Streamdal } from "@streamdal/node-sdk/streamdal"; export const example = async () => { const streamdal = new Streamdal({ streamdalUrl: "localhost:8082", streamdalToken: "1234", serviceName: "test-service-name", pipelineTimeout: "100", stepTimeout: "10", }); const result = await streamdal.processPipeline({ audience: { serviceName: "test-service", componentName: "kafka", operationType: OperationType.PRODUCER, operationName: "kafka-producer", }, data: new TextEncoder().encode(JSON.stringify({ key: "value" })), }); }; Important These are basic, minimal examples and should NOT be used in production code. Refer to the instrumentation docs for more thorough directions. How Does It Work? Streamdal consists of three main components: * Server * Console * SDKs The basic flow is that you install the server and console and wrap any reads or writes in your app with our SDK. Once that's done, you will be able to see the data your app is reading or writing in the console (or use the CLI). You will also be able to enforce rules on your data (such as "this should be valid JSON", "message should contain a field called foo", "strip all email addresses" and so on). Important For a more in-depth explanation of the flow and the various components, visit our docs. Community We're building Streamdal in the open and we'd love for you to join us! Join our Discord! Resources Getting Help Stuck? Something not working right? Have questions? * First and easiest way to get help is to join our Discord * If you're not in the mood to chat - there's docs * If all else fails, open an issue! Manifesto To get a better understanding of what we're trying to do, our ethos, principles, and really, our state of mind - check out our manifesto. Roadmap You have control over what we're building - our roadmap is 100% public! Feel free to stop by to discuss features, suggest new ones or just keep an eye on what's in the pipeline. Contributing We [?] contributions! But... before you craft an amazing PR, please read through our contributing docs. License This project is licensed under the Apache-2.0 license. See the LICENSE file for more info. About Deployment for Streamdal server www.streamdal.com Topics nodejs javascript python go docker rust reactjs event-driven observability streaming-data message-queues deno tail-f data-contracts astrojs Resources Readme License Apache-2.0 license Activity Stars 168 stars Watchers 3 watching Forks 1 fork Report repository Releases No releases published Packages 0 No packages published Contributors 5 * @dselans * @fritzstauff * @jacobheric * @blinktag Languages * Shell 56.0% * Smarty 44.0% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time.