https://github.com/kestra-io/kestra Skip to content Sign up * Product + Features + Mobile + Actions + Codespaces + Packages + Security + Code review + Issues + Integrations + GitHub Sponsors + Customer stories * Team * Enterprise * Explore + Explore GitHub + Learn and contribute + Topics + Collections + Trending + Learning Lab + Open source guides + Connect with others + The ReadME Project + Events + Community forum + GitHub Education + GitHub Stars program * Marketplace * Pricing + Plans + Compare plans + Contact Sales + Education [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this organization All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} kestra-io / kestra Public * Notifications * Fork 19 * Star 494 Kestra is an infinitely scalable opensource orchestration and scheduling platform, creating, running, scheduling, and monitoring millions of complex pipelines. kestra.io Apache-2.0 License 494 stars 19 forks Star Notifications * Code * Issues 63 * Pull requests 16 * Discussions * Actions * Security * Insights More * Code * Issues * Pull requests * Discussions * Actions * Security * Insights This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. develop Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags 26 branches 52 tags Code Latest commit @tchiotludo tchiotludo fix(core): prevent error for simple duration format ... 0f71964 Mar 16, 2022 fix(core): prevent error for simple duration format 0f71964 Git stats * 1,006 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time .github feat(cicd): build docker & maven in parallel Mar 14, 2022 cli fix(core): use a cached thread pool for worker to reduce memory usage Mar 2, 2022 core fix(core): prevent error for simple duration format Mar 16, 2022 docker feat(docker): build a full image with all plugins Nov 22, 2020 gradle chore(deps): update java deps Feb 22, 2022 indexer-kafka-elasticsearch chore(deps): update java deps Feb 22, 2022 repository-elasticsearch fix(kafka-runner): handle flow with multiple trigger that trigger mul... Feb 23, 2022 repository-memory chore(deps): update all java deps Dec 17, 2021 runner-kafka chore(docker): try a better startup process with docker-compose Feb 28, 2022 runner-memory feat(kafka-runner): used an in memory storage for flow & template (# 467) Feb 10, 2022 storage-local feat(core): add a retry on storage interface Jan 26, 2022 ui feat(ui): display color on log Mar 14, 2022 webserver feat(core): add a retry on storage interface Jan 26, 2022 .editorconfig fix(ui) add a better linter configuration Nov 5, 2020 .gitignore feat(ui): introduce unit test (#263) Jan 7, 2021 Dockerfile feat(docker): load some examples flow on docker-compose startup Mar 8, 2022 LICENSE chore(name): rename to final name Dec 27, 2019 README.md chore(docs): fix readme contributing link Mar 2, 2022 build.gradle chore(deps): update deps Mar 2, 2022 docker-compose-ci.yml fix(cicd): fix not starting kafka server Dec 11, 2021 docker-compose.yml chore(docker): fix kestra port Mar 10, 2022 gradle.properties chore(version): update snapshot version 'v0.4.3-SNAPSHOT'. Mar 11, 2022 gradlew chore(deps): update all java deps Dec 17, 2021 gradlew.bat chore(deps): update all java deps May 10, 2021 lombok.config feat(test): add jacoco report Nov 2, 2019 settings.gradle feat(indexer-kafka-elasticsearch): introduce bulk indexer for kafka & ... Nov 19, 2020 View code [ ] Infinitely scalable open source orchestration & scheduling platform. Demo What is Kestra ? Getting Started Prerequisites Launch Kestra Plugins Community Support Roadmap Developing locally & Contributing License README.md Kestra workflow orchestrator Infinitely scalable open source orchestration & scheduling platform. License Commits-per-month Github star Last Version Docker pull Artifact Hub Kestra infinitely scalable orchestration and scheduling platform Discord Github discussions Twitter Code Cov Github Actions Website * Twitter * Linked In * Discord * Documentation modern data orchestration and scheduling platform Demo Play with our demo app! What is Kestra ? Kestra is an infinitely scalable orchestration and scheduling platform, creating, running, scheduling, and monitoring millions of complex pipelines. * Any kind of workflow: Workflows can start simple and progress to more complex systems with branching, parallel, dynamic tasks, flow dependencies * Easy to learn: Flows are in simple, descriptive language defined in YAML;u don't need to be a developer to create a new flow. * Easy to extend: Plugins are everywhere in Kestra, many are available from the Kestra core team, but you can create one easily. * Any triggers: Kestra is event-based at heart--you can trigger an execution from API, schedule, detection, events * A rich user interface: The built-in web interface allows you to create, run, and monitor all your flows--no need to deploy your flows, just edit them. * [?] Enjoy infinite scalability: Kestra is built around top cloud native technologies--scale to millions of executions stress-free. Example flow: id: my-first-flow namespace: my.company.teams inputs: - type: FILE name: uploaded description: A Csv file to be uploaded through API or UI tasks: - id: archive type: io.kestra.plugin.gcp.gcs.Upload description: Archive the file on Google Cloud Storage bucket from: "{{ inputs.uploaded }}" to: "gs://my_bucket/archives/{{ execution.id }}.csv" - id: csvReader type: io.kestra.plugin.serdes.csv.CsvReader from: "{{ inputs.uploaded }}" - id: fileTransform type: io.kestra.plugin.scripts.nashorn.FileTransform description: This task will anonymize the contactName with a custom nashorn script (javascript over jvm). This show that you able to handle custom transformation or remapping in the ETL way from: "{{ outputs.csvReader.uri }}" script: | if (row['contactName']) { row['contactName'] = "*".repeat(row['contactName'].length); } - id: avroWriter type: io.kestra.plugin.serdes.avro.AvroWriter description: This file will convert the file from Kestra internal storage to avro. Again, we handling ETL since the conversion is done by Kestra before loading the data in BigQuery. This allow you to have some control before loading and to reject wrong data as soon as possible. from: "{{ outputs.fileTransform.uri }}" schema: | { "type": "record", "name": "Root", "fields": [ { "name": "contactTitle", "type": ["null", "string"] }, { "name": "postalCode", "type": ["null", "long"] }, { "name": "entityId", "type": ["null", "long"] }, { "name": "country", "type": ["null", "string"] }, { "name": "region", "type": ["null", "string"] }, { "name": "address", "type": ["null", "string"] }, { "name": "fax", "type": ["null", "string"] }, { "name": "email", "type": ["null", "string"] }, { "name": "mobile", "type": ["null", "string"] }, { "name": "companyName", "type": ["null", "string"] }, { "name": "contactName", "type": ["null", "string"] }, { "name": "phone", "type": ["null", "string"] }, { "name": "city", "type": ["null", "string"] } ] } - id: load type: io.kestra.plugin.gcp.bigquery.Load description: Simply load the generated from avro task to BigQuery avroOptions: useAvroLogicalTypes: true destinationTable: kestra-prd.demo.customer_copy format: AVRO from: "{{outputs.avroWriter.uri }}" writeDisposition: WRITE_TRUNCATE - id: aggregate type: io.kestra.plugin.gcp.bigquery.Query description: Aggregate some data from loaded files createDisposition: CREATE_IF_NEEDED destinationTable: kestra-prd.demo.agg sql: | SELECT k.categoryName, p.productName, c.companyName, s.orderDate, SUM(d.quantity) AS quantity, SUM(d.unitPrice * d.quantity * r.exchange) as totalEur FROM `kestra-prd.demo.salesOrder` AS s INNER JOIN `kestra-prd.demo.orderDetail` AS d ON s.entityId = d.orderId INNER JOIN `kestra-prd.demo.customer` AS c ON c.entityId = s.customerId INNER JOIN `kestra-prd.demo.product` AS p ON p.entityId = d.productId INNER JOIN `kestra-prd.demo.category` AS k ON k.entityId = p.categoryId INNER JOIN `kestra-prd.demo.rates` AS r ON r.date = DATE(s.orderDate) AND r.currency = "USD" GROUP BY 1, 2, 3, 4 timePartitioningField: orderDate writeDisposition: WRITE_TRUNCATE Getting Started To get a local copy up and running, please follow these simple steps. Prerequisites Make sure you have already installed: * Docker * Docker Compose Launch Kestra * Download the compose file here and save it with the name docker-compose.yml, for linux and macos, you can run wget https:/ /raw.githubusercontent.com/kestra-io/kestra/develop/ docker-compose.yml * Run docker-compose pull * Run docker-compose up -d * Open http://localhost:8080 on your browser * Follow this tutorial to create your first flow. * Read the documentation to understand how to + Develop your flows + Deploy Kestra + Use our terraform provider + Develop your own plugins Plugins Kestra is built on plugin systems. You can find your plugin to interact with your provider; alternatively, you can follow simple steps to develop your own plugin. Here are the official plugins that are available: Amazon S3 Avro Bash Big Query CSV ClickHouse ElasticSearch Email Google Cloud Storage Google Drive Google Sheets Groovy Http JSON Jython Kafka Kubernetes Microsoft SQL Server MongoDb MySQL Nashorn Node Open PGP Oracle Postgres Python Redshift SFTP Singer Slack Vectorwise Vertica XML This list is growing quickly as we are actively building more plugins, and we welcome contributions! Community Support Join our community if you need help, want to chat or have any other questions for us: * GitHub - Discussion forums and updates from the Kestra team * Twitter - For all the latest Kestra news * Discord - Join the conversation! Get all the latest updates and chat to the devs Roadmap See the open issues for a list of proposed features (and known issues) or look at the project board. Developing locally & Contributing We love contributions big or small, check out our guide on how to get started. See our Plugin Developer Guide for developing Kestra plugins. License Apache 2.0 (c) Kestra Technologies About Kestra is an infinitely scalable opensource orchestration and scheduling platform, creating, running, scheduling, and monitoring millions of complex pipelines. kestra.io Topics workflow data pipeline etl workflow-engine scheduler orchestration data-engineering dataflow elt data-pipeline workflow-automation kestra Resources Readme License Apache-2.0 License Code of conduct Code of conduct Stars 494 stars Watchers 6 watching Forks 19 forks Releases 3 v0.4.2 Latest Mar 13, 2022 + 2 releases Contributors 9 * @tchiotludo * @eregnier * @brahimalm * @djKooks * @dependabot[bot] * @v1nc3n4 * @corentinghigny * @loris-intergalactique * @aurelienWls Languages * Java 81.2% * Vue 13.9% * JavaScript 3.6% * SCSS 0.8% * Handlebars 0.2% * CSS 0.1% * Other 0.2% * (c) 2022 GitHub, Inc. * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.