[HN Gopher] Windmill: Open-source developer platform to turn scr...
___________________________________________________________________
Windmill: Open-source developer platform to turn scripts into
workflows and UIs
Author : InitEnabler
Score : 401 points
Date : 2023-05-12 18:20 UTC (1 days ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| xrd wrote:
| Can this be used to host external services? Is this only for faas
| that are inaccessible to the outside world? I'm curious if you
| could use this to build web hook tools specifically.
| rubenfiszel wrote:
| Webhooks require to be authentified through tokens passed as
| bearer tokens or query args. This is because we have a tight
| permissioning system around visibility of
| scripts/variables/resources/flows, etc.... But nothing prevent
| you to add a proxy on top that call every webhook with a
| sufficiently permissioned token.
| xrd wrote:
| Appreciate the comment and your hard work on this.
|
| I reviewed the examples section, but didn't see one (a
| webhook?) as a good place to start. Do you have a suggestion?
| fulafel wrote:
| (Not op)
|
| https://docs.windmill.dev/docs/core_concepts/webhooks#trigg
| e... seems to cover a hello world for hooks.
| nico wrote:
| Would love a Jupyter or chat style interface for producing the
| scripts, from within the platform
|
| Amazing concept, thank you
| rubenfiszel wrote:
| We have someone from our team working on exactly this. We will
| release that feature in the following months.
| sisve wrote:
| I actually got codeium to work on windmill today trough their
| chrome extension. the extension lets you whitelist sites to
| work on. They do not offer a chat interface yet for the chrome
| extensions but they do have it on other integrations they
| provide. So I would not be surprised if they got it on the
| chrome extension also, at least if it turns out to be popular.
| rubenfiszel wrote:
| Hello, founder here. Thank you for submitting Windmill. We're
| really close from hitting v2 (adding multiplayer with yjs is our
| last milestone) and would have waited a bit before submitting but
| here we go.
|
| We are fully open-source (AGPLv3) with just one ee plugin for
| syncing the cache of the workers at large scale (more on that
| below). You can deploy it on a small ec2 instance with our
| docker-compose or on very large clusters with our helm charts:
| https://docs.windmill.dev/docs/advanced/self_host
|
| We are an onion of 3 layers, which are usually separated
| verticals and the closest services we replace in each vertical
| are: Lambda + n8n/temporal + Retool:
|
| - Workers implemented in Rust that can run any scripts from
| source in python/typescript(deno)/go/bash. They are extremely
| efficient, with most typescript scripts being able to run in 40ms
| e2e. This can be seen as a self-hosted lambda except it works
| very differently. Lambdas spawn microvms using firecracker and
| launch http servers within those micro-vms. We run the scripts
| bare (with isolation using nsjail on our multi-tenant instance)
| but cache very aggressively the dependencies. For instance, in
| python we analyze the dependencies using the AST:
| https://github.com/windmill-labs/windmill/blob/main/backend/...,
| infer the dependencies from there, have them go through pip-
| compile to generate a lockfile (and cache the result), then
| create a pathset similar to how virtualenv behave with the
| version of each dependency locked. For typescript, we leverage
| the very well made immutable deno cache. The cache works well but
| if you scale your cluster to more than 10 workers, the likelihood
| of having your worker see dependencies for the first time
| increase, which is why we reimplemented a pip scheme backed by s3
| that is extremely fast (we do not compress, just tar individual
| wheels and s3 is very fast on local networks). For all the
| supported languages, we parse the signatures of the main function
| to infer the corresponding json schema of the payload to trigger
| them, and from that jsonschema we can then generate automatically
| the UI and infer the input shape required in the flows for that
| particular step.
|
| - A workflow engine re-implemented from scratch which can be seen
| as a distributed FSM and whose state is stored in postgresql and
| every transition done through postgres transactions. Every worker
| is picking a job and progressing the fsm at completion. The full
| spec is at: https://docs.windmill.dev/docs/openflow. We support
| branches, for loop, suspend/sleep, approval steps, sharing the
| same folder between steps and everything you would expect from a
| temporal like engine (retries, error handlers).
|
| - A dashboard/app builder that is very similar to Retool but much
| faster because it's implemented in Svelte and which can also run
| python scripts directly leveraging the 1st layer. We also support
| full react apps: https://github.com/windmill-labs/windmill-react-
| template
|
| We have a CLI that allows you to sync from github directly:
| https://github.com/windmill-labs/windmill-sync-example or your
| local filesystem. Our scripts are absolutely normal scripts so
| there is no lock-in, and they are executable locally
| https://docs.windmill.dev/docs/advanced/local_development so you
| do not have to use our webeditor if you do not want to but our
| editor supports pyright and deno lsp through our dedicated
| backend communicating to monaco using websockets.
|
| We also support worker groups to run some jobs on hardware
| accelerated machines (GPU) and a lot of other features (oauth
| syncing of the resources, schedules, granular permissioning,
| groups and folders).
|
| Every script/flow deployed has a dedicated webhook endpoint to
| run synchronously and asynchronously. Scripts are never
| overwritten, they keep a lineage of hash for each version and
| have per-hash dedicated webhook to ensure their behavior never
| change.
|
| We also have a hub for sharing re-usable scripts
| https://hub.windmill.dev to be used in flows to have some of the
| same convenience you can find in Pipedream or Zapier.
|
| Our closest alternative would be Airplane.dev, but they're not
| open-source, not self-hostable air-gapped, less performant, less-
| featured and much more expensive. You can self-host us fully
| free.
|
| We are used in production at scale in clusters of 50 nodes+ and
| thanks to the reliability of Rust and Postgresql have had no
| downtime to report.
|
| I'm a solo founder, YC S22. The MVP got some attention at the
| very beginning: https://news.ycombinator.com/item?id=31272793 but
| was very bare. Now we're ready to bring this to the next level
| and enable all engineers to spend more time building and less
| time reinventing the wheel thanks to it :)
| SteveNuts wrote:
| What did you use to generate that layered 3d diagram on y the
| readme?
| rubenfiszel wrote:
| Figma :)
| EMIRELADERO wrote:
| Your product sounds extremely great! I'm hooked!
|
| One nitpick: In my humble opinon, you should switch from
| Discord chat to Zulip.
|
| It's completely open-source (Apache 2, self-hostable, free full
| cloud plan for open-source projects) and has an excellent
| threads-first (called "topics" model): https://zulip.com/why-
| zulip/
|
| It can run fully in the browser, but it has desktop and mobile
| clients as well.
|
| It also supports public channels which AFAIK discord doesn't.
| So you can link to specific messages inside a GitHub issue and
| anyone who doesn't even have an account can see the relevant
| messages and context.
|
| Interestingly it's what the Rust language team (and community)
| uses for comms: https://rust-lang.zulipchat.com
|
| (I'm not affiliated in any way with Zulip btw, I just love the
| product)
| rubenfiszel wrote:
| I'm very tempted and agree that Zulip sounds much better than
| Discord. The only thing I'm a little bit worried about is
| that I have noticed that most people hate having multiple
| chat clients open and that discord seem to be the de factor
| standards for open source projects. I apologize to be
| contributing to the network effect and will strongly consider
| switching to Zulip.
| alfonmga wrote:
| Discourse[0] is a perfect fit for Windmill, IMO. Maybe I'm
| biased because I don't like Discord for this kind of
| project. Let the search engines leverage your project.
|
| [0] https://www.discourse.org
| enlyth wrote:
| I would suggest against it, Discord is the de facto
| standard, especially with the younger crowd, millennials
| and Gen Z. Everyone already has it installed and everyone
| knows exactly how it works, and it also "just works". I've
| never even heard of Zulip and I hang out on HN almost every
| day.
| EMIRELADERO wrote:
| Just because everyone knows how it works doesn't mean
| alternatives shouldn't be tried, especially when they're
| superior in the aspects that matter for the audience
| you're interested in.
|
| Besides, how do you think a competing product should rise
| then?
| enlyth wrote:
| I'm not trying to start an argument here, but I consider
| less friction and more familiarity to be superior for
| this use case.
|
| I was trying to provide a contrary opinion because I
| often see the HN hivemind hate on Discord heavily for no
| reason, while it's become a ubiquitous piece of software
| used by younger people that the older generations have
| been completely sleeping on and still think it's some
| niche version of IRC for gamers. Discord has hundreds of
| millions of users, it's a behemoth rivaling the big
| social networks.
|
| As a random user, if I were trying to ask a question
| about this piece of software, I want to open the Discord
| server and ask it, I don't want to start researching
| whatever Zulip is and how to use it and figuring out how
| a new interface works
| terinjokes wrote:
| It it against the Discord Terms of Service to extend the
| accessibility of the client past what is available in web
| standards (and what of that they implement). The
| regularly ban accounts of people who try to make it more
| accessible (and often of the users too).
|
| These might be good qualities for a gamer guild or
| YouTuber community, but it's a real barrier to
| participating in open source software communities.
|
| Zulip is open, and doesn't have these restrictions (and
| already had many bridges and accessibility integrations).
| I participated in the Oil Zulip, and it was less of a
| hassle tHan to join a new Discord guild and navigating
| their adhoc rules bots.
| EMIRELADERO wrote:
| That's a great response, thank you!
|
| As a sidenote: is the windmill hub open-source too? Does it
| make use of the base product to operate? I think it might
| be a good candidate for showing what your product is
| capable of, as an actual example of use in production
| rubenfiszel wrote:
| Windmill Hub is not open-source, not because we want to
| keep it private but because I wrote it a bit quickly
| using svelte kit and it's not ready for the world to see
| yet. It does not use windmill to operate but every
| actions trigger a windmill webhook and alert the discord.
| EMIRELADERO wrote:
| Got it. Thanks!
| abathur wrote:
| https://oilshell.zulipchat.com/ is a good ~applied-zulip
| example in OSS.
| revskill wrote:
| Could you consider these 2 things ?
|
| - Allow manage users via Keycloak or similar tools.
|
| - Allow embed UI to external React application.
| rubenfiszel wrote:
| Yes, you can do both. We actually have a guide about
| keycloak:
| https://docs.windmill.dev/docs/misc/setup_oauth#keycloak
|
| For the second part, you can just iframe the whole public app
| and we will provide a React component soon.
| revskill wrote:
| Thanks. But i prefera convenient ReactSDK which allows more
| flexibility via props.
| morgante wrote:
| Hey Ruben! The product looks pretty interesting and I love the
| underlying technical choices.
|
| One suggestion: I think you could do improve the docs by (a)
| providing a direct comparison to Airplane and (b) providing
| more code-oriented documentation.
|
| Most of you docs highlight the web editor which, frankly, I
| have little interest in. I want to write my scripts and views
| in my IDE and store them in GitHub. This is something Airplane
| really nails: their docs are clearly about how to use this as a
| _developer_. Compare your flows quick start [0] with their
| tasks quick start [1] and it really feels like Airplane is more
| suited to a developer audience. I 'm worried I will be locked
| into static control flow with yours, whereas Airplane makes it
| clear I can do ~any code I want, as long as it runs in
| Temporal.
|
| Of course, totally fair if you are trying to target more no-
| code use cases.
|
| [0]:
| https://docs.windmill.dev/docs/getting_started/flows_quickst...
| [1]: https://docs.airplane.dev/getting-started/orchestration
| rubenfiszel wrote:
| You're absolutely right. We're pretty weak in docs for
| developers that only want to use the CLI and the reason why I
| wanted to wait for the V2 is that we wanted to do a full
| cycle on showing the potential of the CLI for local dev and
| airplane-like kind of development.
|
| For the temporal thing, that's absolutely fair and completely
| understand the worry but since most workflows are about the
| steps and not the glue between them, you will be able to
| reuse most of your code. However, temporal is quite an heavy
| format for writing workflows so of course it will still be a
| bit of work.
|
| Our views support our retool like builder but also full react
| apps, or svelte, or vue, or anything that can compile to an
| iife. The example provided above can be iterated on with just
| a simple 'npm run dev': https://github.com/windmill-
| labs/windmill-react-template
| CSSer wrote:
| Why did you choose AGPLv3?
| rubenfiszel wrote:
| I love open-source and wanted it to be open-source but given
| that's our whole product and not just a part of a bigger
| ecosystem, and given that we intend to commercialize mostly
| the large enterprise audience, we needed to have a reason for
| the legal departments of those enterprises to push to buy our
| commercial license. So otherwise said, we do not want to
| employ dark patterns and want it to be as easy as possible to
| self-host but also need to be sustainable as a company. Most
| features that you would expect to be premium plugins are
| baked-in and open-source (sso, audit logs, smart
| autocompletion on the webeditor, etc etc)
| brunoqc wrote:
| > In addition, a commercial license grants you a dedicated
| engineer to transition your current infrastructure to
| Windmill, support with tight SLA, audit logs export
| features, SSO, unlimited users creation, advanced
| permission managing features such as groups and the ability
| to create more than one workspace.
|
| From the README. It sounds like a commercial license is
| needed for SSO and that the free version has a limited
| number of users.
| rubenfiszel wrote:
| Sorry that's obsolete, we will remove it from the Readme!
|
| The updated paragraph:
|
| > In addition, a commercial license grants you a
| dedicated engineer to transition your current
| infrastructure to Windmill, support with tight SLA, and
| our global cache sync for high-performance/no dependency
| cache miss of cluster from 10+ nodes to 200+ nodes.
| brunoqc wrote:
| No worries, thanks!
| abathur wrote:
| Is providing dependencies part of your problem domain? (I've
| done a little work on shell script dependencies in the Nix
| ecosystem.)
| jchw wrote:
| I am a fan of this concept, I've thought a lot about it actually.
| Having a proper open source option is awesome and there is no
| doubt I'll be trying this on some of my home/hobby infra to see
| what it has to offer. If nothing else, I really think we need
| more exploration into the space of adding some basic engineering
| and ops on top of things like this.
|
| Not just for servers and infrastructure, but does it bother
| anyone else that for example, the Linux desktop has robust APIs
| for managing powerful service daemons like systemd, but
| relatively few applications that use them and provide useful
| UIs/control panels/dashboards? Somehow for many of the tools we
| DO have, the Linux desktop is stuck in the age of using popen and
| regex against command output even though we really ought to be
| able to do better. It's an odd blindspot in a lot of open source
| work, maybe it is deceptively difficult!
|
| That's mostly nothing to do with this project though, but the two
| problems are intertwined in my head as problems where a little
| bit of engineering could really go a long way.
| 0r30 wrote:
| For your consideration. A single tool that does many jobs is in
| some ways contrary to linux/unix philosophy of creating tools
| that do a single job.
| jchw wrote:
| I am not really asking for tools that do a lot of jobs,
| though. A simple program that manages systemd services or
| provides a nice UI around journald logs seems completely
| reasonable to me.
|
| That said, I don't take the UNIX philosophy to be gospel, and
| frankly, very little of the Linux kernel or desktop really
| seems to. That seems to be closer to what the *BSD
| derivatives would steer towards. I just want a functional
| operating system, and the specific ideology or design of
| packages isn't too big of a concern so as long as the end
| result seems solid.
| 0r30 wrote:
| >> A simple program that manages systemd services or
| provides a nice UI around journald logs seems completely
| reasonable to me.
|
| Completely Agree
|
| >> [...] the specific ideology or design of packages isn't
| too big of a concern so as long as the end result seems
| solid.
|
| The framework of thinking, while I agree is not gospel, is
| helpful in guiding design thinking towards a shared goal.
| jameshart wrote:
| So what are all these 200 line bash scripts that hold the
| world together?
| hhh wrote:
| the EC2 networking script would be a good example...
| djbusby wrote:
| I thought the philosophy was to just build tools you want. So
| we have, eg systemd and OpenRC. Where one is many-tools and
| the other is composed of many-small-tools.
|
| Choice is better than "one true way".
| llanowarelves wrote:
| There's limits on either end of that spectrum (pile of
| bespoke pipes vs. Lotus Notes), that we can see a
| demonstrated interest in, somewhere in the middle: the
| success of tools like Docker, and arguably even something as
| banal as the concept of an OS (distribution). Any user-facing
| software, etc.
|
| You don't want everyone wasting too much time handrolling
| slightly-differing monstrosities instead of a cohesive design
| if it's not a business or engineering differentiator.
| robertlagrant wrote:
| Without looking in detail, I've had some experience with internal
| business process/app development, and I always thought Temporal
| would be a great technology to build a platform on. Great choice.
| thawab wrote:
| Hi, windmill is not using temporal, it's an alternative to it.
| robertlagrant wrote:
| Oh! I misread the readme. Ta!
| irf1 wrote:
| in this interview Ruben, the founder of Windmill, talks about his
| background, shares how the project got started (bootstrapped solo
| founder, went through YC last summer) and encourages more
| software engineers to start their own companies
|
| interview highlights (4min):
| https://www.youtube.com/watch?v=_jsl0k7LX34
|
| full interview (33min):
| https://www.youtube.com/watch?v=rPLVwFRjw7o
| ttt3ts wrote:
| Was hoping it allowed my runtime. I have a bunch of scripts in
| NodeJS, Python 2, C#, etc. I don't care to rewrite them and/or
| write scripts tied to a weird sandbox runtime.
|
| Any support for docker based workers?
| rubenfiszel wrote:
| We could but decided against it because we have no way to run
| those efficiently given our architecture and prefer to support
| a few languages and ensure we can run them blazing fast than
| all languages but with no guarantees of performance. In
| windmill, you have the guarantee to have no cold-start thanks
| to leveraging the great design of the deno runtime. For python,
| we had to do a LOT of engineering to make it work well and did
| not have the bandwidth to do the same for all languages.
| Ultimately, we could decide that some steps could be slow and
| it doesn't matter but it wasn't requested enough to prioritize
| it.
|
| To be clear, the workers can run in docker, but do not run
| docker jobs.
| ttt3ts wrote:
| Thanks for the clarification.
|
| Looks like you're a new y combinator startup. Kool. I would
| suggest you find away to allow existing code to be reused.
| Deno market share is very small and experienced/larger teams
| are not going to want to rewrite and learn that stack. They
| would have to depend on you, a small new company. Too much
| risk.
| simtel20 wrote:
| It depends on how efficient you need to be. There is the
| option to just use "shell" scripts where you run the
| command. I haven't done much with that, but it's there.
| jeremyjh wrote:
| airplane.dev is a commercial offering very similar in spirit;
| it is not free but not expensive. It has native support for
| node, python, SQL, shell scripts and a couple other options as
| well as docker images for everything else. It's been a game
| changer for me.
| pjmlp wrote:
| Go is now a script language?
| revskill wrote:
| Yes, just create a script.sh and append #!/usr/bin/go then
| you're ready to go.
| pjmlp wrote:
| If that is the measure for scripting, then any language goes.
| revskill wrote:
| Not quite, not all compiled language provide interpreting
| capability.
| rubenfiszel wrote:
| Our users requested it, so we build it. We use scripts where we
| could just say "code" or "serverless functions".
| iamjackg wrote:
| I get what they mean -- it has definitely gained a foothold in
| the tool/utility world thanks to easy cross-compilation and
| single-binary output. Given their target, I definitely
| understand why they would put it in that group.
| foreigner wrote:
| If I use your cloud hosted version, can I sync everything
| (scripts, config, and UI) to a Git repo at e.g. GitHub?
| rubenfiszel wrote:
| Yes: https://github.com/windmill-labs/windmill-sync-example
| foreigner wrote:
| That looks perfect thank you! This is a big thing I miss with
| Retool.
| sisve wrote:
| yes you can sync the scripts, the json for the flows and the
| app UI. Including variables, secrets, resource types etc.
|
| See https://github.com/windmill-labs/windmill-sync-example
|
| it should be trivial to push that to a self-hosted setup if you
| need it.
|
| I have not tried to take my entire workspace from self-hosted
| to cloud. But I have been exporting and importing individual
| scripts/flows/apps from self-hosted to cloud.
|
| (no affiliation, just a happy user)
| kosolam wrote:
| Cool project!
|
| I wonder which kind of organizations adopt tools of this kind?
|
| How does this play in the long run with maintenance and upgrades?
|
| It would be great to hear from someone that ran such thing for a
| few years and has some insights to share..
| metadat wrote:
| This looks cool as a concept, but I've already got script-server
| [0] and it works pretty great as a no/low code solution for UI-
| ifying shell scripts.
|
| What does windmill do that script-server can't? Or is windmill
| targeting a completely different area, and is thus irrelevant to
| me and my pile of 500+ python, shell, Golang, and java bin dir.
|
| https://github.com/bugy/script-server
| rubenfiszel wrote:
| We do automatic parsing of the parameters in
| typescript/python/go/bash, we infer your dependencies
| automatically.
|
| And we have workflows and a retool alternative/app builder
| baked-in. And granular permissions and a lot of other things...
| The UI of script-server looks a bit bare.
|
| But like with a lot of those tools, if you're happy with
| script-server then that's probably the most important!
| radiowave wrote:
| This looks very impressive.
|
| Thinking about it as an alternative to scripted workflows with
| Rundeck, it seems like the main gap here is getting scripts to
| run on a specific machine without needing to have set that
| machine up to run worker processes.
|
| (I could be misunderstanding here, because I haven't spotted
| anything in the docs that talks about the minimum deployment
| necessary just to allow a particular machine to run workers.)
|
| Really simple example: there's an administrative command on a
| mailserver, which can be used to create a new mailbox. I want to
| make that operation part of some workflow. With Rundeck, I can
| script this and I just have to make sure Rundeck has ssh
| credentials onto the mailserver; at execution time it'll deploy
| the script, substitute in whatever variables, run it and capture
| the output.
|
| I'm not clear on how I would achieve that with Windmill. Would it
| involve installing Windmill (or some component of it) on the
| mailserver, and setting it up as an individual worker group?
| rubenfiszel wrote:
| We have worker groups that would allow you to do exactly this:
| https://docs.windmill.dev/docs/core_concepts/worker_groups
|
| And you would do exactly what you suggest in the last paragraph
| radiowave wrote:
| Thanks for the reply. Winmill is definitely something that
| I'll need to take a closer look at, when time permits. I do
| suspect though that it's going to be a bit too heavyweight to
| be a direct replacement for Rundeck.
| slig wrote:
| Yesterday I submitted a one-click install to Caprover, never been
| so easy to self host. It's really fast on Oracle's free 4vCPU,
| 24GB RAM machine.
| rubenfiszel wrote:
| Thanks a lot for that contribution :)
| GGO wrote:
| If I use self-hosted version with my closed source software (by
| calling my app's APIs), will I have to disclose my App's source
| code due to AGPLv3?
| rubenfiszel wrote:
| No, just don't modify the software itself without making your
| changes open-sources and you will be good to go. Having scripts
| on the platform is not modifying the software.
| GGO wrote:
| Thanks Ruben
| mikepurvis wrote:
| Thank you for this: https://github.com/windmill-labs/windmill#cli
|
| My first questions with any new tool of this kind is immediately:
|
| - Can I run/validate branch versions of the scripts non-
| destructively, for example as part of pre-merge checks in an IaC
| CD workflow?
|
| - Can I run my scripts locally without a lot of drama, while
| developing?
|
| If the answer to either of these questions is no, then the tool
| is the moral equivalent of directly editing PHP files on the prod
| machine, unless you're committed to deploying an entire second
| instance of the tool for staging/dev purposes.
| rubenfiszel wrote:
| In opposite order
|
| 2) yes,
| https://docs.windmill.dev/docs/advanced/local_development
|
| 1) since you can run them locally, you can also run tests as
| part of your CI. You can also deploy a full windmill as part of
| your CI, deploy to that local instance, run some e2e tests by
| calling the webhooks, and then only deploy once those work.
|
| Note that we have a full drafting system baked in into the
| webeditor as well:
| https://docs.windmill.dev/docs/core_concepts/draft_and_deplo...
| and have support for folders with different permissions and you
| can create workspaces so you may implement dev/staging/prod in
| any fashion you like (different folder, workspace, instances)
| gregwebs wrote:
| How would I run unit tests on drafts?
| rubenfiszel wrote:
| You would run your CI in github like you would for normal
| codebase, the draft aspect is something that is only for
| the frontend editor. Every time you push from the the CLI
| (which you could do upon merge to main), it is akin to do a
| deployment of the new versions.
| [deleted]
| replwoacause wrote:
| Does this work with PowerShell scripts?
| sisve wrote:
| Currently the only support shell is Bash
| rubenfiszel wrote:
| We will add powershell support soon (within the next month),
| it's very easy to add given our current architecture and has
| been requested a few times.
| replwoacause wrote:
| Great, can't wait to try it out. I'll be checking back!
| ropeladder wrote:
| I'm confused as to how this compares to tools in the workflow
| manager space such as Airflow/Perfect/Dragster. The words
| describing the software are almost the same but it seems to come
| from alternate universe of tools I'm not familiar with. Could
| someone fill me in?
| pas wrote:
| this is targeting the "low-code" audience, like https://n8n.io
| rubenfiszel wrote:
| I like to think that we target hybrid teams made of low-
| code/semi-technical people AND engineers that expect to have
| the same characteristics seen in temporal/airflow.
| Abstraction without compromise, or zero-cost abstractions as
| the rust community like to say
| rubenfiszel wrote:
| It's similar to airflow/prefect/dragster in that it can run
| workflows, and those workflows can do ETL and ops and can run
| arbitrary code.
|
| It's different in those fashions. Windmill can run python but
| also typescript, go and bash. Airflow/Prefect etc uses code
| annotation which mean the code is actually specific to the
| platform. We actually have the steps be directly scripts/code
| with a main function, and then have a low-code builder to
| compose them into powerful workflows. That low-code builder
| produces a yaml like that one: https://github.com/windmill-
| labs/windmill-sync-example/blob/... which you could
| theoretically write by hand.
|
| The low-code builder is intuitive and similar to n8n except it
| has the feature-set of the above framework. It's easy to test
| the individual steps or to test the whole flow. The workflow
| engine is very efficient and running 50 steps will essentially
| have about 1s overhead total, and the steps are executed bare
| so it will take as much time as if you were running them
| without windmill on your node directly. They can be hardware
| accelerated easily in that fashion. We also support some
| features borrowed from temporal like the ability to suspend at
| no-cost flows until they are resumed by an external event such
| as an approval from slack.
| debarshri wrote:
| I have been following this space for really long time. I think
| closest competitor is Airplane.dev [1]. One of the most difficult
| thing to sell as a sales person is to sell an open ended workflow
| and UI engine and ask me to segment and prospect customers. It is
| not that you cant sell it, it is just too broad even for your
| customers to figure out a usecase, because theres always a
| specific tool to solve the usecase. For instance, stripe refund
| is one most common usecase posted by all of these platform, but
| there are actual tools that just does stripe refund and they
| solve it in depth. You can argue these tools are value for money
| but then if it involves build time, in long run it becomes
| expensive. Retool is actually captured the market really well. It
| is basically become the verb in the internal tool market, it is
| hard to compete with that even as an opensource project.
|
| [1] https://airplane.dev
| wanderingmind wrote:
| Its hard to predict how a good open source tool that solves
| hard and painful problems will evolve in a marketplace. I don't
| think Linux developers would have remotely dreamt of a cloud
| revolution when they were creating an open source kernel. Maybe
| windmill will enable a marketplace of automation tools that can
| be leveraged across multiple business for multiple usecases.
| Never underestimate a open source solution solving painful
| problems.
| debarshri wrote:
| You are argument is fair but being a VC backed company puts a
| deadline on the org. They have to not only prove that it
| solves a pain point but also make a ton of money and growth.
| rubenfiszel wrote:
| Do not worry about us, we are profitable, I have a majority
| stake, and our investors trust :)
| majkinetor wrote:
| Couple of questions:
|
| 1. Can I see list of all executions, their parameters and
| complete stdout, with handy URL I can refer too?
|
| 2. Can I get notifications (email etc) when script fails or
| certain conditions are met (something appears in the output)?
|
| 3. Can you set execution timeout?
| rubenfiszel wrote:
| 1. yes (that's exactly what the run page of each execution is)
|
| 2. yes (our flows have optional error handlers)
|
| 3. yes (TIMEOUT is an env variable of the workers)
| chillbill wrote:
| > To get credentials without the OAuth providers above, send an
| email at contact@windmill.dev
|
| Why? it's like they're going out of their way to add friction
| here.
| rubenfiszel wrote:
| We provide a generous free-tier of 1000 executions a month and
| as a very small team I'd rather rely on github/google bot abuse
| prevention than reinvent the wheel there. Email verification is
| not strong enough.
| marcodiego wrote:
| Can it be combined with Gitlab and set us free from Jira?
| rubenfiszel wrote:
| Yes, we expose webhooks for all scripts and flows so you can
| integrate it for anything that can send webhooks, including
| Gitlab, and our workflows can be used as CI/CD.
| thawab wrote:
| The best part about windmill is the team behind it, join the
| discord server to see how fast they fix issues and add features.
| The blog is also top notch: https://www.windmill.dev/blog
| slig wrote:
| Ruben is a machine, I've reported things and he fixed within
| minutes. Their discord server is top notch, 10/10.
| simtel20 wrote:
| Agreed. I hope that they are rewarded greatly for his and the
| team's fanatical attention to everyone who brings up
| suggestions and bugs.
| stavros wrote:
| We were just looking for something similar to this to run
| workflows at $JOB and decided to build our own in the end,
| because nothing we found fulfilled our requirements.
|
| A few questions, just in case:
|
| 1. Does Windmill support long-running workflows? Our use case has
| our customers being in a workflow for years (as long as they have
| a relationship with us) and events moving them around states.
|
| 2. Does it support rule-based state transitions? Everything I've
| seen is "when this webhook comes, go to that state", but we want
| something that will trigger the evaluation of a set of rules
| (with an event or webhook), and the rules will contact APIs to
| judge whether everything is satisfied for a transition to some
| next state.
|
| 3. Does it support synchronous calls? We usually want the
| customer to ask "OK where am I now?" and the system to trigger
| evaluation, transition them to the state they should be in, and
| tell them where they are. This is for use cases like the website
| asking "the user has just submitted the data we wanted, where
| should I redirect them now?".
|
| Those were the biggest things we couldn't find.
| rubenfiszel wrote:
| 1. Yes there are no time limits for workflows and there is no
| cost to keep a workflow suspended. It will just be scheduled to
| be reactivated in the future
|
| 2. The webhook can contain a payload, and you can but the
| suspend step into a for-loop. You could literally say, for-loop
| forever the suspend until the result of the suspend step (which
| can be the result of the evaluation of the webhook payload by
| your rule is true). For that, you will need to use another
| primitive that we expose which is "Break" that will break out
| of the for-loop
|
| 3. I'm a bit unsure of the term synchronous call. We have a
| long list of APIs https://app.windmill.dev/openapi.html#/ and
| you fetch the status of the flow or trigger a flow in a
| synchronous http webhook to wait for the result. As to force
| the state of the workflow. We do not expose something like that
| but it could be added easily.
|
| Happy to chat if we could be a good fit for your needs:
| ruben@windmill.dev
| stavros wrote:
| Thank you! It doesn't sound like an exact fit but Windmill
| looks great in general, well done!
|
| For the second point, the idea is that we don't want to rely
| on webhooks at all, except as evaluation triggers. The rules
| are built in to each state, so even if we've missed a
| webhook/event, the customer can still proceed.
|
| For an example, one of our flows requires the user to have
| filled out a consent form in the last three months. As I
| understand it, with your design, we need to send a webhook
| when a user has given consent, and then we can proceed to the
| next state, so we'd have to somehow check for previous
| consent in some different state? With our design, when the
| user reaches the state, the rules will be evaluated, and "has
| given consent within the last three months" will be true, and
| the user will proceed to the next state without ever seeing
| the consent form.
|
| For the synchronous bit, it kind of follows from the above
| feature, and it is so we evaluate state transitions while the
| user is waiting. We run the workflow synchronously, in the
| request (since no workflow _needs_ external events to
| continue), and return the new state to the user.
|
| Since this is synchronous, it avoids races or long waits for
| events to be delivered.
|
| Thanks for listening, maybe you'll find some of the above
| helpful in your design!
| rubenfiszel wrote:
| Makes sense. I understand the synchronous part now. I think
| there are ways to achieve the same benefits with an
| asynchronous system but if what you have works, no point
| changing it!
| 1000thVisitor wrote:
| How does this compare to Ansible, and AWX?
|
| https://github.com/ansible/awx
| deadfece wrote:
| Is there a comparison of this vs something like Rundeck?
| SPascareli13 wrote:
| How does it compare to something like apache airflow? I like the
| native Go integration, and that it is specifically mentioned that
| the overhead is minimal to start a job. Airflow can take quite a
| varying amount of time to schedule your job.
| rubenfiszel wrote:
| Passing information between steps is a pain in Airflow
| (https://docs.astronomer.io/learn/airflow-passing-data-
| betwee...) whereas it's quite trivial in windmill, the
| performances/latency is better, and it supports typescript.
| Overall, the biggest improvement is the support for
| typescript/go/bash and the low-code builder that will allow you
| to build extremely complex flows that would be hard to
| visualize with just code annotations but with exactly the same
| featureset and more (retries, error handlers, suspend/sleep).
| We also make it really easy to test individual steps and the
| whole flow since web apps allow you to build advanced UX fit
| for big graphs. But Airflow is a great framework that we took a
| lot of inspiration from.
| Too wrote:
| From that link
| print_the_cat_fact(get_a_cat_fact())
|
| Doesn't seem like a big pain?
|
| Looks like that page overcomplicate it by describing every
| underlying layer and old history of it.
| rubenfiszel wrote:
| That's because you're looking at the TaskFlow syntax and a
| trivial example of passing all the results from a Task A as
| all the inputs of a Task B that follows it immediately.
| TaskFlow works mostly in trivial case. The true syntax for
| it is in the tab "Traditional syntax". For a complex graph
| where you're passing data from step A to a node that is far
| down the tree and deeply nested and you will see that it's
| quite cumbersome. Even in their trivial example:
| res = requests.get(url)
| ti.xcom_push(key="cat_fact",
| value=json.loads(res.text)["fact"]) ...
| cat_fact = ti.xcom_pull(key="cat_fact",
| task_ids="get_a_cat_fact")
___________________________________________________________________
(page generated 2023-05-13 23:02 UTC)