[HN Gopher] Launch HN: Windmill (YC S22) - Turn scripts into int...
___________________________________________________________________
Launch HN: Windmill (YC S22) - Turn scripts into internal apps and
workflows
Ruben here, software engineer, long-time lurker of Hacker News and
founder of Windmill. Windmill is a fully open-source self-hostable
platform and runtime to build complex workflows, internal apps and
integrations using any scripts in Python or Typescript-deno. I am
back after having been revealed a bit too soon on HN and
miraculously getting into YC
(https://news.ycombinator.com/item?id=31272793). To build internal
apps for ops, integrations between services that cannot talk to
each other directly, or to run background jobs that run your
business logic and analytics, the two main options today are no-
code solutions and old-fashioned, roll-your-own scripting. Both
have problems, and our goal with Windmill is to find a new sweet
spot between the two. No-code solutions are productive _if_ your
problem matches the tool exactly - but it not, they are rigid, hard
to extend and quickly become tech debt, annihilating their initial
time advantage. Indeed, no-code is just code but made by an
opinionated someone else and hidden as a blackbox with an UI. The
alternative is to do it the old-fashioned way, writing everything
from scratch, both backend and frontend, perhaps deploying it on
the latest flavor of serverless, and pray to never have to touch it
again because that took way too much time and it has now became a
burden that the ops and business team might poke you about
regularly. Furthermore, the landscape of SaaS is specialized tools
for everything--alerting, data analytics, administration panels,
support management, integration between services--when it feels
like a few scripts would have been as good or even better and
spared you the need of depending on one yet another tool. This
could be even further facilitated if there was a way to import the
right bunch of scripts from a fellow community of engineers, tweak
it and deploy it like you can do in communities where automation
can be shared as simple JSON files, for instance in the node-red or
home assistant community.. That's the idea of Windmill: to bring
back the power of scripting in an easy way. With Windmill, you
write normal scripts, or reuse ones made by others, and we make
them production-grade and composable. You shouldn't have to worry
about things like http requests or scheduling jobs. We abstract
much of that away, making your scripts be both more focused and
more composable. You end up doing things the right way but much
quicker. We reduce the complexity of workflows, integrations and
internal apps by uniting them all under one banner. At the heart,
they mostly have the same needs: workflows with a UI or a schedule.
One tool that does it all out-of-the-box offers greater consistency
and allows you to grow the complexity of your toolset at your own
pace. I have an academic background in compilers and industry
experience in distributed systems. My compiler work made me wary of
solving every problem with a domain-specific-language (DSL) or
complex frameworks. We can just do more with the well-crafted
existing languages like Python or Typescript. Rolling up your own
DSL is nice in theory, you can make it very ergonomic and focused
on the task at hand, but then you start adding features and either
reinvent existing - albeit worse - programming language or decide
to stop there. In the very large majority of cases, a well crafted
library is vastly superior to any DSL. By being able to use any
library of Python and Typescript, we stand on the shoulders of
giants. I have also observed that the best distributed systems are
often the most simple as they are more predictable and have
invariants that are easier to reason with and scale horizontally.
This is why for Windmill, we rely solely on Postgres + our native
workers + our http REST api layer. Later on, we plan to build
adapters to host the workers on AWS lambda or Cloudflare workers,
and the queue on Kafka if your needs are exceptionally high. At
the heart of what we have built is a queue implemented in Postgres
and workers implemented in Rust that create a sandbox (using
nsjail), fetch dependencies, and execute scripts. Every script can
be triggered through its name with an HTTP POST by passing a JSON
payload in which every field corresponds 1:1 to an argument of the
script's main parameters. Most primitive types in Python or
Typescript have a natural corresponding type in JSON so the
conversion is always what you would expect. We then execute the
script inside a new sandbox and then store the results in the same
Postgres DB at the end of the job execution. The HTTP payload can
be sent from your own frontend or you can use our automatically
generated UI. Indeed, we do a simple, yet effective analysis of the
parameters of your script, and from it, generate the jsonschema
corresponding to your parameters. That schema is what enables us to
convert any script into a no-code like module for flows, or a
standalone internal app with its auto-generated UI. In the case of
Python, we also look at the imports to deduce the Pypi dependencies
without you having to declare them. For flows, we defined an open
spec for building them out of those scripts we call OpenFlow:
https://docs.windmill.dev/docs/openflow. It is essentially a json
format for describing a sequence of steps with for loops and soon
branching. The most interesting bit here is that each input of each
step can define its input as a javascript expression that refers to
and transforms the output of any previous step. We make it fast by
leveraging native v8 integration in Rust (thanks to the deno team)
for executing those expressions. This makes this apparently linear
sequence a flexible DAG in which one can express complex workflows.
Then on top of that we have an UI builder for flows that hides most
of the complexity to give an experience that is similar to a low-
code platform where every step is treated as a blackbox. The
platform itself offers all the features that you would expect: a
variable and object store for storing states, plain values and
credentials; a cron scheduler, tight permissioning for the
sensitive credentials, groups, a webeditor with smart assistant to
edit the scripts directly in the platform etc. Finally, we made a
hub (https://hub.windmill.dev) to share flows and scripts with
everyone. The goal is to grow over time an exhaustive library of
pre-made modules and flows to tweak from so that you can focus on
what is actually custom to you. Windmill is open-source and self-
hostable. You can think of it as a superset of both Pipedream and
Airplane.dev. Compared to Temporal, the scripts themselves are
agnostic of the flow in which they are embedded, which has the
benefit of making it easier to build a hub of reusable modules. We
are the only ones as far as we know to convert script parameters to
UI automatically. We see ourselves as complementary to UI builder
solutions like Retool or Tooljet as we do not want to focus too
much on the auto-generated UI and could be used solely as the
backend part of the two aforementioned tools. We are now a team of
3 senior engineers and the product is progressing faster than ever
with a public roadmap: https://github.com/orgs/windmill-
labs/projects/2 We make money from commercial licenses, support
and team plans on the hosted solution. You can self-host it or try
it https://app.windmill.dev, the free tier is generous (and the
paid one is not enforced yet). Our landing page is:
https://windmill.dev. We would appreciate your feedback and ideas
and look forward to all your comments!
Author : rubenfiszel
Score : 114 points
Date : 2022-08-09 17:19 UTC (5 hours ago)
| ersatz_username wrote:
| Congrats on the launch Ruben! It's really exciting to see an open
| source tool like this out in the wild.
| wizwit999 wrote:
| Awesome, Airplane for free, I love it.
| robertlagrant wrote:
| Don't look for free (as in beer) tools; look for good tools
| that are free (as in speech) and fund them.
| rubenfiszel wrote:
| Not just free, open-source! :)
| wizwit999 wrote:
| One q: why are you building a code running/functions platform
| inside your project, sounds very overdone and not related to
| your project. Also scary as a customer. I would recommend
| just make people host workers, we'll need it anyway to access
| resources, VPCs, etc.
| rubenfiszel wrote:
| I can understand being skeptic about that but we did not
| invent nsjail nor linux forks so it's pretty simple and
| rock solid: take source code, wrap it a little, execute it
| with python or deno. As soon as cloudflare workers is open-
| source, we will provide adapters. We did not have much
| alternative to make it easily self-hostable but interested
| to hear your suggestions.
| raviparikh wrote:
| I'm one of the founders at Airplane-we have a free plan as well
| up to 5 users: https://www.airplane.dev/pricing
|
| If pricing is a problem above that, I'm happy to chat about a
| discounted plan or higher free limit depending on your use
| case! Feel free to send me a note at ravi@airplane.dev
| radiojasper wrote:
| Oops, we killed it already.
|
| 503 Service Unavailable
|
| No server is available to handle this request.
| rubenfiszel wrote:
| It is back up!
| radiojasper wrote:
| legend! Thank you, that was quickly restored. Now let me go
| check it out!
| rubenfiszel wrote:
| Actually it's not completely up, the hub is down but will
| be up in a few mins.
| rubenfiszel wrote:
| On it!
| bearjaws wrote:
| I could see my company using this, we have some internal tooling
| that we were considering moving to Retool. I think for any of
| these platforms to succeed long term they need to be just as easy
| to back out of (e.g. once it gets too complicated, go back to
| normal code) as they are to start using.
|
| One thing that I misread and I believe others would too is on the
| main landing page where it has the list of features is one of
| them says "UI? Done (checkmark)" and the rest of the items don't
| have any Done or checkmark. It reads like the rest of the items
| listed aren't ready at all, which based on the docs they are at
| least available.
| rubenfiszel wrote:
| checkmark removed, thanks for the feedback. Looking forward to
| have demanding users with strong use-cases.
|
| About lock-in I completely agree which is why you can just
| deploy from github and export a tarball of your entire
| workspace at any point. The scripts can be used outside of
| Windmill (except if you use our K/V store but you do not have
| to). The day you want to migrate out, you will just have to
| redeploy your scripts as lambdas somewhere else.
| debarshri wrote:
| My 2 cents, choose retool. I have no association with retool
| but as a product/business owner question is do you want to risk
| with a startup or an established product that reasonably solves
| your problem. Retool is pretty reasonable pricing wise, very
| stable, great support and inherently you dont have to deal with
| startup like bugs.
| raviparikh wrote:
| Retool is a great product, but I don't think your reasoning
| is entirely sound.
|
| > do you want to risk with a startup or an established
| product that reasonably solves your problem
|
| I think it depends greatly and I don't think this is a hard-
| and-fast rule. If you're adopting a product that is easy to
| replace and doesn't have huge vendor lock-in, and the startup
| offering solves your problem better than the established
| vendor, then a startup might be a better choice. Windmill is
| an open source product, so even if they went out of business,
| you'd still have access to the code which is a great way to
| avoid some amount of the risk surface area.
| dang wrote:
| You've posted four comments now in someone else's launch
| thread. One was arguably maybe ok (though also arguably
| kind of rude). But you're going overboard now, so please
| stop.
|
| Whether it's YC-funded or not, each startup deserves to be
| the focus of its own launch thread. There will be plenty of
| other opportunities for you to make a case for your own
| startup.
| raviparikh wrote:
| Sorry about that - I edited my comment to keep the focus
| on Windmill.
| dang wrote:
| Ok, appreciated!
| aliasxneo wrote:
| I'll admit I only gave the docs a cursory review, but it might
| help to expand on what self-hosted means. Over at r/selfhosted,
| for example, it typically translates to, "can run on my hardware
| with no external dependencies." It's hard for me to tell if
| Windmill meets this definition, but early indications seem to
| suggest the answer is no.
|
| -- Edit --
|
| I just saw the Github repo. Does this section exist[1] in the
| docs anywhere? Might be good to include.
|
| [1]: https://github.com/windmill-labs/windmill#how-to-self-host
| rubenfiszel wrote:
| Thank you for giving me the opportunity to clarify this!
| https://github.com/windmill-labs/windmill#how-to-self-host
|
| You just need to docker-compose up https://github.com/windmill-
| labs/windmill/blob/main/docker-c... and you are all set :)
| nolok wrote:
| I believe his question was about "does this docker instance
| then rely on an external service/api to run", which it
| appears it does, and thus wouldn't quite qualify for fully
| self hosted.
|
| Many reasons why it's important to know that, the one that
| matters to me personnaly is because of data storage location
| requirements.
| rubenfiszel wrote:
| It does not rely on external api or services, you can run
| this completely isolated (well you would need a postgres
| instance). The WindmillHub that Windmill fetch data from is
| completely unidirectional and something purely out of
| convenience. In theory shouldn't have an impact in your
| ability to use the product without it, although we should
| introduce a flag for the product to not even attempt to
| list the available scripts from the community if that's
| your wish. I will write an issue on the roadmap about that.
| nolok wrote:
| Ah then I guess my (very quick) look at your link was
| wrong then, I apologize.
|
| I'm personally from the EU AND in an industry with
| stricter data control than usual which is why I made that
| particular point. If that is a possibility for your
| product then I believe you should absolutely have it
| somewhere on your roadmap, and then make it extra clear
| in your "self host" description (things like "on
| premise", "your data never leaves your server", "retain
| control of your data", ...). It's usually a lot easier to
| do it early and stand by it than try to make it possible
| later.
|
| Best of luck !
| rubenfiszel wrote:
| Yes one of our primary target is actually clients with
| regulatory, compliant or ethical concerns to not control
| all of their data on-prem. By being open-source and self-
| hostable, we can meet those standards. I am from France
| myself and am pretty sensitive about the data privacy
| issues.
| ninjaa wrote:
| This is awesome
| rubenfiszel wrote:
| Thank you!
| toomuchtodo wrote:
| Will you eventually support importing workflows from similar
| closed low code/no code tools?
| rubenfiszel wrote:
| If there is a way to do this in an automatic manner for sure
| but it's of course hard to reverse engineer proprietary spec.
|
| On the other hand, one model is to have low code/no code tools,
| or SaaS that want to add automation as a feature of their
| product, wrap us to run their backends as we have done a lot of
| effort on the engineering of the orchestrator/backend and we
| cannot compete on every vertical. If they all run deno/python
| code with the OpenFlow format, then everything is
| interchangeable everywhere!
| toomuchtodo wrote:
| Tremendous, wishing you much success!
| mxcrbn wrote:
| Windmill rocks, and the cloud version makes it easier to automate
| my stuffs now! Thanks guys
| rubenfiszel wrote:
| Thank you!
| mwcampbell wrote:
| The ability to run sandboxed workers from a PostgreSQL-based
| queue sounds useful all by itself, particularly for multi-tenant
| SaaS applications that need to do compute-intensive tasks on
| untrusted inputs. The difference, of course, is that in this
| case, the worker implementation is part of the application, not a
| user-defined script. How tightly are the work queue and sandboxed
| worker runner coupled to the rest of the product?
| raviparikh wrote:
| Congrats on the launch! I'm working on something similar,
| Airplane.dev, which you mentioned in your initial post.
|
| One thing you mentioned:
|
| > You can think of it as a superset of both Pipedream and
| Airplane.dev
|
| I don't think this is true that Windmill is a strict superset of
| Airplane. Airplane has a lot of functionality that Windmill
| doesn't seem to offer (based on my reading of your docs). A few
| things I noticed that are unique to Airplane:
|
| * Ability to store everything in code that you control. In
| Airplane, we have a CLI that lets you deploy scripts from your
| own machine to Airplane. You can version control the script code
| as well as the metadata (name, parameters, validation rules,
| etc). In Windmill you have to create everything in their UI as
| far as I can tell. Airplane's approach is more developer-friendly
| and also allows you to reference other code in your codebase if
| you want.
|
| * Many ways to create scripts/tasks outside of just Python and
| Typescript. We have a direct SQL integration, REST-based tasks
| without writing code, Shell, Docker, etc
|
| * Third-party integrations (Sendgrid, Slack, etc). I only saw
| Slack in your docs.
|
| There are of course things that Windmill does that Airplane
| doesn't (for example, Windmill is open source). Just wanted to
| point out that while our tools have some overlap, there are also
| many differences that might make one or the other a better fit.
| rubenfiszel wrote:
| > Ability to store everything in code that you control. In
| Airplane, we have a CLI that lets you deploy scripts from your
| own machine to Airplane. You can version control the script
| code as well as the metadata (name, parameters, validation
| rules, etc). In Windmill you have to create everything in their
| UI as far as I can tell. Airplane's approach is more developer-
| friendly and also allows you to reference other code in your
| codebase if you want.
|
| https://github.com/windmill-labs/windmill-gh-action-deploy lets
| you sync from github, including metadata that are stored in a
| json file alongside the code with the same name. What this
| integration does is basically just sending a tar of your repo
| to Windmill to sync it so you do not have to centralize your
| code on Windmill if you do not want to. We will provide the
| same mechanism for local filesystems and CLI or any other
| source control providers very soon as the approach will stay
| similar.
|
| It is not a strict superset and the approach are radically
| different in fundamental aspects, like operating at the source
| code-level versus container level, but what about the features
| you do not provide, like a fully-featured web editor with lsp
| in the backend for smart assistant. The ability to convert
| script parameters to UI directly and so on.
|
| > * Third-party integrations (Sendgrid, Slack, etc). I only saw
| Slack in your docs.
|
| This is disingenuous, we provide integrations for anything that
| has OAuth
| https://docs.windmill.dev/docs/contributors_guide#add-an-oau...
| by just needing to append the oauth informations to this file:
| https://github.com/windmill-
| labs/windmill/blob/main/backend/.... We have a hub for anyone
| to provide the no-code module part out of it. We will do it
| before the community reach a critical mass. I built this in 7
| months, the product just got extremely stable and feature-
| complete so we can now focus on adding integrations. We expect
| this ways to have thousands of integrations by the end of this
| quarter when given your current needs for custom work on each
| module, it will be harder for you to grow your integrations at
| the same rate. We have much more than just slack already and
| it's comparable to your current integrations.
|
| We have a public roadmap with all the features we intend to
| build over the next month if you are curious:
| https://github.com/orgs/windmill-labs/projects/2/views/1
|
| You have an amazing product but being open-source is not our
| sole differentiator, we want to build an even better amazing
| product.
|
| EDIT: The point holds that our docs need improvement and it's
| fair to assume that every integration that is not named
| explicitly does not exist. Our approach is very different as we
| believe less in custom integrations and more on relying on
| existing libraries in Deno and Python. In Windmill, it is
| sufficient to write a main function that call an existing
| library, and that we add the OAuth provider information to the
| library of integrated OAuth to have a very high quality
| integration experience so we did not feel the need to name all
| of them.
| raviparikh wrote:
| > we provide integrations for anything that has OAuth by just
| needing to append the oauth informations to this file
|
| Thanks for the clarification. I do think there is a
| difference between a dedicated, bespoke integration to a tool
| as compared to your approach, and that different customers
| may have different needs. I previously worked on an analytics
| startup, and in that industry, many products took a similar
| approach to what you're mentioning. However, Segment's
| strategy was to build first-party, high-quality connectors
| that took into account the particular nuances of each tool,
| and they were massively successful as a result.
|
| > We expect this ways to have thousands of integrations by
| the end of this quarter when you will only have a dozen.
|
| Interesting, looking forward to seeing this!
|
| > You have an amazing product but being open-source is not
| our sole differentiator, we want to build an even better
| amazing product.
|
| I didn't mean to imply it was the only differentiator, I
| meant that it was an example of a differentiator - apologies
| if that was unclear.
| [deleted]
| [deleted]
| [deleted]
| andrewguenther wrote:
| I beg of you, please reconsider the SSO tax (https://sso.tax/).
| This is becoming a big pain point for small orgs trying to become
| compliant that a small change like this can be huge in decision
| making. If I look at Pipedream, Airplane, Temporal, and Retool
| none of them will even list pricing if I need SSO, I'm suddenly a
| 40 person company with "Enterprise" needs. If you want an easy
| differentiator, including SSO in your Team tier is a simple way
| to do that.
| jedberg wrote:
| The reason a lot of companies charge extra for SSO is because
| it is a support nightmare. Most users can't tell the difference
| between "my SSO provider is broken" and "your service is
| broken" so they always blame the service, who now has to either
| tell the customer "talk to your SSO provider", which of course
| they don't like, or having to diagnose and fix SSO problems.
|
| I agree that it should be a core feature, but I sympathize with
| companies that need to charge extra for it.
| krashidov wrote:
| I wish this website had a wall of fame on it as well
| rubenfiszel wrote:
| I don't disagree with what you are saying. It's just that
| everything is already free and as friction-less as possible. I
| made a point to remove any lock-in. My goal is to make Windmill
| fully-featured and available to the widest range of people. On
| the other hand, we do need to make some money at some point. We
| could have an "enterprise edition" with features that are
| proprietary. This is what most open-source companies do, but it
| would create misaligned incentives to not bake those into the
| product and make the product worse. If I am being honest I will
| probably bake SSO into the team-tier as soon as we have some
| healthy financials :)
| Saaster wrote:
| If I as a SaaS provider get my SSO SAML integration via a
| provider like Okta or Auth0, the auth provider pricing itself
| is also on a "call us" tier, with a per-federation pricing in
| the low four figures for each individual company connecting to
| me via SAML.
|
| It's pretty insane, so I'll state it again: To have a company
| connect to my SaaS via SAML, I as the SaaS provider have to pay
| my auth provider $X,000 per year for the privilege. Not
| counting the base enterprise tier pricing for the auth solution
| itself. So then I have to roll my own solution if I want to
| provide it for free, and I get the joy of supporting the long
| tail of broken SAML implementations on both the service and
| identity provider sides. For free. In a perfect world SSO
| wouldn't be a shitshow and everyone could have it for free,
| unfortunately that is not this world.
| apugoneappu wrote:
| From my limited knowledge of how SSO works, is this possible?
|
| 1. A company Acme wants to have SSO in the tools A, B and C
| that it uses.
|
| 2. Another company Balloon integrates with A, B, C to use the
| admin API for an admin account to modify or delete users in
| that account
|
| 3. Acme logs in to Balloon and connects its admin account of A,
| B, and C to these integrations.
|
| 4. Now Acme has access to employee's accounts in A, B, C
| through the Balloon's dashboard to modify or delete users etc.
| jon-wood wrote:
| Yes, it's typically referred to as SCIM and is supported by
| most serious SSO services.
| rubenfiszel wrote:
| I was not aware of this and will look it up. I was of the
| naive opinion of the parent thread.
| poulpi wrote:
| Very cool!
|
| I've seen too many companies with spaghetti integration built on
| Zapier / Make to orchestrate the user onboarding (CRM, database,
| Finance tool)
|
| At the beginning it's a good idea and at one point everything
| start to crash and you wonder how to put everything in a got
| repository!
|
| Windmill seems really cool
| rubenfiszel wrote:
| Thank you. I agree, everything starts very nice and it's all
| downhill from there with rigid tools.
| rubenfiszel wrote:
| It is back up, sorry for the inconvenience, should have rented
| more servers!
___________________________________________________________________
(page generated 2022-08-09 23:00 UTC)