[HN Gopher] Fast self-hostable open-source workflow engine
___________________________________________________________________
Fast self-hostable open-source workflow engine
Author : rubenfiszel
Score : 200 points
Date : 2023-11-22 18:33 UTC (4 hours ago)
(HTM) web link (www.windmill.dev)
(TXT) w3m dump (www.windmill.dev)
| zegoverno wrote:
| great product!
| slig wrote:
| I've been following Windmill since their HN Launch (my first
| script was edited 540 ago, just checked). Less than a year ago I
| started using it more heavily and it's been a breeze. Their
| Discord server is very active and Ruben usually replies and fixes
| bugs in minutes, even on weekends.
| thelastparadise wrote:
| What do you use it for?
| slig wrote:
| Daily puzzle generation, daily PDF updates/creation (if the
| original content is changed, it triggers a new generation),
| triggering and editing of Remotion videos.
| Kiro wrote:
| I don't think this should be a Show HN.
|
| > Off topic: blog posts
|
| https://news.ycombinator.com/showhn.html
| dang wrote:
| Thanks, I've taken Show HN out of the title now.
| rubenfiszel wrote:
| Curious, what's the difference between an article and a blog
| post ?
| pvg wrote:
| Articles generally can't be show hn's either so it doesn't
| matter much for the purposes of Show HN. The full thing is
| "Off topic: blog posts, sign-up pages, newsletters, lists,
| and other reading material. Those can't be tried out, so
| can't be Show HNs. Make a regular submission instead."
|
| https://news.ycombinator.com/showhn.html
| knowsuchagency wrote:
| Windmill is fantastic. It manages to cover so much ground i.e.
| airflow, prefect, retool, budibase but better
| nextworddev wrote:
| It's mainly for workflows but not for building rest endpoints?
| sisve wrote:
| I think the workflow engine is the _best_ part of it, but
| together with the script editor (online code editor with LSP
| support) and a drag-n-drop app builder, that is for
| developers, so you "drop down" to code much of the logic and
| connect it to script or workflow, it's a really nice combo.
| (you see a lot of other companies trying to span out to
| workflow + script + appbuilder).
|
| It's really made for developers or at least people that want
| code as a "first class citizen" of the platform. So i has a
| cli to sync to/from a VCS, VS code extention. But at the same
| time I have gotten a tech savvy person on the customer
| success team to create a workflow where they connect to our
| API and actually create meaningful services _way faster_ then
| the enterprise developer team would have done.
|
| The script and flows get automatically endpoints to connect
| to, but all request is saved to db, and picked up by a worker
| that has to save the result to disk, before the server again
| pick up the result and send it to a client. So I would not
| use it to customer. I have used it for a lot of internal
| tools to employees. Meaning they can wait 1-2 sec, because
| you can teach them that it works and you provide a lot of
| value.
| ryanjshaw wrote:
| > It's really made for developers or at least people that
| want code as a "first class citizen" of the platform.
|
| Power Automate doesn't have this (unless you hack it into
| custom connectors) and it's a really puzzling decision.
|
| I guess they want you to use Azure Functions but those need
| a PhD to understand and the deal with the security surface.
| hintymad wrote:
| Does being fast beyond certain threshold really matter for a
| workflow engine, though, especially given that many workflows
| have long-running tasks? What I find matters is multi-tenancy.
| That is, can a workflow system support as many jobs as a user
| desires, yet each job gets scheduled and executed as if it is the
| only one on the workflow engine. For instance, Amazon EC2 and EBS
| probably have millions of jobs and job queues running at any
| moment, yet each job has pretty predictable and stable startup
| delay and execution time. In contrast, each of Nomad and k8s has
| a global queue that sometimes gets stuck because of bad actors or
| too many pending jobs (I understand that an orchestrator is far
| more than just a workflow engine. I'm just contrasting the user
| experience).
| rubenfiszel wrote:
| So I agree that for most jobs taking already more than 300ms, a
| 50ms per job will not be perceivable anymore. What matters is
| the DX and even though I'm very proud of our performance, a lot
| of our work is focused on providing a consistently great
| developer experience in all aspects (vscode extension, instant
| deployment, etc, etc).
|
| However, there is one big benefit to being performant which is
| that you can use the same stack for performance sensitive jobs
| such as event-streaming use cases. That remove the duplication
| of infrastructure. The other benefit is that it improve the
| developer experience overall with fast previews.
| 3cats-in-a-coat wrote:
| A workflow engine is indistinguishable from a regular
| interpreter or runtime, except for the fact it's suspendable
| (hence can run tasks long).
|
| Ideally a workflow engine being fast means it can be just the
| platform you write everything in. The artificial separation
| between "normal" code and "workflow engines" is completely
| unnecessary. Our platforms are extremely fragmented,
| pointlessly so. Remote vs local, statically vs dynamically
| typed, async vs sync, short running vs long running...
|
| If an engine is written without these arbitrary limitations, it
| can be more.
| analyte123 wrote:
| Yes. Just think of how annoying it is when you have to wait 5
| seconds instead of 0.5s for a 2FA message, then multiply that
| by everything that you ever do with your workflow engine.
| That's not even to speak of cases where running the workflow
| (e.g. acquiring data) faster is a competitive advantage,
| although this thing is still probably too slow for truly HFT-
| level tasks.
| rubenfiszel wrote:
| Yes it's too slow for HFT, but anything would be too slow for
| HFT unless it's custom built. You likely want a streaming
| engine or even hand-optimize all your event handlers to
| shave-off any nanoseconds.
| samsquire wrote:
| Just because I'm interested in this topic. I've been trying
| to shave nanoseconds on ringbuffers and barriers in C.
|
| LMAX Disruptor can send a message between threads in 53
| nanoseconds on average.
|
| So you can essentially fork control flow in 53 nanoseconds
| and do 2 things in parallel.
|
| Eyeballing my barrier I can synchronize in 42 nanoseconds
| and up.
| 0xbadcafebee wrote:
| > we wanted to prove it's also the best system to deploy at scale
| in production
|
| > The queue is implemented in Postgresql itself
|
| Tell me you don't understand system design without telling me you
| don't understand system design
| dang wrote:
| " _Please don 't post shallow dismissals, especially of other
| people's work. A good critical comment teaches us something._"
|
| If you know more than others, that's great, but in that case
| please share some of what you know--without putdowns or snark--
| so the rest of us can learn.
|
| https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...
|
| https://news.ycombinator.com/newsguidelines.html
| ryanjshaw wrote:
| I would argue that implementing your internal queues inside
| your DB is _sound_ system design. Your backups and system
| restores are as simple as it gets. You should have a specific
| reason to do anything more complicated. And no, "deploy at
| scale" is puffery, not a good reason.
| igortg wrote:
| Any known issue on implementing a queue with Postgres? Do you
| at least know it has a pretty good pub/sub mechanism?
| mmcclure wrote:
| Postgres is having a moment with excellent queueing libraries
| right now, not sure what GP is referring to. We use Oban[1]
| at Mux for some workloads and love it, and a former colleague
| just built something similar in Go that was on the front page
| yesterday (River)[2].
|
| [1] https://github.com/sorentwo/oban
|
| [2] https://github.com/riverqueue/river
| boxed wrote:
| It always amazes me that people spend this much time and effort
| on writing articles like this and don't run their text through a
| spell checker even once.
|
| It also amazes me that people still use text editors that by
| default doesn't spellcheck. Anyone know what editors that could
| be? Seems pretty crazy in 2023!
| andrewstuart2 wrote:
| If this is an engineer, they probably use whatever tool they're
| most familiar with, and that tool is probably set up for mostly
| engineering (code) work, in which case in my experience spell
| checking is a major distraction with an outsized ratio of false
| positives, because my identifiers aren't chosen for their
| validity in a wordlist. Even in markdown, it's more annoying
| than helpful most of the time, so I tend to leave it as a
| manual step that I mostly ignore. People know what I mean
| 99.99% of the time. So, that'd be my guess. Any sort of editor
| that's been tuned to an engineer's preferences which includes
| only manual spell checking.
| j_maffe wrote:
| Yeah, but this sort of article is what you get at the end of
| the day. Plus there are several ways of getting rid of false
| positives and as GP said: at least do it before posting.
| boxed wrote:
| PyCharm has a spell checker. It's pretty great. Not using it
| seems madness.
| code_biologist wrote:
| I turn off spell check in any text editor I touch because I
| can't handle the visual noise. Agreed on running spell check.
| The few grammar issues in the article made my brain glitch for
| sure.
| boxed wrote:
| What noise? The noise of you making tons of spelling errors?
| rubenfiszel wrote:
| You're right, I apologize. Truth is I wrote everything in
| vscode markdown and really wanted to finish the article before
| going out for dinner and I ended up skipping it. We fixed
| everything
| cooperaustinj wrote:
| The Code Spell Checker extension is great. It has proper
| handling for camelcase and it's fast to add words to the
| dictionary (cmd + .). Catches many typos when coding.
|
| Probably not best for the last line of defence for public
| articles, but probably good enough.
|
| https://marketplace.visualstudio.com/items?itemName=streetsi.
| ..
| victorbjorklund wrote:
| Looks very similar to the UI of Pipedream that I used before. Not
| a negative thing. Will try it.
| slig wrote:
| I migrated from Pipedream, Windmill is much, much more
| developer friendly.
| gigatexal wrote:
| Hi Windmill devs -- why build this on relational Postgres at all?
| I've been toying with the idea of forking airflow and trying to
| swap out the backend for something document based either on Mongo
| or something else.
|
| The reasons being db migrations are a pain in the relational
| strongly typed schema world vs the more forgiving paradigm of
| documents.
| cyanydeez wrote:
| what'd be wrong just storing your documents in postgres JsoN Or
| XML fields
| DonnyV wrote:
| Because Postgres is a relational database.
| cyanydeez wrote:
| that wasn't the concern being presented. The concern is "I
| want to store mostly unknown structured data, so why dont I
| use a DB already stupd to do that"
| gigatexal wrote:
| That could work too I guess.
| rubenfiszel wrote:
| because 95% of the tables you need for this are relational
| (queue, completed_job, users, etc, etc) and for the few things
| that are unstructured, you can represent them well in PG with
| JSONB (which is what we do for inputs and outputs).
| code_biologist wrote:
| Windmill is great. Definitely don't stray from the self-hostable
| + DX mission! I haven't had need to use it professionally, but I
| use it on a home server to run several small web crawlers and yt-
| dlp jobs. Really fun piece of tooling.
| darkteflon wrote:
| Could you clarify what you mean by "definitely don't stray from
| the self-hostable + DX mission"?
| code_biologist wrote:
| I hate cron for running small personal scripts and jobs. I
| hate ad-hoc handling of logging / observability / success
| monitoring. I want a web UI for all of it. I hate trying to
| figure out how to handle event based job triggers that are
| more fancy than the 10 line bash script they end up running.
| I hate trying to wrap up solutions for all those into
| infrastructure as code -- I will end up running an unpatched
| Ubuntu 20 for the next decade if I can't just nuke the box
| and reload my config + scripts. I hate the idea of relying on
| the continued existence of a hosted service, as well as them
| being able to see that I'm fine tuning an LLM for erotic
| stories (or whatever).
|
| Windmill solved all that for me. I write a short little
| Python script, paste it into the web UI and boom it's running
| with good answers for all the above issues. That's great DX.
|
| If workflow engines try to solve every problem they can
| devolve into plugin and complexity hell. Jenkins issues have
| caused a lot of headache at my day job lately. I don't want
| Windmill to fall into the same trap.
| darkteflon wrote:
| Thanks, that's really helpful - agreed on all accounts!
| DonnyV wrote:
| This looks interesting. I wonder if they will be adding compiled
| languages like C#?
| CuriousSkeptic wrote:
| Might want to checkout DurableTasks[1] for that
|
| [1] https://github.com/Azure/durabletask
| rubenfiszel wrote:
| you can always run compiled programs from bash as binaries (but
| yeah it's not as great as an experience than being able to run
| interpreted languages directly).
| darkteflon wrote:
| While appreciating that they're not direct substitutes, does
| anyone have experience comparing Windmill with Dagster for
| managing dependency resolution in complex data pipelines?
| bafe wrote:
| I briefly tried windmill for a project that involved creating
| custom workflows on-demand from a configuration file. I can't
| recommend windmill for data pipelines, it is meant to be more
| of a low-code internal app platform like retool or budibase. It
| is meant for a relatively static workflow that requires human
| intervention, like a simple business process involving some API
| calls and humans approvals in the loop. For complex (and
| potentially reconfigurable) data pipelines, dragster is a much
| better choice
| rubenfiszel wrote:
| When you tried, we didn't have s3 integrations nor
| restartable flows. We will present all of this in day 5 of
| our launch week and it might be a good time to revisit.
|
| I agree we were not a good fit prior. I think we would now
| compare favorably as we will offer excellent ergonomics for
| data processing, leveraging polars, duckdb, and other OLAP
| libraries to their full extent.
| bafe wrote:
| It was several months ago, so it is entirely possible. Back
| then I got the feeling that windmill tried to be more of a
| low code business/internal tool platform than a data/ETL
| workflow tool, I especially missed an expressive way to
| define workflows programmatically (i think you had a JSON
| schema but nothing as powerful as dragster where you can
| define a whole workflow in pure python)
| rubenfiszel wrote:
| We added a vscode extension to build everything from
| code, including manipulating the yaml or UI and have them
| instantly impact each other:
| https://www.windmill.dev/blog/launch-week-1/vscode-
| extension
|
| yaml is not real code and so the part that I consider to
| be real code is that each step is its separate file in
| python, typescript, that you can edit in your code editor
| and have your plugins working, testing frameworks. It's
| normal functions that you can run locally.
|
| It would be possible for us to do like
| dagster/prefect/airflow which is to use a macro-
| processing step/decorators to build dynamically the graph
| which is what our yaml is in the end, a 1:1 encoding of
| our dag spec called openflow:
| https://docs.windmill.dev/docs/openflow/
|
| We didn't do it yet because in most cases, the decorators
| are a lot like yaml, they are a very rigid way of
| declaring that some functions are nodes that you can
| chain in limited ways. On the other hand, not providing
| that mode allow us to put more efforts in the low-code
| graph builder for now.
|
| But, as someone that love compilers and built a few, I'm
| very eager for us to provide such a mode so it's probably
| a few months away :)
|
| We'd love to have your input on the DX for data/ETL once
| we present it Friday so feel free to join our discord or
| shoot me an email at ruben@windmill.dev
| bafe wrote:
| Thank you for the extensive reply! The first part you
| mentioned is exactly what we were missing back then, we
| wanted to dynamically generate workflows starting from a
| configuration selected by the user. This wasn't really
| possible unless we would generate the YAML openflow
| specification ourselves. At the end we gave up and rolled
| our own simple tool that just does what little we need.
| This said, it is cool that you are considering offering a
| more code-friendly way to define workflows. I still think
| this doesn't offer the same level of dynamism of
| dragster, where you could easily design
| branching/conditional workflows. I suppose your
| considerations regarding the decorators/compilers go
| exactly in that direction
| darkteflon wrote:
| Thanks Ruben - looking forward to the Dagster comparison
| post in the coming days.
| darkteflon wrote:
| Thanks - great to hear your perspective.
| jgil wrote:
| What are the units for the x-axis? I'm assuming milliseconds from
| context, but not having the axis labeled obfuscates the
| interpretation.
| rubenfiszel wrote:
| It's seconds. You can find all the detailed numbers there:
| https://www.windmill.dev/docs/misc/benchmarks/competitors
| haroldp wrote:
| Is this what I need? We have ad hoc business processes like:
|
| 1. Sell service to client
|
| 2. Email client with scheduling info
|
| 3. Agree to scheduling date with client
|
| 4. Send client email with documentation and request their data
|
| 5. Remind client to send data
|
| 6. Alert admins data hasn't been received
|
| 7. Beat data out of client
|
| 8. Prepare data
|
| 9. QA and checkoff service
|
| 10. Email client to review and approve
|
| 11. Publish/run service
|
| 12. Send client stats/results
|
| And I want to move that from spreadsheets, personal emails and
| admins keeping in their head where everything is at, to web forms
| and uploads, automated emails and dashboards. I looked at
| airtable, smartsheet, budibase and many others but they seem to
| be very project-based, where you are inventing a unique workflow
| for each project and using it to keep managers on top of where
| their team is at with the various steps. Project Management vs
| process management. None of them seem to have decent calendar
| integration. Few of them seem to be great about emails or
| scheduled scripts.
|
| I have APIs for my data, or I can code them if needed. I'd prefer
| a low-code to a no-code approach, where managers have a
| spreadsheet view and can do some of of the UI work and
| programmers can make it do stuff and handle integrations.
| goodpoint wrote:
| It is not.
| erichocean wrote:
| I don't think that'll achieve your goals, it's far too clunky
| and inflexible for actually running a business end-to-end. (No
| knock on Windmill, they're all like this in my experience as
| the way they model the world is how a programmer thinks about
| things, not a business owner. BPMN is straight up cancer--
| avoid.)
|
| I would suggest looking at workflow systems within the VFX
| space, which they call "pipelines". They are human-driven and
| maintained, but support very high levels of process automation.
|
| VFX pipelines are written in Python, but the key thing is,
| they're very good at a few things simultaneously:
|
| (1) Highly irregular work...
|
| (2) That nevertheless requires lots of process, including of
| the ad hoc variety...
|
| (3) With high human touch (human in the loop), so calendars,
| task assignments, timelines, approvals, producer dashboards,
| etc.
|
| (4) That's repeatable on the next project/opportunity, with as
| much customization as needed.
|
| As a bonus, pipelines handle billions of dollars of work
| annually under high pressure, with tens of thousands of users,
| so there's a lot of hard won experience built into them.
|
| I personally have found that there are three levels of VFX
| pipeline tech that need to be combined and customized to do
| everything. This is the most recent stack I've set up:
|
| * Kitsu for the producer/manager level, human task assignments.
| Mainly for managers, has the calendar functionality.
|
| * Prism Pipeline for software integration (i.e. keeping the
| humans following the process). This is what people actually
| _do_.
|
| * Kabaret for compute work. People usually kick off these jobs,
| but they're handled by the farm. You'll be responsible for
| everything that happens here. [0]
|
| Don't try to get by with just one or two of the three, even
| though feature-wise it looks like you might be able to.
| Incorporating computer-driven workflows is tricky and requires
| purpose-built libraries in my experience. Just use the right
| library for the job, and accept that they all have somewhat
| overlapping functionality.
|
| For most non-VFX work, you'll need to script access to the
| browser, since that's where people do stuff today (i.e. with
| the Prism Pipeline library). Launch Chrome, Firefox, or Edge
| with a debug port and use Playwright as your "plugin"
| mechanism, with Prism/Python as the driver. You can make your
| pages do whatever you want and script literally any web-based
| process, filling in fields from your database as needed, adding
| data to the page, etc.
|
| For the last six years, I've run a business that does around
| $5M of annual revenue using this approach, with 4 FTE people
| (including me). We handle around 40 million compute tasks a
| day, but also have all of the "daily work" where humans are in
| the loop. We do all of the tasks of the type you've listed.
|
| HTH!
|
| [0] If you need fully automated task generation (so, no human
| in the loop), write "pump" scripts in plain Python that push
| jobs into Kitsu or Kabaret at a particular interval. I
| personally run them under a god monitor, and have them exit
| after, say, 15 minutes (or whatever interval makes sense) to
| get cron-like functionality. This is useful because you can
| pause execution of any pump process easily, change the
| interval, restart them, etc. plus you get easy access to the
| logs for debugging. The scripts themselves are usually less
| than 100 LOC. I have around 60 of these that run the automatic
| parts of our business.
|
| Whenever I need custom, temporary behavior in this part of the
| business, I copy the pump script, edit the original to avoid
| the special case, and have the new script only handle the
| special case (with whatever new behavior I need). Done, takes
| very little time, and is easy to run temporarily (e.g. for the
| next week, a common occurrence because some downstream vendor
| has caused a problem we need to work around).
|
| Another approach to trigger tasks/workflows is to have a server
| that gets pinged by 3rd parties (e.g. Webhooks) and does the
| same whenever the endpoint is invoked. We have that too, but
| it's literally just a few scripts. (Most workflow systems seem
| to believe this is damn near 100% of the stuff that needs to be
| "automated.")
| sisve wrote:
| Searching for "Vfx pipeline" just gave me visual effects
| stuff. Quite sure that was not what you talked about. Could
| you explain a bit more what ypu mean by it and point to a
| resource. Keen to learn more.
|
| I think windmill could have helped with several of the task
| and automated some. Its very easy to create dashboard and
| table with statuses. But its def not a process tool, if that
| was what he was after.
| Zalastax wrote:
| I would also like to understand better. But it seems like
| it is visual effects that is meant. These tools were
| mentioned by name: https://www.kabaretstudio.com/
| https://prism-pipeline.com/
| erichocean wrote:
| Sorry, yes, VFX is "visual effects".
|
| A sibling already gave links to Kabaret and Prism.
|
| Here's the link to Kitsu: https://github.com/cgwire/kitsu
| haroldp wrote:
| Thank you for the detailed reply.
| bafe wrote:
| Looks like the use case for BPMN and an engine that can execute
| processes described in that notation.
| michaelmcmillan wrote:
| Same here. Use n8n. Self hosted and open source.
| jwildeboer wrote:
| It's not open source.
| rubenfiszel wrote:
| Yes, windmill would likely be a great fit for that as long as
| you're willing to do a little bit of coding.
|
| For instance, we do not have a calendar integration per se but
| can manage your oauth tokens towards any service (including
| gcalendar) and allow you to write code using their official
| sdks.
|
| For spreadsheets, you can build a similar interface in windmill
| using aggrid.
|
| It seems you also need approval steps and have your workflows
| wait for events, and that's built-in in our workflows.
| MuffinFlavored wrote:
| I wonder how far you'd get with the very "not technical"
| https://getjobber.com/
|
| Only because you said "sell service to client", schedule date
| with client, etc.
| j_maffe wrote:
| That graph is utterly useless. How hard is it to place units on a
| freaking graph?
| rubenfiszel wrote:
| Apologies, the units are seconds, fixing the graph right now.
| You will find more ample details on the benchmarks there:
| https://www.windmill.dev/docs/misc/benchmarks/competitors
|
| We made them as easy as possible to be reproduced
| alexchamberlain wrote:
| Interesting that Windmill's backend seems to be implemented in
| Rust, but the docs talk about supporting jobs written in Python,
| TypeScript and Go. I understand there is generic Docker support,
| but does anyone know if you can run Rust jobs natively?
| sisve wrote:
| It can not. (It does support go)
| rubenfiszel wrote:
| We could add support for Rust, and it would work the same as
| go, do a built step prior, and then cache the binary. If we
| have enough demand for it, we will add it (we are a small team
| and can only prioritize what's popular unfortunately).
|
| On a personal note, I write a lot of rust (and love it) but
| using rust for writing non-backend type of stuff is not
| something I would do myself. I'm more of a believer of hybrid
| like polars that optimize the compute in Rust but provide a
| compatible sdk in python.
| paxys wrote:
| The devs of Windmill seem to have taken the winning advice of "do
| one thing and do it well" and done the exact opposite. Going over
| windmill.dev I have no idea what specifically the software should
| be used for. Is it a competitor to Retool? Airflow? Temporal?
| Apparently there's a no-code workflow builder? A drag and drop UI
| builder? An online IDE? Dozens of integrations? What on earth is
| going on??
| debarshri wrote:
| Doing one thing well is not a winning advice. You can do
| multiple things, it really depends on the execution and other
| factors. They are doing one that is workflows and doing it
| well.
| sisve wrote:
| Also they are using postgres for everything. I think that is
| a good example of picking one thing that you know very well
| and using that to the limits.
|
| But i do agree that windmill is doing so many things that the
| elevator pitch need some work. But a lot of companies are
| trying to do the same. So i do not think people will think of
| workflow engine + online ide + app builder as a natural
| grouping in 5 years
| MuffinFlavored wrote:
| I don't really understand what a "workflow" is / what it
| means.
|
| Like GitHub Actions is a workflow, but it's a convoluted YAML
| wrapper around shell scripts.
|
| Is a workflow shell scripts? I'm going to guess no. I'm going
| to guess it's "code". So... why is a workflow code but code
| isn't a workflow?
|
| Windmill is a "workflow (code?)" orchestrator?
| jagtstronaut wrote:
| Yeah I assumed this was a camunda competitor (a bpmn
| workflow product), but it seems like it is not that.
| uoaei wrote:
| One task is the elementary unit of business operations, and
| one workflow is one DAG of tasks.
| Bonapara wrote:
| Looks like a workflow engine to me
| rubenfiszel wrote:
| The critics is more than fair and our positioning is not very
| clear.
|
| We are doing a developer platform for enterprise to build
| internal software, including apis, workflows, background jobs,
| and UIs using code, but only where it matters. We happen to be
| quite decent in all aspects, with a focus on performance, and
| we have an active community of users and customers that are
| mostly developers and hence have plethora of feedback and
| feature requests, so we're happy to oblige.
|
| By being at the right level and trying to expose the code as
| much as possible, we can be great generalists without
| sacrificing on the quality. Python, Typescripts, Go, Bash and
| all the query languages are where all the complexity is and
| capabilities come from. They are well developed languages with
| great ecosystem. We merely provide a way to run those language
| in a system fit for production at enterprise scale.
| MuffinFlavored wrote:
| > We are doing a developer platform for enterprise to build
| internal software
|
| > for enterprise
|
| I mentioned this product to my enterprise boss and his
| enterprise answer was "we can't easily get this approved by
| tech-selection committee for no good reason" (and that would
| have just been to have an off-to-the-side proof-of-concept
| instance or two running in our sandbox k8s cluster)
|
| How do enterprise users not already have the functionality
| you are offering covered one way or another and why would
| they migrate off of what they have to what you are offering?
| What kind of business today doesn't have something like
| ActiveBatch, AWS/Azure functionality with massive cloud
| support contracts, etc. already?
| rubenfiszel wrote:
| You can't win every contracts but the market we're after is
| immense and so even a small percentage of enterprises being
| more daring is sufficient.
|
| But the argument is true for any enterprise software, you
| cannot be small and sell to enterprise (or you must have
| amazing product market fit), that's where open-source comes
| in.
|
| We have overlap with many existing products, but we are the
| only one to provide such product, with emphasis on DX and
| excellent performance and scalability, and to be open-
| source. We do currently have a few big enterprise customers
| and many enterprise open-source users. It's a lot easier to
| get approval since they get it for free and it's fully
| self-hostable and air-grappable. Once they have tried it on
| a few non-essential workflow and see the benefits, then
| they are more motivated to make a case internally. Wide
| net, some catches until it becomes ubiquitous.
|
| I happen to be a really strong believer in open-source so
| it's not just an adoption strategy but I think it happens
| to be also fortunately the best strategy for such infra
| level software.
| Mandatum wrote:
| > but only where it matters.
|
| What does this mean?
| rubenfiszel wrote:
| Happy to expand on this. Only where you would consider to
| be not boilerplate, aka your business logic. I hate low-
| code that force me to use a rigid system for the core of my
| tools, but I love the velocity that it brings. What if we
| could have high-velocity for all the boilerplate
| (permissioning, queuing, frontend, secret management, etc,
| etc) but code for the business logic. That's what windmill
| is. In a way, this is what serverless was meant to be,
| write your business logic and servers autoscale and you can
| hook a gazillon managed services on top.
|
| We bring that vision in a fully open-source and consistent
| platform that you can host wherever you please.
| MuffinFlavored wrote:
| > Is it a competitor to Retool? Airflow? Temporal?
|
| If it makes you feel any better, I'm pretty sure I personally
| couldn't tell you specifically what any of those specifically
| do in terms of "the one thing they do well" / why anybody would
| ever use one instead of the other / not need to use all 3 to
| accomplish the same thing slightly different ways
| knowsuchagency wrote:
| I had the same thought when it popped up on my radar a year
| ago. Now that I've been using it for a few weeks, it's
| difficult to go back to the exact tools you're naming. I didn't
| realize how large of an impediment it is to move back and forth
| all of them. Windmill is the thing I never knew I needed. It's
| changed how I think about delivering data products/solutions.
| punnerud wrote:
| The GitHub repo for the open source project gives a better
| intro/sell: https://github.com/windmill-labs/windmill (Especially
| the video)
| majkinetor wrote:
| I am using Rundeck now to schedule project jobs, get notified on
| errors, provide UI for PowerShells/bash scripts, and have decent
| project and user management. It uses Quartz and can handle
| hundreds of jobs per minute without a problem. It can connect to
| git repository to keep job definitions there. I also use local
| file system a lot (on Windows or Linux) as I don't like to edit
| my scripts in browser, given that they are mini programs with
| lots of separate includes.
|
| I would like to try windmill instead. Can it cover all those
| cases, particularly the part where scripts are on the file
| system? Can I use automatic GUI creation for PowerShell scripts
| of windmill in such case? Rundeck transmits environment variables
| with GUI values when starting script.
| rubenfiszel wrote:
| Not only it can cover those cases, you will likely be pleased
| with the advantages of such system (the automatic GUI creation
| is just one of those).
|
| The binary itself doesn't sandbox your processes unless you're
| in the nsjail mode. Hence, if you either use the binary raw
| (without docker) or mount the filesystem to be available to
| your container, it can run anything that is available on the
| filesystem.
| atlas_hugged wrote:
| It says open source, then it says limit of 10 SSO users. IANAL so
| this confuses me. Can you explain what this means?
|
| Actually, that sounds rude, excuse the 'tism, I'm sorry. I'm just
| not super familiar with the license stuff, I generally only touch
| MIT licensed code when it's work related, but I thought open
| source generally allows modification to code. So how do you
| enforce a limit of 10? I just skimmed the code a bit because of
| this confusion and I see stuff about enforcing a license in
| there. So couldn't anyone just remove that license check code if
| it's open-source?
|
| If that can't be modified, then it's just "source available",
| isn't it? Which to me is also fine, I just feel like I'm being
| either misled, if you get what I'm saying?
|
| Basically I think this project is pretty cool, so I wanted to
| bring it up to my boss because I think we have a need for this,
| and I had previously brought up Airflow, but I don't know how to
| explain this to her.
|
| Sorry for the stream of consciousness. Typing on mobile while
| answering Teams messages and tickets.
| rubenfiszel wrote:
| Some people will argue we are open-core because almost
| everything is open-source (AGPL) but our sso and enterprise
| plugins are under a source-available proprietary license.
|
| The reason that we didn't split them out of the codebase is
| that it's harder to maintain and would require to load the
| plugins. We didn't want to waste time on that when we had so
| much to build.
| atlas_hugged wrote:
| oh ok I see. Just the SSO part is like that. That makes more
| sense.
|
| So I guess I can just self-host for our team of 7 without SSO
| and we can finally organize these messy cron jobs we've
| accumulated. We would not be reselling at all. Just
| organizing some annoying aspects of our day to day.
|
| Thanks for the clarification. Much appreciated. If we end up
| expanding our usage down the road, I'll see if I can convince
| her to consider shelling out funds for the SSO stuff too. I'd
| love to be able to support this project. Seems pretty cool!
| rubenfiszel wrote:
| You can even self-host with SSO for 7 people. The SSO is
| free up to 10ppl.
|
| You will never have to pay if you do not want to.
|
| We believe there are many reasons to start using Windmill
| and most of them are not worth monetizing by themselves but
| we strive to build a software so great that you will move
| more and more stuff on there and at that point, you will
| want our enterprise plugins.
|
| We also mostly monetize bigger companies.
| jagged-chisel wrote:
| I would hazard a guess that the SSO part is not open source and
| could be handled by a non-open plug-in. I should go read the
| code ...
|
| Edit: sniped by someone in-the-know
| vlugorilla wrote:
| Been using windmill for a few months, and it's very pleasing. One
| of the best self-hosted tools I have!
| MuffinFlavored wrote:
| https://www.windmill.dev/docs/misc/benchmarks/aws_lambda
|
| I did not realize at first glance you can write a script and then
| trigger it (with an API key, not sure if you can make a public
| one or if that's a security nightmare) through HTTP to basically
| have your own self-hosted AWS Lambda / the whole "serverless HTTP
| triggers/functions" craze
| rubenfiszel wrote:
| Yes absolutely, we're even faster than lambda with dedicated
| workers since the design is simpler, while loop jobs from the
| queue while still keeping all the error handling,
| observability, and easy deployment.
| MuffinFlavored wrote:
| Is there a workaround I'm missing where the URLs are exposed
| without needing an API key? (Something I'm pretty sure Lambda
| allows if you configure it properly)
| iFire wrote:
| LICENSE ???
|
| https://github.com/windmill-labs/windmill/blob/main/LICENSE
|
| I'm a bit confused.
| rubenfiszel wrote:
| We use a mix of AGPL for 95% of the code, Apache 2.0 for specs
| definitions and apis, and source-available for our enterprise
| plugins.
| SpaghettiCthulu wrote:
| What on Earth is confusing about that?
| brunoqc wrote:
| It's open-core (if you care about that).
|
| "The SSO is free up to 10ppl."
|
| It's weird that "Content Search" is such a pro feature.
| rubenfiszel wrote:
| Only if you deem SSO to be of an essential nature to such
| products. To me it's all semantics at this point.
|
| But it's a complex software that require software engineers
| with salaries and we do not want to rely on donations so we do
| have to give reasons for people to pay. SSO and audit logs are
| good ways to segment enterprises to support the project.
___________________________________________________________________
(page generated 2023-11-22 23:00 UTC)