[HN Gopher] Netflix has open-sourced its Maestro Workflow Orches...
___________________________________________________________________
Netflix has open-sourced its Maestro Workflow Orchestrator
Author : kaypee901
Score : 107 points
Date : 2024-07-22 18:21 UTC (4 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| vb-8448 wrote:
| This appears to be the high level architecture:
| https://miro.medium.com/v2/resize:fit:4800/format:webp/0*SDt....
|
| To me seems to overengineered for most of the companies outside
| there, how many people do you need to manage it?
| xyst wrote:
| In the blog, it claims:
|
| > ... schedule hundreds of thousands of workflows, millions of
| jobs every day and operate with a strict SLO of less than 1
| minute of scheduler introduced delay even when there are spikes
| in the traffic.
|
| I wouldn't necessarily call it over engineered if it can handle
| this type of workload.
|
| But if you are running a fraction of these workloads with a
| much more relaxed SLO (service level objection), then yea this
| is overengineered.
|
| Like any solution from FAANG, or "big tech". It works for
| _them_ at scale but for 90% of companies the sheer management
| of this service will quickly outweigh the benefits.
| galaxyLogic wrote:
| Over-engineering is in the eye of the beholder
| silisili wrote:
| > Like any solution from FAANG, or "big tech". It works for
| _them_ at scale but for 90% of companies the sheer management
| of this service will quickly outweigh the benefits.
|
| In other words, learn it now and get it on your resume,
| because within a year every 200-hit-per-day service and site
| is going to rearchitected to use it.
| nrclark wrote:
| That chart has real Krazam Microservices vibes to it.
| darknavi wrote:
| > First we call Bingo to get our Name-o.
| alex_lav wrote:
| All time great Youtube video. Still heavily relevant today.
| qntty wrote:
| I was hoping to try this out, but it doesn't build on my mac :/
|
| Here's their tech blog post about it:
| https://netflixtechblog.com/orchestrating-data-ml-workflows-...
| vips7L wrote:
| It doesn't build on windows either. It looks to be missing some
| dependency. > Did not resolve
| 'com.github.vmg.protogen:protogen-annotations:1.0.0' which is
| part of the dependency lock state
| qntty wrote:
| I get a similar error
| sitkack wrote:
| Use podman desktop, or is it an Arm issue?
| _joel wrote:
| I've been able to build and run it sucessfully via IntelliJ, on
| an M2 Air.
|
| Takes a few mins to get all the dependencies and build, but it
| definitely works.
|
| Perhaps you're missing some dependency steps before hand if
| you're hand cranking gradle.
| rickette wrote:
| I was just thinking didn't Netflix already have its own workflow
| engine: https://github.com/Netflix/conductor, but found out it's
| archived/EOL so this makes more sense now.
| opiniateddev wrote:
| Conductor was moved here: https://github.com/conductor-
| oss/conductor
| wg0 wrote:
| After almost demise of nomad, market needs another general
| purpose orchestrator that isn't Kubernetes or docker swarm.
|
| This is not it however.
| Keyframe wrote:
| So, is this something like prefect, dagster, flyte..?
| sa-code wrote:
| How could you forget airflow?
| ChrisArchitect wrote:
| [dupe]
|
| More discussion on official post:
| https://news.ycombinator.com/item?id=41037745
| opiniateddev wrote:
| An abstraction on top of Conductor:
| https://github.com/Netflix/maestro/blob/main/maestro-engine/...
| Thaxll wrote:
| Interesting that they use CockroachDB.
| pshirshov wrote:
| 1) Scheduling seems to be primitive? (strategy: sequential) 2)
| That's seems to be just a DAG executor? No high-level frontend?
| 3) No executiun context? 4) No typings? 5) No concept of a
| stream?
|
| It would be way too ambitious to call this thing an orchestrator,
| it seems to be just a primitive distributed DAG executor.
___________________________________________________________________
(page generated 2024-07-22 23:03 UTC)