[HN Gopher] Argo Workflows v3.0
___________________________________________________________________
Argo Workflows v3.0
Author : dnsmichi
Score : 44 points
Date : 2021-02-01 17:47 UTC (5 hours ago)
(HTM) web link (blog.argoproj.io)
(TXT) w3m dump (blog.argoproj.io)
| rektide wrote:
| I mainly became familiar with Argo Workflows when looking
| at/lightly evaluating Argo Events to run my home-(k8s)-cluster's
| Event Driven Architecture (as an esb/lambda-architecture/event-
| fabric/&c system). Also poked at Brigade and Tekton (who sadly
| are now CI/CD focused, not general EDA) and local Github Actions
| runners.
|
| Workflow was mentioned in the Argo Events docs a couple times.
| Events is a pretty general system, but if you need to assemble
| more complex inter-dependent tasks, they had Workflow
| integration. I didn't get super far in, look very deep at how the
| integration worked, or how pleased I was going to be using
| Workflows, as I was mainly interested in the EDA end to start.
|
| Anyhow, picking through Workflow 3.0, my main interest is:
|
| > Argo Workflows v3.0 comes with a new UI that now also supports
| Argo Events! The UI is also more robust and reliable.
|
| Really seems like Argo Workflow has been made the over-arching UI
| for both of these systems in this 3.0 release. There's now some
| event-flow pages in Workflow 3.0, that will be interesting to
| check out. I'm very interested in how the APIs have all been
| updated, how the systems work together. Especially given Argo
| Event's fall 1.0 release & the new architecture[1].
|
| One other cool feature from the Workflow 3.0 is Widgets. Embed
| some code in other pages so you can see build status, job status,
| &c.
|
| [1] https://blog.argoproj.io/argo-
| events-v1-0-released-b69668de0...
| halfmatthalfcat wrote:
| Speaking as someone whose spent the last 18 months building out
| Tekton pipelines, I can confidently disagree that it's
| architecture (or general ethos) is built around CI/CD. The docs
| might allude to a lot of CI/CD use cases but the framework is
| very general in that you have the building blocks for any event
| driven workflow: Tasks, Pipelines and Events.
| rektide wrote:
| Tekton was very high in my runnings, very attractive. I liked
| the models they built around, they seemed like a good
| structure. Everything built very "cloud-native" (system state
| stored as Kubernetes objects).
|
| But the project did explicitly pivot, sometime in the last 12
| months, to explicitly declare themselves/think of themselves
| as a CI/CD system. They're also under the Continuous Delivery
| Foundation now. I spent 5 minutes looking for the specific
| event, where this was determined/declared, & will try to
| follow up; hope I find it.
|
| I don't think it precludes Tekton from still being used more
| generally, thought of more generally. But now, doing so puts
| one out of alignment with the stated goals, with the project
| itself. I'm quite sad about this development: I think it
| radically undershoots the relevance of Tekton & what role it
| ought to fill. We have all this wonderful new cloud based
| pieces of state, but in terms of how one piece actuates
| another, how we can make systems that behave autonomically:
| we direly need some core Event Driven Architecture logic, and
| Tekton definitely has some really good tools for doing that.
| I want very much for Tekton to meet it's potential head on.
| EQVEYWDCHQ wrote:
| Why would I use this over airflow?
| theptip wrote:
| Here's a conference talk that gives a detailed side-by-side
| comparison: https://youtu.be/oXPgX7G_eow
|
| I'm strongly considering moving my fairly immature Airflow
| pipeline to Argo Workflows because:
|
| * the Airflow DAG deploy/versioning is surprisingly primitive.
| The best option here seems to be to use the KubernetesOperator
| to version your steps, and if you're using k8s to execute, why
| not use it for the rest?
|
| * the Airflow UI is pretty confusing to use, maybe this gets
| easier once you know your way around it.
|
| * my team has k8s expertise and we don't know Airflow well yet;
| seems like less to learn running Argo Workflows, assuming
| you're already fluent in k8s.
|
| * if you're already running k8s, it seems like you have to add
| fewer components to get Argo running; more duplication with
| Airflow-on-k8s.
|
| On the other hand, being able to unit test / locally run your
| DAGs on your dev machine is a big plus for Airflow, where Argo
| Workflows seem to have a less strong testing story. And writing
| YAML is not preferable to writing Python DAG files.
| dimberman wrote:
| Hi @theptip,
|
| I'm an Airflow PMC and would love to know a bit more about
| your comparison :).
|
| 1. Have you tried Airflow 2.0? We made some pretty big
| overhauls both in terms of UI and backend. 2. DAG versioning
| is currently problematic, but DAG versioning is a "when" and
| not an "if" so should be in a future 2.x version :). That
| said could you describe a bit more about your deployment
| issues? User stories like this help us improve the product.
| 3. Have you looked into using KEDA with the CeleryExecutor?
| You could create KEDA queues for a lot of commonly used
| workflows and then you'd only need to use the python or bash
| operator to run those tasks instead of k8spodop. 4. Are you
| using the Airflow helm chart or did you custom roll a
| deployment?
|
| Any feedback would be highly appreciated and I'm also glad to
| answer any questions you might have!
| ellisv wrote:
| and I'm sitting where wondering, why would you use Airflow over
| Argo Workflows?
|
| Everyone should evaluate the options for their own needs.
| jw887c wrote:
| If you prefer yaml > Python or if you prefer installing one k8s
| app instead of managing all of the airflow dependencies
| (scheduler, webserver, workers, etc)
___________________________________________________________________
(page generated 2021-02-01 23:02 UTC)