[HN Gopher] The Unbundling of Airflow
       ___________________________________________________________________
        
       The Unbundling of Airflow
        
       Author : gorkemyurt
       Score  : 25 points
       Date   : 2022-02-15 19:32 UTC (3 hours ago)
        
 (HTM) web link (blog.fal.ai)
 (TXT) w3m dump (blog.fal.ai)
        
       | gorkemyurt wrote:
       | High Resolution Version of the diagram if anyone is interested
       | 
       | https://drive.google.com/file/d/1btZ0yck9SdgsUdNom0WXgHcSQvO...
        
       | DevKoala wrote:
       | I love Airflow. Plenty of data businesses I've built are nothing
       | more than just one DAG.
       | 
       | As for the article, I don't think we are yet at the point in
       | which a competing stack comprised of individual specialized
       | components do things better since Airflow is more than the sum of
       | its parts imho.
        
       | fdgsdfogijq wrote:
       | This post is hard to follow. But I'll give my unsolicited opinion
       | on airflow:
       | 
       | Its too complex to run as a single team and there are far better
       | tools out there for scheduling. Airflow only makes sense when you
       | need complex logic surrounding when to run jobs, how to backfill,
       | when to backfill, and complex dependency trees. Otherwise, you
       | are much better off with something like AWS step functions.
        
         | DevKoala wrote:
         | I run Airflow even for my local trading setup. For large teams,
         | I often go with managed solutions like Astronomer.
        
         | zukzuk wrote:
         | Everyone's context is different, but I've found the exact
         | opposite to be true. Airflow is simple and dumb enough that it
         | can be easily understood and managed by a small team, but it's
         | also flexible and powerful enough that we can't come up with a
         | good enough reason to switch to anything else.
        
         | samdjstephens wrote:
         | Interesting, I wouldn't say that I've found it difficult to run
         | in even a small team.
         | 
         | The problem I've always had with Airflow has been with non-
         | cron-like use cases, for example data pipelines kicked off when
         | some event occurs. Sensors were often an awkward fit and the
         | HTTP API was quite immature back when I was using it
        
         | ajoseps wrote:
         | do you have recommendations on alternatives that are not tied
         | to a cloud provider?
        
           | jonpon wrote:
           | We are trying to build something like this at
           | https://www.magniv.app/.
           | 
           | Would love to have you join our beta if you are interested!
        
       | fmakunbound wrote:
       | It's ok, but seems to be a bit too complex for what it does. It
       | was pretty janky running it locally (pegged the CPU), and now
       | that we have it in MWAA we've got several support issues on it
       | with AWS for unkillable task instances and scheduler problems.
        
       ___________________________________________________________________
       (page generated 2022-02-15 23:00 UTC)