Flyte 1.10: Self-hosted solution to build production-grade data and ML
pipelines; now ships with monorepo, new agents and sensors, eager
workflows and more 🚀 (4.1k stars on GitHub)
reddit / selfhosted

----------------------------------------------------------------------
allasamhita · 2023-10-31 13:06

[GitHub Repo](https://github.com/flyteorg/flyte)

Hello, I'm Samhita from Flyte, and I'm thrilled to share an exciting
update about our latest release!

Flyte is an open-source orchestrator that facilitates building
production-grade data and ML pipelines. It is built for scalability
and reproducibility, leveraging Kubernetes as its underlying platform.
With Flyte, user teams can construct pipelines using the Python SDK,
and seamlessly deploy them on both cloud and on-premises environments,
enabling distributed processing and efficient resource utilization.

## Monorepo

The backend development of Flyte has been transitioned to a monorepo.
This change will significantly enhance the contribution experience,
making it easier to test and merge changes into the backend code. In
the near future, a more detailed explanation of why we opted for this
monorepo structure, how we executed the migration, and what the
development experience will look like will be provided.

# New agents

Flyte 1.10 [agents](https://docs.flyte.org/projects/cookbook/en/latest
/auto_examples/development_lifecycle/agent_service.html) are not only
more performant than ever — we also support more of them, including
Airflow, Memverge, Snowflake, Databricks and sensors!

**Airflow**

The Airflow agent enables the smooth execution of Airflow tasks in the
Flyte workflow without requiring any code changes. All the Airflow
tasks will be executed on an Airflow agent (long-running server)
rather than launching a new pod for each task, significantly reducing
overhead.

Here's an example of an Airflow file sensor:

    from airflow.sensors.filesystem import FileSensor
    from flytekit import task, workflow


    @task
    def t1():
        print("flyte")


    @workflow
    def wf():
        sensor = FileSensor(task_id="id", filepath="/tmp/1234")
        sensor >> t1()


    if __name__ == "__main__":
        wf()

**File sensor**

Sensors are valuable for waiting for specific events to occur. You can
inherit the BaseSensor class to create a custom sensor in Flyte.
Here's an example of a file sensor that waits for a file to be
available:

    import asyncio
    from typing import Optional, TypeVar

    from flytekit import FlyteContextManager
    from flytekit.sensor.base_sensor import BaseSensor

    T = TypeVar("T")


    class FileSensor(BaseSensor):
        def __init__(self, name: str, config: Optional[T] = None,
**kwargs):
            super().__init__(name=name, sensor_config=config,
**kwargs)

        async def poke(self, path: str) -> bool:
            fs = FlyteContextManager.current_context().file_access.get
_filesystem_for_path(path, asynchronous=True)
            return await asyncio.to_thread(fs.exists, path)

# Eager workflows

Eager workflows enable the use of familiar Python constructs through
the `asyncio` API. Here's a simple eager workflow using the `@eager`
decorator.

    from flytekit import task, workflow
    from flytekit.experimental import eager


    @task
    def add_one(x: int) -> int:
        return x + 1


    @task
    def double(x: int) -> int:
        return x * 2


    @eager
    async def simple_eager_workflow(x: int) -> int:
        out = await add_one(x=x)
        if out < 0:
            return -1
        return await double(x=out)

**What can you do with eager workflows?**

* Operate on task and sub-workflow outputs
* Define Python conditionals 
* Define loops
* Invoke static workflows
* Nest eager subworkflows
* Catch exceptions

Read the full announcement [blog
post](https://flyte.org/blog/flyte-1-10-monorepo-new-agents-eager-
workflows-and-more) to discover everything that has been shipped in
our 1.10 release.

GitHub:
[https://github.com/flyteorg/flyte](https://github.com/flyteorg/flyte)

Website: [https://flyte.org/](https://flyte.org/)

Docs: [https://docs.flyte.org/](https://docs.flyte.org/)

Cheers!