[HN Gopher] Show HN: Pathway - Build Mission Critical ETL and RA...
___________________________________________________________________
Show HN: Pathway - Build Mission Critical ETL and RAG in Python
(NATO, F1 Used)
Hi HN data folks, I am excited to share Pathway, a Python data
processing framework we built for ETL and RAG pipelines.
https://github.com/pathwaycom/pathway We started Pathway to solve
event processing for IoT and geospatial indexing. Think freight
train operations in unmapped depots bringing key merchandise from
China to Europe. This was not something we could use Flink or
Elastic for. Then we added more connectors for streaming ETL
(Kafka, Postgres CDC...), data indexing (yay vectors!), and LLM
wrappers for RAG. Today Pathway provides a data indexing layer for
live data updates, stateless and stateful data transformations over
streams, and retrieval of structured and unstructured data.
Pathway ships with a Python API and a Rust runtime based on
Differential Dataflow to perform incremental computation. All the
pipeline is kept in memory and can be easily deployed with Docker
and Kubernetes (pipelines-as-code). We built Pathway to support
enterprises like F1 teams and NATO to build mission-critical data
pipelines. We do this by putting security and performance first.
For example, you can build and deploy self-hosted RAG pipelines
with local LLM models and Pathway's in-memory vector index, so no
data ever leaves your infrastructure. Pathway connectors and
transformations work with live data by default, so you can avoid
expensive reprocessing and rely on fresh data. You can install
Pathway with pip and Docker, and get started with templates and
notebooks: https://pathway.com/developers/showcases We also host
demo RAG pipelines implemented 100% in Pathway, feel free to
interact with their API endpoints:
https://pathway.com/solutions/rag-pipelines#try-it-out We'd love
to hear what you think of Pathway!
Author : janchorowski
Score : 50 points
Date : 2024-06-13 13:31 UTC (9 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| articsputnik wrote:
| Great job on Pathway. It's impressive to see a Python tool for
| ETL and RAG tasks with such strong features. The Python API and
| Rust runtime for quick updates look interesting. Focusing on
| security and performance, especially with self-hosted RAG
| pipelines, is fantastic. Excited to see how this OSS repo grows.
| janchorowski wrote:
| Thanks for the kind words!
| pipboyguy wrote:
| I've built DE and AI solutions based on Pathway for multiple
| clients. It's robust and fast.
| janchorowski wrote:
| Thanks! Can you share some more details on the usecases and
| features used?
| alexmarquardt wrote:
| Looks awesome!
| snowpid wrote:
| Good old "Enterprise" NATO! Always good for a surprise
| suziemanul wrote:
| Some folks say it's not Fortune 100 but Fortune 1 ;-)
| Arimbr wrote:
| If all the pipeline and the vector index is keep in memory...
| does Pathway still persist state somewhere?
| dxtrous wrote:
| (Adrian from the Pathway team here.) Indeed, everything is RAM-
| based, and persistence/cache relies on file backends. The
| precise backend to use is a code configuration parameter. S3 or
| local filesystem are the currently supported options. For
| documentation, see the user guide under Deployment ->
| Persistence.
| Arimbr wrote:
| Nice, thanks! I was reading
| https://pathway.com/developers/user-
| guide/deployment/persist.... If I understand correctly you
| persist both source data and internal state, including the
| intermediary state of the computational graph. And you only
| rely on the backend to recover from failures and upgrades. So
| if I want to clone a Pathway instance, I don't need to
| reprocess all source data, I can recover the intermediary
| state from the snapshot.
|
| Is it the same logic for the VectorStoreServer?
| https://pathway.com/developers/user-guide/llm-
| xpack/vectorst...
| dxtrous wrote:
| For indexing operators, there is some flexibility regarding
| the amount of internal operator state that is persisted.
| Say, in a stream-stream join structure, it's actually often
| faster to rebuild its state from its "boundary conditions"
| than persist it fully. For vector indexes, it is necessary
| to persist rather more of the internal state due to
| determinism issues (the next time the index is rebuilt, it
| could come back different, and could give different
| approximate results, which is bad). Currently, the HNSW
| implementation which is the basis of VectorStoreServer is
| still not fully integrated into the main Differential
| Dataflow organization, and has its own way of
| persisting/caching data "on the side". All in all, this
| part of the codebase is relatively young, and there is a
| fair amount of room for improvement.
| devnull777 wrote:
| Looks nice! The examples on your site look easy to reproduce!
|
| BTW. Super nice and clear website!
___________________________________________________________________
(page generated 2024-06-13 23:02 UTC)