[HN Gopher] Show HN: Hamilton's UI - observability, lineage, and...
___________________________________________________________________
Show HN: Hamilton's UI - observability, lineage, and catalog for
data pipelines
Hey HN - Stefan and Elijah here from DAGWorks (http://dagworks.io/,
YC W23). If you don't remember us from our previous HN launch
(https://news.ycombinator.com/item?id=35056903), we're the authors
of Hamilton (https://github.com/dagworks-inc/hamilton), an open-
source library for building self-documenting, modular dataflows in
python that works for data, ML, LLM pipelines, & even web-
workflows. We've been developing this UI for a while and we're
excited to say we open-sourced it! It comes out of the box with the
following capabilities, and only requires a single line code change
to get: 1. Execution + metadata capture, e.g. automatic code
profiling 2. Data/artifact observability, e.g. summary statistics
over dataframes, pydantic objects, etc... 3. Lineage & provenance
of data, e.g. quickly see what is upstream & downstream of
code/data. 4. Asset/transform catalog, e.g. search & find if
feature transforms/metrics/datasets/models exist and where they're
used. While the UI currently only self-populates for Hamilton
dataflows, we're looking to expand to other frameworks (we'd love
your feedback!). Check out the following video for an overview:
https://www.youtube.com/watch?v=0VIVSeN7Ij8, as well as the
documentation: https://hamilton.dagworks.io/en/latest/concepts/ui/.
We're looking for feedback/adopters - feel free to reach out if you
have any questions!
Author : elijahbenizzy
Score : 26 points
Date : 2024-05-02 13:22 UTC (9 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| magicaltrout1 wrote:
| Hamilton is a great pipeline platform, super lightweight and easy
| to use. I'm happy they've opensourced this UI to give us deeper
| insights into our code and hows it's being executed!
| krawczstef wrote:
| Thanks!
| talos_ wrote:
| This looks like an interesting tool to log data pipeline runs. Is
| it closer to an Airflow ETL dashboard or a MLFlow experiment
| manager? Who's supposed to manage it?
| elijahbenizzy wrote:
| A bit of both! But closer to MLFlow.
|
| Hamilton/the UI doesn't run it, but it does give visibility. So
| it has the tracking/visibility of airflow and the
| metrics/artifact tracking of MLFlow bundled together. Can also
| be used with those systems happily.
|
| Our goal was to provide an all-in-one system that provides a
| host of data/ML/LLMops needs.
___________________________________________________________________
(page generated 2024-05-02 23:02 UTC)