[HN Gopher] Kedro - Creating reproducible, maintainable and modu...
___________________________________________________________________
Kedro - Creating reproducible, maintainable and modular data
science code
Author : maydemir
Score : 61 points
Date : 2022-01-22 16:00 UTC (7 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| rladd wrote:
| Anything similar for R?
| joelschw wrote:
| Some users have reported the Targets library is similar for
| that ecosystem
| troiskaer wrote:
| How does Kedro compare to MLFlow and Metaflow?
| joelschw wrote:
| Kedro sort of fits into a niche where it just overlaps somewhat
| with 'orchestrators' like Prefect, Metaflow, Dagster, Airflow
| and others. What makes it slightly different is that it it is
| focused on the rapid development journey to production,
| providing guardrails for teams to co-develop ML projects in a
| way that nudges software engineering best practice and clean
| code.
|
| The 'finished article' in many cases should be deployed in
| production in one of those tools which provide specialised
| bells and whistles like scheduling, monitoring and
| observability.
|
| Regarding MLFlow, there is also a slight overlap in terms of
| experimentation, but not things like model serving. Kedro has a
| mechanism to track experiments, but it's more designed to give
| users with zero infrastructure something for free out of the
| box. It's been built in a way that it can be repurposed for
| more dedicated experiment tracking tools - the folks at
| neptune.ai built their own plug-in for this purpose:
| https://docs.neptune.ai/integrations-and-supported-tools/aut...
| troiskaer wrote:
| Seems like Kedro has a similar thesis to Metaflow - I will
| look into it.
| vtuulos wrote:
| Yep, Kedro and Metaflow are more similar to each other than
| to other generic DAG orchestrators like Airflow.
|
| Kedro and Metaflow make it easier to develop robust ML
| projects where orchestration plays an important role but it
| is not everything. They are two separate projects, so the
| way how they approach the problem differs greatly in
| details.
| stevesimmons wrote:
| And Prefect
| version_five wrote:
| This is an ML framework (I'm not actually sure what it does) made
| by McKinsey. I don't know if it was always open sourced but now
| it has been "donated to the community" which I think means
| abandoned?
|
| I'd be very curious to hear if anyone has used it and what
| impressions are on it's value. I've only heard some businessy
| talking points about it, and am hopefully understandably
| skeptical. If someone not from McKinsey had used it, I'd love to
| hear impressions.
| waylonwalker wrote:
| I use it quite heavily at my day job. The core pieces of kedro
| are indispensable on large projects (Pipeline, Catalog,
| DataSets, Runner, hooks). The marketing has a lot of focus on
| the template, It's a great template, but not the key feature
| that brings me back to it. Really like that I never have to
| worry about io, the catalog abstracts this away. I don't worry
| about long runs to get started with my work. It saves all
| intermediate steps automatically, and the Pipeline (DAG) lets
| me run just the parts that I need for my task. Before using
| kedro I would need to scaffold out a way to save all of these
| intermediate steps, or suffer from long run times to get
| started on my task. Lastly the DAG is indispensible during
| environment migrations, it can quickly tell me all of its edges
| that I need to be concerned with picking up and moving with me.
|
| Unlike similar projects kedro is just a python framework. It
| lets me build, deploy, and orchestrate however works best for
| my team.
| barefeg wrote:
| Could you point me to the component responsible for storing
| the intermediate data of the DAG run? I was looking for this
| but couldn't find it from quickly scanning the docs
| joelschw wrote:
| Here you go - https://kedro.readthedocs.io/en/0.17.6/09_dev
| elopment/03_com...
| ekianjo wrote:
| Worked with McK on a modeling tool built with a data pipeline
| in Kedro - it's very opinionated as to how the pipeline has to
| be built, and I was not super convinced that it would be useful
| to use it in any other context (as in, outside of a
| collaboration with McK). Also, YMMV, but I have never heard of
| any company doing ML or data pipelines using Kedro on their
| own.
| oneoffmk999 wrote:
| It's a framework to organize data science code into pipelines.
|
| E.g. if your default might be a series of Python files
| connected by a master script and some config files, Kedro
| organizes them into an explicit Pipeline object. Think loading,
| cleaning, feature gen, model training, mode predictions, etc.
| joelschw wrote:
| Hello maintainer here - hopefully some of our open source users
| chime in, but my favourite will always be NASA:
| https://github.com/nasa/ML-airport-configuration
|
| Donation is the normal term for proejcts joining the foundation
| - in doing so you establish a steering committee where members
| are entitled to voting rights. To graduate as an incubation
| project, 5 organisations need to join your board and the
| project is thus the priorities of the project will no longer be
| driven by just one organisation.
|
| In the short term there is still a full time internal team
| staffed and maintaining the project, but excitingly we now have
| a mechanism for new collaborators to properly come onboard.
| joelschw wrote:
| I should clarify by short term - the foreseeable future is a
| more appropriate description
| holocen wrote:
| Am from McK not involved with Kedro, it's been open source for
| awhile now and I don't think we are abandoning it, but it was
| apparently donated to the Linux foundation.
|
| As far as impressions of it as a tool go it's really just an
| opinionated way to build a pipeline and structure a project.
| Which is pretty useful when you have many. If you're doing
| something that you know will be one off it is a bit overkill.
| Like using airflow when a makefile will do. There are some
| pretty nice plug-ins though if you want to try and have an
| opinionated ecosystem.
| oneoffmk999 wrote:
| I am from McKinsey. I do not work on Kedro, but am a data
| scientist who has used it. I have mixed feelings on Kedro.
|
| Pros:
|
| * Forces data scientists to produce an end product that is not
| poorly organized Jupyter notebooks.
|
| * Data Catalog is good for well structured systems
|
| * Pipeline visualization stack is great (Kedro viz)
|
| * Config options are pretty good
|
| * Seems stable. Dev team is pretty good on this and avoiding
| breaking changes.
|
| Cons:
|
| * Data catalog is kind of bad for any non structured setup with
| flat file data with manual file movement (which is bad to begin
| with but sometimes that's life)
|
| * Productivity of making brand new data science code seems to
| drop when data scientists leave notebooks and
|
| * Most of the time I get brought into a client context because
| the client doesn't know anything about data scientist. A lot of
| data scientists, from both parties, come from academic
| backgrounds and aren't great at code. The nice thing about
| notebooks is that they run. Kedro requires you to create pipeline
| and node objects to wrap around your code before it runs. It
| requires some familiarity with Kedro to understand, run, or
| modify. This makes it seem like a bad idea to dump on a novice
| client. If the data scientist on their side inheriting it doesn't
| really get it, or leaves, there's unlikely to be enough internal
| knowledge to maintain it. I try to avoid pushing any new tech
| stacks on my clients where I can for this reason.
|
| So... I like it but don't love it for consulting work, which is
| ironic.
| waylonwalker wrote:
| My experience with McK came in two phases. Phase one started
| with Kedro, and phase 2 was a cash grab, move as hot and fast
| as you can. In my experience phase 2 got off the ground
| quickly, but after week the notebook was riddled with run these
| sections, but not these, 4 devs on the team were hanging out
| having coffee much of the day because they were waiting for
| time in the notebook to implement their changes. There was no
| version management so days were lost to, well someone deleted
| something they shouldn't have now we need to rewrite it. The
| lack of code review, linting, and formatting tools left them
| swimming in messy code that they would clean up later, but
| later never came.
|
| The kedro project is still running nearly 3 years after it
| started, the notebooks are long forgotten.
|
| Notebooks are fine for single contributors if that is what they
| are comfortable with. If that is what they are comfortable with
| its probably because they have not experienced engagements like
| you are bringing to them that require them to collaborate as a
| larger team. If you plan to effectively run projects that last
| longer than a few weeks with more than one data scientist, I'd
| really challenge them to lean into kedro. The long term
| productivity of the project will greatly benefit from it. You
| will be showing the team a more sustainable way of creating
| pipelines that will lead to continued success after you leave.
| This leaves a better name for you than a quick cash grab that
| gets long forgotten in my opinion.
| sails wrote:
| No pros or cons on deploying models. Seems like that is a
| significant part of the functional value?
| oneoffmk999 wrote:
| Not my area of expertise. Can't comment on it either way. But
| I would tend imagine it's a strength of Kedro.
| joelschw wrote:
| Will point to some experiences open source users have had on
| this journey
|
| https://medium.com/hacking-talent/production-code-for-
| data-s...
|
| https://medium.com/google-cloud/migrate-kedro-pipeline-on-
| ve...
| bshipp wrote:
| I've apparently already starred this repo in the past and have no
| recollection of it. I really need to find a mechanism to sort and
| categorize my starred repositories; this is getting out of hand.
| pjot wrote:
| You should be able to do that natively in GitHub with lists!
|
| https://docs.github.com/en/get-started/exploring-projects-on...
| bshipp wrote:
| Oooh, is that new? I have used Astral[0] in the past which
| works well but required me to both define all my individual
| categories (no social/group suggested tags) and remember to
| use it every time I went on a starring spree.
|
| This looks really interesting!
|
| [0]: https://github.com/astralapp/astral
| tomrod wrote:
| Good to see McKinsey donating something potentially beneficial to
| the world. Kudos to the team that developed this.
| lordgroff wrote:
| I love this project, not from McK.
|
| I mean I guess it's "opinionated" but having worked in this field
| for some time, ML code and pipelines are often laughably
| immature, stuck with spit, glue and sellotape. Kedro forces
| people into a bit of a straight jacket for sure, but man does it
| make things more readable.
|
| Plus points, it's very extensible so if there's a design element
| you're missing or one of its opinions you just can't abide by,
| then you can extend it. I think it deserves more exposure.
| bshipp wrote:
| It might be opinionated but it's no less accurate.
|
| Data science is many researchers first foray into programming
| in any language and, speaking from my own path, was a
| relatively self-directed discipline in the early days.
|
| I often longed for a 'standard approach' but many of the
| projects that suggested them were packed full of boilerplate
| that made code even harder to read and write, especially if
| your background wasn't heavily into OOP.
___________________________________________________________________
(page generated 2022-01-22 23:01 UTC)