[HN Gopher] Comparison of Data Lake Table Formats (Iceberg, Hudi...
___________________________________________________________________
Comparison of Data Lake Table Formats (Iceberg, Hudi and Delta
Lake)
Author : anhldbk
Score : 91 points
Date : 2022-06-13 09:41 UTC (13 hours ago)
(HTM) web link (www.dremio.com)
(TXT) w3m dump (www.dremio.com)
| ajantha wrote:
| Nicely summarised and visualised :+1
| alexmerced wrote:
| venki80 wrote:
| Wondering if this is basically what all data lakes will look like
| in the future. All data stored in these table formats...
| jerf wrote:
| Consider the whole data sector as: Generation
| -> Ingestion -> Transform (and possible looping
| back as derived data is created) -> Resting place ->
| Final Useful Product
|
| Not saying that's a perfect model, just something to hang the
| terms I use in this post on.
|
| And then consider that in order to get from the beginning of
| that process to the end, there is a certain amount of "Data
| Cleanup" to be done, ranging from merely validating that the
| data is sensible to in the limit literally handing huge blobs
| of text and unstructured data to humans and making them input
| something useful into the system out of it.
|
| My assessment of the whole data community right now (and
| please, by all means react to this with your own opinions, I'm
| curious about them) is that the entire flush of fads going back
| and forth right now amounts to an argument about how exactly to
| distribute the necessary Data Cleanup work across that
| pipeline. The theoretical ideal is for everything to just be
| super awesome at the generation phase and nothing else has to
| worry about it, but it was rapidly discovered that making the
| generation part so expensive inhibits the data from ever being
| generated. With clean data, downstream could do all sorts of
| database-y storage technologies and do all sorts of clever
| things with the clean data, but the data is never clean.
|
| The natural overreaction is to flip entirely in the other
| direction and just get it in and push the validation as far
| down the pipeline as possible. Here you get the "big piles of
| vaguely organized files". You get more data this way because
| you lower the costs of generation and to some extent ingestion,
| but you complicate everything downstream.
|
| It seems to me we're currently in a phase where everyone is
| just sort of hoping somebody else will do it, and we're
| flailing around a bit.
|
| Very opinionated: Where we're going to settle in, and where you
| can already see the shape forming up, is that it'll be a little
| mix & match at each level. Do what's easiest in each level at
| that level, and you end up with the cheapest and most effective
| result across the pipeline considered as a whole, even though
| no individuals working in any part of it will be 100% happy.
| There won't be a magic solution, but if, for instance,
| Ingestion demands that the Generation at _least_ be amenable to
| some tabular view, even if there are some escape hatches for
| generic JSON bits, they can start operating with sensible tools
| (SQL-ish like Clickhouse or something) instead of just having a
| pile of opaque nothingness, and then the next levels down won
| 't be able to count on data quality or coherence 100% but you
| can start layering in cleanliness and coherence as you go, etc.
| There just isn't a magic solution that fits into bullet points
| cleanly.
|
| (There's this "bronze/silver/gold" thing going on, which I
| think is silly because there's really not much benefit to
| trying to force an arbitrarily-deep and complicated pipeline
| into such classifications, but the idea is there.)
|
| Or, in short, yes I expect to see more tabular data. It just
| won't be tabular for the same reason that relational DBs use
| tables. It'll be tables even fairly early just because you need
| _some_ sort of handle on the data to do any sort of useful
| manipulation on it. If relational DBs use tables as an emphasis
| on tables _qua_ tables of data, data lakes will use tables as
| defined handles on individual pieces of data to be able to
| manipulate them as opposed to pure unstructured piles of
| "something".
|
| It reminds me of the 20+ year, still ongoing argument about
| where in the "Browser -> Server -> Backend Services (including
| DB)" stack the work needs to be done. There's a certain amount
| of work that has to be done. You've got a bajillion choices
| about where to do it, and it's been sloshing back and forth
| across the entire time the web has existed ("do it all in SQL
| procedures! Do it all on the client!") because there is no
| simple hard & fast correct answer that everyone can follow for
| every case.
|
| Just as with that world, this reality won't stop a pile of
| vendors from promising they can somehow make this problem go
| away, but they really can't. They can reduce the accidental
| complexity, and that's cool and may be worth paying for, but
| there's essential complexity that isn't going anywhere.
|
| (Stretching even more abstractly, I'm writing a bit about how
| to do stream processing with io.Reader in Go, and it reminds me
| a bit of that, too. Stream processing is too complicated to
| write a single-shot conversion from "whatever's coming in" to
| the golden data you're looking for in many cases, and the
| solution is to fold in several transforms at a time, each
| comprehensible and testable, until you get what you need. The
| whole composed stream would be impossible to understand at
| once, but each piece can make sense. Trying by ideological fiat
| to jam it all into one piece or forcing the wrong place to do
| something is a recipe for disaster. You have to let the problem
| guide along its solution, or you'll end up wasting effort
| fighting to impose your beliefs on a system that doesn't care
| about them at all.)
| SpicyLemonZest wrote:
| I definitely think that some kind of semi-structured storage is
| the future. "Here's a giant heap of files" was always kinda a
| hack for when you outscaled your RDBMS but didn't have time to
| build something better.
| henrydark wrote:
| A major problem with these table formats that will surface soon
| enough is that they use serial numerical ordering for versions.
|
| It's like inventing SVN for data. Soon enough git will have to be
| invented as well.
| mrry wrote:
| "Project Nessie: Transactional Catalog for Data Lakes with Git-
| like Semantics"
|
| https://projectnessie.org/
|
| This supports lightweight branches, and transactional commits
| and merges. I haven't used it--and it seems cool--but it also
| seems a little heavyweight to get cross-table transactions on
| top of these table formats (which would be my primary use
| case).
| alexmerced wrote:
| Project Nessie also powers Dremio's Arctic service, so you
| can get all the branching and benefits of Nessie with an
| intuitive UI to browse branches, create branches and merge
| branches. Also, it is a cloud managed service with a free
| tier.
| hrosen wrote:
| Helpful to see a concise comparison!
| alexmerced wrote:
| anonymousDan wrote:
| How does the concept of a table here differ from that of a
| standard relational table (if at all)? Is it that the table is a
| logical abstraction over a distributed set of files?
| SpicyLemonZest wrote:
| The underlying storage is less structured, which means that
| some common database optimizations (in particular row-level
| indexes) are unavailable in these kind of systems, but it's
| conceptually the same as a standard relational table.
| chrisjc wrote:
| These are effectively external tables. They likely exist on
| blob storage and not in any particular DB's storage system. As
| a result, you have a myriad of ways to interact with that data.
| Moreover, compute and storage resources are now decoupled.
| alexmerced wrote:
| evilturnip wrote:
| We're currently looking into datalake implementations. Right now,
| we only have 1 or 2 data sources. Current thinking is reading
| them on the fly, combine them using pandas dataframe and query
| that. Anyone have experience with doing something similar?
| alexmerced wrote:
| This presentation may help:
| https://www.youtube.com/watch?v=LiC9vZATv0o&t=8s
| tomnipotent wrote:
| It can seem like overkill, but I can't recommend Snowflake
| enough. It's so simple to setup and manage, and JSON support
| makes it easy to just drop JSON line files into a table and
| query on the fly.
|
| Depending on your data/query volume, it can also be very cheap.
| I_Love_Data_22 wrote:
| When I think of SNOW, cheap doesn't come to mind.
| chrisjc wrote:
| Also recommend Snowflake!
|
| However, this article was about table formats so maybe OP
| wasn't thinking about JSON. If so, looks like Snowflake only
| supports read operations atm. Snow Summit this week, so maybe
| that will change?
| mason55 wrote:
| Depends on the type of data you're processing, your business
| goals and the existing consumers that you need to support.
|
| At a minimum I'd suggest planning to load the data from the
| data lake into an RDBMS (OLAP/columnar preferably). Then it's
| accessible to more than just Python scripts (BI tools, users of
| other languages, etc).
|
| Depending on how much data there is, should also plan on data
| summarization strategies. You can either build some common
| rollups to ensure that consumers are all looking at the same
| summaries or you can let consumers build their own
| transform/load pipelines from the raw data lake or you can let
| consumers build their own transform pipelines from the data in
| the data warehouse (using something like dbt).
|
| The benefits of a data lake architecture really appear when you
| have lots of sources, lots of disparate consumers, and lots of
| data, with some schema evolution & unstructured parts thrown
| in. If you only have 1 or 2 sources, small enough data to query
| raw data in Pandas, and consumers are restricted to Python
| scripts, then you can skip a lot of the architectural headache
| of building a data lake for now (just make sure to archive your
| raw data somewhere if you want to be able to pull it into a
| data lake in the future).
| eatonphil wrote:
| You can do this for small datasets and I built an app around
| exactly this use case [0].
|
| As your data gets bigger (just over time even if individual
| days don't emit more data) you'll end up building out
| partitioning schemes yourself or you'll move to a system that
| does that for you like bigquery/snowflake/etc.
|
| [0] https://github.com/multiprocessio/datastation
| divbzero wrote:
| Does anyone have good real life stories of how data from a data
| lake made a real difference in a product or a business?
| alexmerced wrote:
| Yep, in this talk I go over an example from Netflix where
| Apache Iceberg originated:
| https://www.youtube.com/watch?v=LiC9vZATv0o&t=8s
| ttunguz wrote:
| Does anyone have experience running either of these three in
| production?
| alexmerced wrote:
| I'll be presenting this content at this meetup if you want to
| learn more: https://www.meetup.com/subsurface-
| global/events/286442080/
| curiousDog wrote:
| Yes, we run iceberg and it's pretty thorough
| ramesh31 wrote:
| Curious why you chose it over Delta Lake? What does your
| ETL/BI stack look like?
| chrisjc wrote:
| Looking into setting up a Lakehouse/Datalake over the
| coming quarters. I'd say the biggest appeal to us is that
| Iceberg can handle schema drift/evolution, and a little
| more open.
| henrydark wrote:
| The specification is more transparent
| vikkes wrote:
| Iceberg is community driven
| venki80 wrote:
| The big tech companies like Apple and Netflix all use Iceberg
| alexott wrote:
| Apple uses Delta:
| https://databricks.com/session_na20/patterns-and-
| operational..., similarly Disney+:
| https://databricks.com/p/webinar/unlock-the-potential-
| inside..., and hundreds other companies
| (https://databricks.com/customers)
| bushbaba wrote:
| Apple has named committees and tech talks on iceberg. Like
| all big companies they use multiple technologies
| diptnt wrote:
| Thanks for bringing this comparison out!
| alexmerced wrote:
| pid-1 wrote:
| The repo comparison was really cool. I guess that could be made
| into a product.
| infogulch wrote:
| Agreed, the repo comparison was surprisingly thorough.
| alexmerced wrote:
| I'll be presenting this content at this meetup if you want to
| learn more: https://www.meetup.com/subsurface-
| global/events/286442080/
| broberts2261 wrote:
| Great comparison!
___________________________________________________________________
(page generated 2022-06-13 23:02 UTC)