[HN Gopher] Lessons Learned from Scaling to Multi-Terabyte Datasets
___________________________________________________________________
Lessons Learned from Scaling to Multi-Terabyte Datasets
Author : thunderbong
Score : 120 points
Date : 2024-06-20 07:20 UTC (15 hours ago)
(HTM) web link (v2thegreat.com)
(TXT) w3m dump (v2thegreat.com)
| atemerev wrote:
| A "multi-terabyte dataset" is something that fits into a single
| machine, and can be loaded into Clickhouse in a few minutes.
| stephankoelle wrote:
| I can confirm that. 10tb uncompressed, 3tb compressed in
| clickhouse, 60 billion Rows, queries are fast.
| ZephyrBlu wrote:
| Really curious what "fast" is in this context. Milliseconds,
| seconds, minutes?
| atemerev wrote:
| Full-range query with aggregations is around 40-60 seconds.
| Can be optimized, of course.
| dleeftink wrote:
| How about many-to-many comparisons? Interested in this
| use-case for fuzzy matching of many data records using a
| local db.
| riku_iki wrote:
| Base on numbers you put, you would need to have stable
| 50-100GB/s speed to your disc system to just read 3TB
| data in 40-60sec. Would you mind sharing your setup?
| winrid wrote:
| That's only if you read everything, and if none is in
| ram. With a columnar DB you can just scan the values of
| the column you need.
| riku_iki wrote:
| Then 3tb of total data volume is irrelevant. What
| relevant is size of column.
| burgerrito wrote:
| I wonder what's the standard of "big" in these context--for
| example I always thought that some million rows is a very big,
| but only recently learned that it's a size that RDBMS such as
| PostgreSQL can handle it with no problem.
| hnthrowaway0328 wrote:
| I think 1pb at least.
| Dylan16807 wrote:
| Whatever big is defined as, it needs to at least be that your
| data can't fit into RAM on a high end server.
|
| There's also the threshold where your indexes don't fit into
| RAM. And the threshold where your data no longer fits into
| PCIe SSDs on a single server. (The combined bandwidth of the
| SSDs will rival the RAM, but with more latency.)
| ants_a wrote:
| For RDBMS we typically consider millions to be tiny to small.
| A billion is somewhere on the boundary of medium to big.
| jandrewrogers wrote:
| These days terabytes is a medium-sized database. A trillion
| rows of indexed data will fit on a single cloud VM and be
| reasonably performant.
|
| I think a good definition of "large" is "several times larger
| than will fit on a practical modern server". Servers with a
| petabyte or more of fast attached storage are increasingly
| common, so that threshold is pretty high. Machine-generated
| data models (sensing, telemetry, et al) routinely exceed this
| threshold though.
| riku_iki wrote:
| > A trillion rows of indexed data will fit on a single
| cloud VM and be reasonably performant.
|
| create and subsequently update index on trillion rows is
| very untrivial task performance wise.. Which DB you would
| suggest to use for this?
| winrid wrote:
| You just have to plan and do it in the background at
| those data sizes.
| riku_iki wrote:
| > just have to plan and do it in the background
|
| this can mean building some untrivial infra, so the task
| is much more complicated than just using cloud vm.
|
| My point is that bottle neck and why you need cluster and
| not single VM is likely CPU and iops, and not data
| storage.
| munchbunny wrote:
| These days I'd probably describe "big" as "doesn't make sense
| to use SQL anymore".
|
| Qualitatively, I think it becomes "big" when you have to
| leave the space of generic "it just works" technologies and
| you have to start doing bespoke optimizations. It's amazing
| how far you can get these days without having to go custom.
| mulmen wrote:
| https://yourdatafitsinram.net/
| JBorrow wrote:
| Well, that is only true (and helpful) if it is a dataset that
| benefits from being in a tabular format.
|
| Consider, for instance, particle-based simulation data (loading
| that into any database is a waste of time...), or a set of
| images that you need to perform non-standard individualized
| processing on (again, a waste of time).
|
| Stuffing stuff into postgres or clickhouse is great for your
| typical 'data science' workflows where data is consistent and
| the problem is 'oh no our transaction volume has increased
| 100x'. But in other (some would claim more interesting) cases,
| using database systems is unhelpful.
| v2thegreat wrote:
| Hi! I'm the author.
|
| You're right! The context that I was talking about was
| actually in the GIS space! Depending on what you're trying to
| do, it's not uncommon to have a single dataset that's 50TB
| large (think: 100-500m resolution global raster with daily
| data for 30 years)
|
| And that's not even considering any memory overhead when
| performing operations on top of that data.
|
| That type of stuff wouldn't fit into Postgres (we tried once,
| a long time ago), and usually, you might be trying to use
| multiple datasets simultaneously, so that data adds up pretty
| quickly!
|
| It's my first post (and writing) ever, so I have room for
| improvement. Thanks for taking the time to read!
| afpx wrote:
| Probably ignorant question, but how do I avoid the issues of
| cross join say across several tables (or equivalents)? It seems
| like I always need more than a many TBs of memory.
|
| Do you spinup a U-18TB1 112xlarge or something to the
| equivalent as needed? Or, are you running it all day? I've
| never used Clickhouse, but I'd dealt with a spark system that
| was written like 10 years ago. I'm wondering if it's something
| easily migrated to Clickhouse?
| fifilura wrote:
| You can use awk for that and pipe it into separate files for
| each bucket. And the you awk a bit more and the you are done.
|
| This was sarcasm based on several earlier discussions. And i
| am prepared for the downvotes.
|
| I think awk is fine for some things but awk proponents
| underestimate the scale of a map reduce job with multiple
| dimensions in both directions.
|
| For example i believe awk will struggle with a question like
| "of one million customers and one billion sales, give me the
| top 10 spenders". And to stick to the topic, i believe a one
| node database will also struggle.
| jakozaur wrote:
| Servers grow much bigger. 256GB of RAM is pretty much a standard
| rack server on a major cloud. On AWS, ultra memory with 24TB of
| RAM is one API call away from you :-). The article took note of
| that, but tooling on one node puzzles me.
|
| So, multi TB can still be a single node with DuckDB. It's also
| rather a small ClickHouse cluster. It sounds easier to use DuckDB
| than the proposed tools.
|
| In many use cases, if it's not 24/7, you can use a data lake
| (e.g. Iceberg) and query it if needed. Databricks seems a way to
| go, since the author uses Spark.
| mulmen wrote:
| What do you mean by "not 24/7"? Why would a data lake go
| offline?
| dimaor wrote:
| not sure, but maybe it's possible only to update the model in
| a specific time? are there other uses to the data apart from
| learning and validation?
| creshal wrote:
| Probably in the sense of "you're only running the really
| heavy analytics queries once a day/week/month", and thus
| don't constantly need the 24TB instances.
| pantsforbirds wrote:
| It's pretty common to run analytics workloads on a fixed
| schedule. I've setup anlaytics workflows that ingest data
| every day/week/quarter and generate a report once a day.
|
| With workflows like that something like DuckDB can be amazing
| jakozaur wrote:
| It is very common that the majority of work happens during
| business hours. In some setups it's easy. It's better to get
| beefy instance during that time and avoid having it running
| all the time.
| mritchie712 wrote:
| Iceberg is working hard to support pure python[0] / rust[1]
| workflows without Spark. Following Tabular's acquisition [2], I
| hope it still moves in this direction at the same clip (very
| impressed with the speed of development on pyiceberg).
|
| We're using iceberg + duckdb to power analytics in our app[3]
| and I'm really happy with the combo.
|
| 0 - https://github.com/apache/iceberg-python
|
| 1 - https://github.com/apache/iceberg-rust
|
| 2 - https://x.com/thisritchie/status/1800522255426072647
|
| 3 - https://www.definite.app/
| alexott wrote:
| You can already have it in Delta with Delta Rust and Python
| bindings: https://github.com/delta-io/delta-rs
| mritchie712 wrote:
| yes, we're evaluating Delta. We went with Iceberg out of
| concern that Delta was too closely tied to Databricks.
|
| Following the Tabular acquisition, the decision is murkier.
| sponaugle wrote:
| When I first read the title I had in my head that it said multi-
| petabyte. When I think Multi-terabyte, I think 'yea, that is
| mostly in RAM'. 4TBs of RAM is sort of the new lower norm in DB
| machines, and GCP and AWS have high RAM machines at 12TB and
| 24TB. Not that these are not good ideas about scaling to multi-
| terabytes.. it is just interesting how much data has grown in
| size.
| v2thegreat wrote:
| Hi! I'm the author. I've seen this comment several times and
| thought I'd address it here.
|
| I should've clarified that when I say multi-terabyte, I meant
| 100s TBs of data. It's my first time writing anything, so I'm
| still getting the hang of things.
|
| Damn, that would've been a more clickbait title, too.
|
| Oh well.
| sponaugle wrote:
| Ah yea, no worries. It is always good to consider ways to
| optimize, either for TB or PB!
| fifilura wrote:
| Since you are already running on AWS, I wonder if you would
| consider AWS serverless EMR an option?
|
| https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGu...
|
| I guess the pricing will be cheaper than renting entire machines
| since you are doing something like micro-spot allocations for map
| or reduce jobs.
|
| We have done a lot of analytical and also batched workflows with
| AWS Athena. This is really cheap and also, once you get a hang of
| it, you get surprised how much you can achieve with only SQL.
|
| I am hoping serverless EMR would be roughly equivalent, just with
| another language...
| sammysidhu wrote:
| https://github.com/Eventual-Inc/Daft Is also great at these types
| of workloads since it's both distributed and vectorized!
___________________________________________________________________
(page generated 2024-06-20 23:01 UTC)