[HN Gopher] HBase Deprecation at Pinterest
___________________________________________________________________
HBase Deprecation at Pinterest
Author : cloudsql
Score : 101 points
Date : 2024-05-14 10:27 UTC (1 days ago)
(HTM) web link (medium.com)
(TXT) w3m dump (medium.com)
| winrid wrote:
| Wow, never realized Pinterest had 6 petabytes of data. I wonder
| if they're including images in that. Even billions of rows is
| usually around 1-2tb, so makes you wonder what they're storing
| many billions of.
| ddorian43 wrote:
| It makes no sense at their scale to store images in Hbase. It's
| probably just user tracking/analytics.
| badpun wrote:
| In NoSQL world, you often store multiple duplicates of the same
| data, each of which is optimized for a different use case.
| dehrmann wrote:
| The SQL world calls this an "index."
| etc-hosts wrote:
| I am attempting to reach enlightenment by contemplating
| your wisdom.
| winrid wrote:
| No. Most NoSQL dbs support indexes and secondary indexes
| too.
|
| In SQL world we call this "denormalization". It's called
| the same thing in NoSQL, too.
| badpun wrote:
| Indices are no good if you constantly do large amounts of
| inserts or updates (say, entire clickstream of a popular
| global site) - keeping them up to date will massively slow
| down the inserts/updates. Wheresas, in NoSQL, you can do as
| inserts as quickly as your disk can write data, and still
| query the dataset as if it had an index.
| jeffbee wrote:
| If a company has HBase and doesn't have anything else, then the
| developers will store everything in HBase due to lack of
| alternatives. They'll store logs in HBase, metrics in HBase,
| images in HBase.
| evanelias wrote:
| They almost certainly have _trillions of rows of data_. The
| popular social media / user-generated content sites are just
| huge. You can easily get to that size just from the core
| product OLTP data/metadata, no need to include media.
|
| For one comparison, Tumblr hit 100 billion unique rows of
| relational data in MySQL (on masters, not including replicas)
| back in _October 2012_. So they 're easily in the trillions of
| rows today, and Tumblr is smaller than Pinterest!
| ddorian43 wrote:
| It reminds me of the often optimization story (how we got 2x+
| faster by not doing very inefficient things), in this case going
| 6->3 replicas.
|
| Example: TiDB at a certain time didn't write rows clustered by
| the primary key on disk (they had a separate index). This is very
| costly in distributed setups (less costly on single-node setups
| like PostgreSQL).
|
| There are many such cases in many dbs. Another point lacking in
| most dbs is the "lsm compaction overhead" you need to do for all
| replicas when you're not using shared distributed storage.
|
| This optimization can be seen on QuickWit (building/compacting
| inverted index is even more expensive than LSM compaction).
| zuck_vs_musk wrote:
| They switched to an SQL database. NoSQL
| redwood wrote:
| Great to see TiDB mentioned for use here, I have to say I love
| their demo at ossinsights. Anyone else using them successfully in
| prod?
| std_reply wrote:
| AirBnB, Databricks, Flipkart 3 of the largest banks in the
| world, some of the largest logistics companies in the world, at
| least 2K seriously large installations.
| redwood wrote:
| Shocking to see a company like Pinterest has built multiple in
| house data stores. Talking about a maintenance burden...
| vkazanov wrote:
| My company's data function, established 14 years ago, has a
| bunch of custom stores, as well as data-related tools. Back
| then data use cases were not really well-known, we had to build
| things from scratch a lot.
|
| It is a very different landscape these days!
| lopkeny12ko wrote:
| This is my first reaction as well. How in the world do you end
| up with close to 10 different databases in production? And
| furthermore, most of which are totally proprietary? Just use
| Postgres...
| aeyes wrote:
| Easy: These companies have a promotion process which require
| you to generate "impact" to get promoted. Using existing but
| boring technology which solves the problem in an efficient
| manner apparently doesn't show off how smart you are.
|
| This is not a snarky comment from an outsider, I very much
| work at such a company. It's ridiculous.
|
| Edit: But I must admit that most of this was probably
| developed 10 years ago when the ecosystem was much more
| limited. At least they now came to the conclusion that it's
| time to move on. I would not have recommended Postgres 10
| years ago and even today there is no way it would work at the
| scale of Pinterest.
| openplatypus wrote:
| So sad because so true.
|
| Seeing working technology replaced because someone doesn't
| like it, get accolades for it, only to face the same
| limitations as old tech is just depressing.
| aeyes wrote:
| You forgot the next step: The people who developed it
| move on to other companies and nobody wants to touch it.
| ted_dunning wrote:
| HBase is coming up on twenty years old.
|
| In 2008, scaling was _really_ hard. And the senior
| engineers at that point had been badly burned by trying to
| scale during the 2000-2005 period.
|
| The comment about servers of the time being less powerful
| than smart phones of today is spot on.
| doctorpangloss wrote:
| Your application wants to use Postgres.
|
| Kubernetes, which it runs on, wants etcd.
|
| Keycloak wants to use Postgres and Redis for sessions.
|
| Plausible Analytics wants to use Clickhouse and Postgres.
| Matomo specifically wants to use MySQL.
|
| Lago Billing wants to use Postgres, Redis and Clickhouse.
|
| Prometheus uses TSDB, but okay, it's "just" files and a WAL.
| Wait a minute, so is Postgres...
|
| Jaeger wants to use Cassandra or Elastic.
|
| Thanos wants S3 APIs. So maybe you use Minio.
|
| Okay. That's 2024: in production, you will wind up with 7
| different databases. At least.
| dehrmann wrote:
| Not sure if I can get you to 10, but different DBs work at
| different scales for different workloads. There's relational,
| key-value, caching, time series, data warehouse, and search
| index.
| llm_trw wrote:
| >Just use Postgres
|
| Now try running Postgres on vintage hardware from the period
| and you quickly see why we did what we did.
| cortesoft wrote:
| They are operating at a scale where you can't "just use
| Postgres"
| zerkten wrote:
| If you rewind 10 years to a time before waves of big tech
| layoffs and the market for engineers was very much in favor,
| you'll find your answer. The explosive growth in hiring coupled
| with FOMO-driven risk taking resulted in many projects which
| were essentially resumeware for engineers. This environment
| permitted people to escape strategic thinking and any
| consequences.
|
| Imagine having to deal with multiple data stores for your daily
| development but then go on-call for a week and have to deal
| with twice as many plus the inevitable lack of runbooks etc.? I
| personally have a big tech experience with the same data store
| proliferation in an organization with pretty solid RDBMS use.
| In the last 3-4 years we've been undoing the damage and
| migrating data. Toil for engineers is lower and uptimes are
| better.
| llm_trw wrote:
| At the time there were _no_ tools that scaled to what people
| needed so they had to build their own.
|
| It's easy to look back and think we were idiots for doing it
| that way but you need to remember that the average server in
| 2014 has the power of the average smartphone today.
| zenbowman wrote:
| This was the norm about a decade ago. When I was at Hulu, we
| built our own analytics platform on top of Hadoop, we almost
| certainly wouldn't have done that today with the amount of off-
| the-shelf stuff available.
|
| Even things like protobuf or Avro weren't as broadly adopted
| (>80%) at the time, many many companies at the time persisted
| stuff using JSON or other text formats (which in retrospect was
| very dumb, but it was very normal for a while).
| pjmlp wrote:
| As usual, with any technology adoption wave, remember this two
| key historical moments,
|
| > Introduced in 2013, HBase was Pinterest's first NoSQL
| datastore. Along with the rising popularity of NoSQL, HBase
| quickly became one of the most widely used storage backends at
| Pinterest.
|
| Followed by,
|
| > For the past few years, we have seen a seemingly steady decline
| in HBase usage and community activity in the industry, as many
| peer companies were looking for better alternatives to replace
| HBase in their production environments. This in turn has led to a
| shrinking talent pool, higher barrier to entry, and lower
| incentive for new engineers to become a subject matter expert of
| HBase.
|
| Lets see how TiDB holds on in the same timeframe, although being
| boring SQL might keep it around.
| pornel wrote:
| > the HBase version upgrade is a slow and painful process due to
| a legacy build/deploy/provisioning pipeline and compatibility
| issues
|
| Is that HBase's fault, or Pinterest's added complexity?
|
| I'm baffled when databases don't support seamless in-place
| upgrades, and require a full dump and restore instead. At certain
| scale a full rebuild is as complex as replacing wheels of a
| moving car.
| beeboobaa3 wrote:
| HBase & Hadoop are painful to upgrade. Honestly doing anything
| with them is painful.
| eclark wrote:
| HBase supports in-place upgrades. Almost* all version upgrades
| have been relatively painless if you have automation to do the
| necessary operations across a cluster of nodes. Hadoop upgrades
| have been a similar story. That minimum automation necessary is
| high for all stateful data stores of this size.
|
| * A few notable exceptions exist where Hadoop and HBase
| upgrades were necessary simultaneously. These have been awful
| experiences that required huge efforts to accomplish.
| lopkeny12ko wrote:
| This article has so many words yet has such little information,
| and is remarkably sparse in technical detail. What did they
| actually do? What did they build? _How_ did they migrate? What is
| "SDS"?
|
| Is every article written by ChatGPT now? My confusion was
| partially answered as soon as I saw the word "delve."
| pxx wrote:
| This blog post starts off saying it's part one of a three-part
| series so the lack of detail makes a lot of sense in context.
| Given that it's a corporate blog post, it's unlikely that we'll
| get a particularly deep technical dive but there is plenty of
| detail in what is stated to be an introduction.
|
| Also this article doesn't feel like it's written by ChatGPT at
| all. "Delve" is not even a very uncommon word; just one use of
| it isn't necessarily indicative, and even if it was, it's used
| in the summary of the rest of the series (which you seem to
| have missed in your hunt for gotchas)! I think LLM bullshit is
| definitely making everything a lot worse but this isn't even an
| example of such.
| rwultsch wrote:
| "Introduced in 2013, HBase was Pinterest's first NoSQL
| datastore."
|
| I don't think this is correct. When I started in late 2013 Redis
| was being used as a persistent data store. And what pain it was.
| I convinced leadership in late 2014 this was a bad and they had
| me keep it alive until it was replaced by MySQL in mid 2015.
|
| HBase was nothing but pain at Facebook where it was supposed to
| replace MySQL and then Pinterest where... I think there was hope
| it would replace MySQL. Once I automated MySQL at Pinterest I
| think it wasn't so bad, particularly given the absurdly limited
| staff they gave the problem.
| softwaredoug wrote:
| I had a (low traffic) app with Redis as the only data store,
| that we'd periodically dump to disk and copy the dump off for
| backup. It was... not great for anyone who was used to
| maintaining a normal web app. And it broke down because the
| underlying relationships we mostly cared about were, in fact,
| relational. So you got a lot of duplication / nesting,
| homegrown hierarchy, or pointers to other areas in Redis...
| jerryjerryjerry wrote:
| Emm, I'm very curious about the reasons why they finally chose
| TiDB.
| eclark wrote:
| > Production HBase clusters typically used a primary-standby
| setup with six data replicas for fast disaster recovery, which,
| however, came at an extremely high infra cost at our scale.
|
| This is the real killer. HBase uses Hadoop for its replication.
| That replication does 3 copies in the same data center. If you're
| a company that requires online data of this scale, you probably
| also have several other data centers or clouds. Having to
| replicate entire datasets 3x for every new data center is cost-
| prohibitive.
|
| That, along with the fact that HBase has issues at NVMe speeds
| and throughputs, are true issues.
| riku_iki wrote:
| > That replication does 3 copies in the same data center.
|
| You can set replication factor 1 if you want. You just will
| have high chance to lose your data forever.
| eclark wrote:
| The HBase write-ahead log requires a full pipeline. It
| speculatively drops hadoop nodes out of the write pipeline.
| Because of the complexity of the WAL, I don't know anyone who
| has tried with less. (There are a good number of ways to run
| HBase with no write-ahead log)
|
| There are people who run HBase with Reed Solomon encoding on
| the HFiles (the store files). That can get a replication
| factor below 3. I don't think that Pinterest ever ran an
| updated enough Hadoop for that to be available.
|
| > You can set replication factor 1 if you want.
|
| You can't, really. HBase will fail to open regions that have
| files that aren't able to be read. So setting the replication
| factor to 1, have a single hard drive go out and the table
| will forever have regions that can't be opened. The name node
| will have a file that's got lost blocks. HMaster will pass
| around the region assignment until it sticks failed to open.
| kristopherkane wrote:
| HBase was a joy to use from a application developer standpoint.
| dehrmann wrote:
| You should check out Bigtable.
| dehrmann wrote:
| I worked for a place that used HBase heavily. They migrated from
| AWS to GCP ~just for BigTable since the data model is essentially
| the same. They're a Java shop, and the drivers are actually the
| same. The workload of managing HBase and HDFS was high, and it
| was unreliable enough that they always had a failover cluster set
| up. Interestingly, the migration surfaced degenerate cells/tables
| that might have been partially to blame for reliability issues.
| Even 5 years ago, HBase was slowly dying, and at the end of the
| day, they didn't want the company's core competency to be HBase
| management.
| ram_rar wrote:
| For companies like Pinterest, where data storage isn't the core
| business, should the focus be on building in-house data
| warehouses or leveraging managed service providers (MSPs)? While
| building in-house offers control and customization, MSPs can
| potentially address complexity and infrastructure costs.
|
| Can someone from Pinterest comment on specific performance needs
| (SLOs) that influenced their choice between TiDB and other
| solutions, including managed services? Considering complexity and
| cost, could an MSP have addressed their needs effectively?
| esafak wrote:
| Cost, complexity, and features like transactions and secondary
| index. For more see
|
| https://static.pingcap.com/files/2023/07/19065834/BRO-Pinter...
|
| https://medium.com/pinterest-engineering/online-data-migrati...
|
| https://www.youtube.com/watch?v=svi5AFi2_To
___________________________________________________________________
(page generated 2024-05-15 23:02 UTC)