[HN Gopher] Rearchitecting Core Services at X
___________________________________________________________________
Rearchitecting Core Services at X
Author : rglover
Score : 41 points
Date : 2024-09-16 20:38 UTC (2 hours ago)
(HTM) web link (twitter.com)
(TXT) w3m dump (twitter.com)
| AI_beffr wrote:
| cool
| jiggawatts wrote:
| Why do I get the impression that all of these FAANG-like
| companies spend billions of dollars of engineering reinventing
| partitioned databases, but with HTTP as the query protocol?
|
| They talk about "granular field-level access" -- that's just a
| SELECT statement. They talk about separating the read only
| workloads -- that's read scale out replicas. They talk about
| Arrow and similar technologies that sound an awful lot like SQL
| queries.
| joshuamorton wrote:
| Because you have a hammer and so everything sounds like a nail.
|
| When you discuss a problem at a high enough level of
| abstraction, then yes everything looks like a database, and at
| a high enough level of traffic all databases need to be
| partitioned/replicated.
|
| The much more interesting question would be "why doesn't the
| thing I'm familiar with solve this problem"?
| CSMastermind wrote:
| My hot take is that 90% of software engineering is just
| transforming data from one shape to another and there's only so
| many way to do that.
|
| It doesn't matter if you're syncing between a database and
| server, between a presentation layer and an object model, or
| across two devices over a network.
|
| Fundamentally you're dealing with an impedance mismatch that
| only has so many types of solutions.
| eightysixfour wrote:
| I don't think that's a hot take at all, I'm pretty sure that
| was lesson one in the day at structures course at my school.
| vinkelhake wrote:
| It could be that they're plowing billions of dollars into
| engineering and missing some obvious solution.
|
| I think it's more likely that once you get down the details
| (which are going to much more intricate than a post like this
| can convey), then it may turn out to actually be a hard problem
| that requires a bunch of engineering.
|
| For a relevant example of the "how hard could it be"-mindset:
| See George Hotz's short stint at Twitter.
| wmf wrote:
| Doing 60 million queries per second on a partitioned database
| isn't going to be easy either, and when you have problems
| you'll have to dig into the code of a database you didn't
| develop.
| darth_avocado wrote:
| There's a lot of finer details that influence how you design
| system. Yeah reading data is just SELECT, but there's too many
| things to consider.
|
| - Volume of reads: how many people are reading tweets? - Volume
| of tweets: how many unique tweets are being served to people? -
| Data skew: Elon's tweets see hundreds of millions of reads, vs
| some tweets only see like 10. - Size of data: A tweet looks
| like only a few characters of data, but every tweet has
| probably hundreds of columns of metadata.
|
| These are just a few considerations. Plus, scaling out replicas
| isn't always the feasible after a point. Otherwise, why not
| just spin up 1 server per tweet and keep things simple.
| blipvert wrote:
| Is this why I see the "Something went wrong - Retry" dialogue
| about fifty times a day now?
| jazzyjackson wrote:
| Still takes 3 seconds to load a text post
|
| 9MB resources and 150 requests lmao
___________________________________________________________________
(page generated 2024-09-16 23:00 UTC)