[HN Gopher] Production Twitter on one machine? 100Gbps NICs and ...
___________________________________________________________________
Production Twitter on one machine? 100Gbps NICs and NVMe are fast
Author : trishume
Score : 725 points
Date : 2023-01-07 18:46 UTC (1 days ago)
(HTM) web link (thume.ca)
(TXT) w3m dump (thume.ca)
| knubie wrote:
| > I'm not sure how real Twitter works but I think based on Elon's
| whiteboard photo and some tweets I've seen by Twitter
| (ex-)employees it seems to be mostly the first approach using
| fast custom caches/databases and maybe parallelization to make
| the merge retrievals fast enough.
|
| I think Twitter does (or at some point did) use a combination of
| the first and second approach. The vast majority of tweets used
| the first approach, but tweets from accounts with a certain
| threshold of followers used the second approach.
| KaiserPro wrote:
| > A friend points out that IBM Z mainframes have a bunch of the
| resiliency software and hardware infrastructure I mention,
|
| Sure its expensive, and you have to deal with IBM, who are either
| domain experts or mouth breathers. Sure it'll cost you $2m but!
|
| the opex of running a team of 20 engineers is pretty huge.
| Especially as most of the hard bits of redundant multi-machine
| scaling are solved for you by the mainframe. Redundancy comes for
| free(well not free, because you are paying for it in
| hardware/software)
|
| Plus, IBM redbooks are the golden standard of documentation. Just
| look at this:
| https://www.redbooks.ibm.com/redbooks/pdfs/sg248254.pdf its the
| redbook for GPFS (scalable multi-machine filesystem, think ZFS
| but with a bunch more hooks.)
|
| Once you've read that, you'll know enough to look after a cluster
| of storage.
| twp wrote:
| This post solves all of the easy problems (i.e. make simple stuff
| go fast) and none of the hard problems (i.e. build a system that
| still works when other stuff breaks).
|
| This post is perfect world thinking. We don't live in a perfect
| world.
| agilob wrote:
| The title reminded me about this
| https://www.phoronix.com/news/Netflix-NUMA-FreeBSD-Optimized
| (2019) and this 2 years later:
| https://papers.freebsd.org/2021/eurobsdcon/gallatin-netflix-...
| (2021)
| syoc wrote:
| Latest version: http://nabstreamingsummit.com/wp-
| content/uploads/2022/05/202... (2022)
| [deleted]
| aetimmes wrote:
| (Disclaimer: ex-Twitter SRE)
|
| > There's a bunch of other basic features of Twitter like user
| timelines, DMs, likes and replies to a tweet, which I'm not
| investigating because I'm guessing they won't be the bottlenecks.
|
| Each of these can, in fact, become their own bottlenecks. Likes
| in particular are tricky because they change the nature of the
| tweet struct (at least in the manner OP has implemented it) from
| WORM to write-many, read-many, and once you do that, locking
| (even with futexes or fast atomics) becomes the constraining
| performance factor. Even with atomic increment instructions and a
| multi-threaded process model, many concurrent requests for the
| same piece of mutable data will begin to resemble serial accesses
| - and while your threads are waiting for their turn to increment
| the like counter by 1, traffic is piling up behind them in your
| network queues, which causes your throughput to plummet and your
| latency to skyrocket.
|
| OP also overly focuses on throughput in his benchmarks, IMO. I'd
| be interested to see the p50/p99 latency of the requests graphed
| against throughput - as you approach the throughput limit of an
| RPC system, average and tail latency begin to increase sharply.
| Clients are going to have timeout thresholds, and if you can't
| serve the vast majority of traffic in under that threshold
| consistently (while accounting for the traffic patterns of viral
| tweets I mentioned above) then you're going to create your own
| thundering herd - except you won't have other machines to offload
| the traffic to.
| MichaelZuo wrote:
| What do you think about his interesting comment on the
| possibility of a mainframe?
|
| "I also didn't try to investigate configuring an IBM mainframe,
| which stands a chance of being the one type of "machine" where
| you might be able to attach enough storage to fit historical
| images."
|
| It seems theoretically possible it could accomodate the
| entirety of Twitter in 'one machine'.
| aetimmes wrote:
| It depends on what you (or OP) mean by "one machine".
|
| There was a HPC cluster at Princeton when I worked there
| (which, looking at their website, has since been retired)
| that was assembled by SGI and outfitted with a customized
| Linux unikernel that presented itself as a single OS image,
| despite being comprised several disparate racks of individual
| 2-4u servers. You might be able to metaphorically duct-tape
| enough machines together with a similar technique to be able
| to run the author's pared-down scope within a single OS
| image.
|
| With respect to the IBM z-series specifically - if the goal
| of the exercise is to save money on hardware costs, I'm
| imagining purchasing an IBM mainframe is in direct opposition
| to that goal. :) I'm not familiar enough with its
| capabilities to say one way or the other.
| MichaelZuo wrote:
| Perhaps the hardware cost would be higher with one big
| mainframe, but there could be many subtle advantages that
| would combine to reduce the overall cost, since personnel
| costs make up a huge chunk of total opex.
|
| I don't have enough experience to say whether having the
| entirety of Twitter sit in one really big metal box would
| be perceived to be sufficiently advantageous or not.
| HAL3000 wrote:
| > OP also overly focuses on throughput in his benchmarks
|
| Because OP is a junior developer, he reads a lot of theory and
| blog posts, does a lot of research, but doesn't have much
| practical experience. Just look at his resume and what he
| wrote. As a result, most of what he write about is based on
| what he have read about senior developers doing in the
| companies he have worked for, perhaps he created some
| supporting software for core services but did not design or
| implemented the core, so he doesn't have firsthand experience.
| This is evident to anyone who has actually used DPDK (which is
| ridiculous proposal for Twitter like service in 2023 where you
| have XDP and io_uring, it's not HFT), designed and implemented
| high volume, low latency web services and knows where the
| bottleneck is in that kind of services from experience, theory
| will not give you that intuition and knowledge.
| henning wrote:
| No rate limiting, API data, quote tweets, view count, threads,
| likes, mentions, notifications, ads, video, images, account
| blocking (permanent or TTL), account muting (permanent or TTL),
| word filtering (permanent or TTL), moderation/reporting, user
| profile storage, or the fact that tweets that display show more
| than just the tweet itself. No mention that tweet activity all
| occurs concurrently and therefore the loading script is not at
| all a realistic estimate of real activity.
|
| But sure, go ahead and take this as evidence that 10 people could
| build Twitter as I'm sure that's what will happen to this post.
| If that's true, why haven't they already done so? It should only
| take a couple weeks and one beefy machine, right?
| PragmaticPulp wrote:
| Very cool exercise. I enjoyed reading it.
|
| I see a lot of comments here assuming that this proves something
| about Twitter being inefficient. Before you jump to conclusions,
| take a look at the author's code:
| https://github.com/trishume/twitterperf
|
| Notably absent are things like _serving HTTP_ , not to even
| mention HTTPS. This was a fun exercise in algorithms, I/O, and
| benchmarking. It wasn't actually imitating anything that
| resembles actual Twitter or even a usable website.
| trishume wrote:
| Which I think I'm perfectly clear about in the blog post. The
| post is mostly about napkin math systems analysis, which does
| cover HTTP and HTTPS.
|
| I'm now somewhat confident I could implement this if I tried,
| but it would take many years, the prototype and math is to
| check whether there's anything that would stop me if I tried
| and be a fun blog post about what systems are capable of.
|
| I've worked on a team building a system to handle millions of
| messages per second per machine, and spending weeks doing math
| and building performance prototypes like this is exactly what
| we did before we built it for real.
| PragmaticPulp wrote:
| > Which I think I'm perfectly clear about in the blog post.
|
| Of course. I was commenting here to counter all of the
| comments declaring that this proves Twitter doesn't need all
| of their servers, etc.
|
| It's a fun article, but the comments here interpreting it as
| proving something about Twitter engineers being bad are kind
| of depressing.
| VLM wrote:
| Interesting optimization idea: If 95% of the users are bots, and
| your ML algorithms are smart enough to figure who's bot and who's
| bio, you could save a lot of traffic by not publishing tweets to
| bots as no one is going to read them anyway. Of course if that
| traffic included advertisements, you'd also lose 95% of your ad
| revenue.
|
| The ultimate extension of this "run it all on one machine" meme
| would be to run the bots on the single machine along with the
| service.
| Hasu wrote:
| > Of course if that traffic included advertisements, you'd also
| lose 95% of your ad revenue.
|
| Not so, serving an ad to a bot gains you no revenue, because ad
| networks charge for clicks, not impressions. If a significant
| percentage of your ad clicks are from bots, you're running a
| defective advertising platform and won't have customers for
| long regardless.
| toteno wrote:
| I agree with you overall.
|
| Just a small nitpick: most ad networks optimize for price of
| impression, so at the end of the day they charge for
| impressions (just not always directly).
|
| If your ad has low click rate and average price then it just
| won't be shown, because it's more profitable for an ad
| network to show ad with better click rate or with better
| price (i.e. with better price for impression) .
| PerilousD wrote:
| [flagged]
| fortran77 wrote:
| I've thought about this problem, too, and blocklists seem like a
| hard problem to implement efficiently. I have a few thousand
| users blocked, and several hundred keywords, phrases and emoji.
| How are these processed efficiently?
| Halan wrote:
| Let's hope Elon doesn't read this
| jeffbee wrote:
| I like this kind of exercise. One thing I am not seeing is
| analytics, logs and so forth that as I understand it are
| significant portions of Twitter's production cost story.
| tluyben2 wrote:
| Anyone have a complete list of functional blocks that form
| Twitter? Beyond the obvious and what we see?
| Marazan wrote:
| You need the blocks for the obvious for what we see because
| it is not necessarily obvious to everyone.
|
| Over the last couple of months I've seen comments that
| summarise Twitter as a read-only service that doesn't have
| any real time posting requirements and similarly other
| comments that treat it as a write-only service with no real
| time read / fast search requirements.
|
| Without _all_ the blocks even the simple surface level
| Twitter will have complexity people miss.
| lazyasciiart wrote:
| If it's this cheap to run you don't need analytics because you
| don't need to monetize it, and if it's this simple you don't
| need logs because it'll all work correctly the first time!
| jeffbee wrote:
| How do you implement trending without logs?
| kevingadd wrote:
| "You don't need to monetize it" who's going to fund your
| Twitter-as-a-charity? What happens when the free money goes
| away? Businesses have to pay the bills eventually one way or
| another, you need to plan for that in advance
| Nextgrid wrote:
| If you're a small team running this and can actually
| deliver it with a single machine (or two), just charging a
| few bucks a month for verification should net enough money
| to run it and provide a decent living (and out of 300M
| users, there _will_ be people who would pay).
| kissgyorgy wrote:
| I feel like people writing posts like this never worked in a big
| team at a big company on a big project. It is so obviously
| impossible to do this and Twitter has so many more features users
| will never even see, but sure, re-implement it in a couple
| hundred lines of Rust and Twitter will be saved...
| LastTrain wrote:
| Every intern proposes something cute like this in my org, every
| year.
| threeseed wrote:
| HN unfortunately has a lot of people like George Hotz.
|
| They are knowledgeable to a certain level but they simply
| aren't great engineers who almost always are humble, cautious,
| thoughtful and respectful of the intentions behind what other
| engineers build.
|
| Anyone who thinks they can jump in and replace any tech stack
| without an extensive deep dive of the business requirements,
| design decisions, cost constraints, resource limitations etc
| that drove the choices deserves the pain and unemployment that
| inevitably follows.
| betimsl wrote:
| I'm just curious as to what kind of motherboard this personal
| computer is going to have? I'm asking this because of the limit
| on PCIe bandwidth. 100gbit NIC? How?
| betaby wrote:
| I have some servers in production with
| https://www.nvidia.com/en-us/networking/ethernet/connectx-6/
| There is a an AUX card to load PCI lanes properly, like on this
| picture
|
| https://lenovopress.lenovo.com/assets/images/LP1195/Mellanox...
| tpmx wrote:
| Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf]
|
| https://news.ycombinator.com/item?id=32519881
| throwmeup123 wrote:
| The title is highly misleading for some theoretical
| "exploration".
| dang wrote:
| Ok, we've put a question mark up there to make it more
| explorationy.
| trishume wrote:
| As the author, this sounds good to me! I'll probably even
| change the actual title to match. I originally was going to
| make it a question mark and the only reason I didn't is https
| ://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...
| when I think the answer is probably "could probably be
| somewhat done" rather than "no".
| dang wrote:
| Well this may be the first time that's ever happened :)
|
| Betteridge antiexamples are always welcome. I once tried to
| joke that Mr. Betteridge had "retired" and promptly got
| corrected about his employment status
| (https://news.ycombinator.com/item?id=10393754).
| jiggawatts wrote:
| Something I've found a lot modern IT architects seem to ignore is
| "write amplification" or the equivalent effect for reads.
|
| If you have a 1 KB piece of data that you need to send to a
| customer, ideally that should require _less_ than 1 KB of actual
| NIC traffic thanks to HTTP compression.
|
| If processing that 1 KB takes more than 1 KB of total NIC traffic
| within and out of your data centre, the you have some level of
| _amplification_.
|
| Now, for writes, this is often unavoidable because redundancy is
| pretty much mandatory for availability. Whenever there's a
| transaction, an amplification factor of 2-3x is assumed for
| replication, mirroring, or whatever.
|
| For reads, good indexing and data structures within a few large
| boxes (like in the article) can reduce the amplification to just
| 2-3x as well. The request will likely need to go through a load
| balancer of some sort, which amplifies it, but that's it.
|
| So if you need to process, say, 10 Gbps of egress traffic, you
| need a total of something like 30 Gbps at least, but 50 Gbps for
| availability and handling of peaks.
|
| What happens in places like Twitter is that they go _crazy_ with
| the microservices. Every service, every load balancer, every
| firewall, proxy, envoy, NAT, firewall, and gateway adds to the
| multiplication factor. Typical Kubernetes or similar setups will
| have a minimum NIC data amplification of 10x _on top of_ the 2-3x
| required for replication.
|
| Now _multiply_ that by the crazy inefficient JSON-based
| protocols, the GraphQL, an the other insanity layered on to
| "modern" development practices.
|
| This is how you end up serving 10 Gbps of egress traffic with
| _terabits_ of internal communications. This is how Twitter
| apparently "needs" 24 million vCPUs to host _text chat_.
|
| Oh, sorry... text chat with the occasional postage-stamp-sized,
| potato quality static JPG image.
| threeseed wrote:
| Your point makes sense if you have no idea how Twitter works.
|
| It needs to assemble tweets internally, sort them with an ML
| model, add in relevant ads and present a single response to the
| user because end-user latency matters.
|
| And each of these systems eg. ads has their own features,
| complexities, development lifecycle and scaling requirements.
| And of course deploying them continuously without downtime.
| That is how you end up with disparate services and a lot of
| them for redundancy reasons.
|
| I know you think you're smarter than everyone at Twitter. But
| those who really know what they are doing have a lot more
| respect for the engineers who built this insanity. There are
| always good intentions.
| [deleted]
| ilyt wrote:
| > I know you think you're smarter than everyone at Twitter.
| But those who really know what they are doing have a lot more
| respect for the engineers who built this insanity. There are
| always good intentions.
|
| You ignored one possibility - that twitter engineers, or
| people managing them _might_ be just incompetent and all of
| that _might_ just be overly complex POS
|
| There is that weird disgusting trend to assume just because
| company got big that means the tech choices were immaculate,
| and not everything else there is to successful companies.
|
| You can make perfectly well doing company on totally mediocre
| product that hit the niche at right time
| threeseed wrote:
| The core of Twitter is open-source:
| https://github.com/twitter
|
| They wrote Finagle which is a very well written and highly
| regarded Scala micro-service framework that is used by
| other companies such as LinkedIn, Soundcloud etc. Likewise
| Pelikan is an excellent caching service.
|
| We know that Scala is a proven, high-performance, type-safe
| language that is optimised for concurrency and stability.
| So it's not like their tech stack is written in Ruby and
| needs to be re-written in Rust or Go. It's already
| performant.
| thriftwy wrote:
| I remember Stack Overflow running on a single Windows Server box
| and mocking fellow LAMP developers with their propensity towards
| having dozens of VMs to same effect.
|
| That was some time ago, though.
| Nextgrid wrote:
| I believe that's still the case:
| https://stackexchange.com/performance
|
| 9 web servers to serve the entire network. I wish more
| developers were aware of just how performant modern (non-cloud)
| hardware is.
| sgt wrote:
| SO only has two proxy servers, in which one is a failover? I
| bet a lot of people wouldn't believe that if you casually
| mentioned it.
| TacticalCoder wrote:
| TFA, to me, touches about something I've wondered about a very
| long time ago: what are the implications of CPU and storage
| growing at _much_ faster rates than human population?
|
| Back in the 486 days you wouldn't be keeping, in RAM, data about
| every single human on earth (let's take "every single human on
| earth" as the maximum number of humans we'll offer our services
| to with on our hypothetical server). Nowadays keeping in RAM,
| say, the GPS coordinates of every single human on earth (if we
| had a mean to fetch the data) is doable. On my desktop. In RAM.
|
| I still don't know what the implications are.
|
| But I can keep the coordinates of every single humans on earth in
| my desktop's RAM.
|
| Let that sink in.
|
| P.S: no need to nitpick if it's actually doable on my desktop
| _today_. That 's not the point. If it's not doable today, it'll
| be doable tomorrow.
| narag wrote:
| _TFA, to me, touches about something I 've wondered about a
| very long time ago..._
|
| It was a little more than ten years ago for me. I realized that
| a hard disk could store a database of every human alive,
| including basic information, links (family relations) and maybe
| a photo.
|
| _I still don 't know what the implications are._
|
| Maybe we don't want to know, but it's not really that difficult
| to think about.
| doublepg23 wrote:
| Is the storage really the complex part? Isn't gathering the
| actual information and avoiding errors (ex: I have a co-
| worker who's name is incorrectly spelled three different ways
| in prod services) the hard part?
| withinboredom wrote:
| Data hygiene is a problem everywhere. Most companies I
| worked at just throw out 'bad data' and file a bug.
| Occasionally, if the data is a secondary source, it will be
| recreated from a primary source (assuming the primary
| source is still available).
|
| In the particular case, your coworker would be stored by
| some identifier (like an SSN or similar) and their name
| would be stored as "aliases" and allow multiple names. I
| have two nicknames that I answer to, depending on when in
| my life you met me, and my family calls me by my proper
| name. Online, I go by several handles depending on whether
| I want the reader to be able to figure out my real name. I
| even used to work somewhere where I was called by this
| handle (withinboredom) more than my real name.
| narag wrote:
| _Is the storage really the complex part? Isn 't gathering
| the actual information and avoiding errors (...) the hard
| part?_
|
| Not for someone that works in consulting, or at least it
| wasn't. I remember that I had Access access to the
| production database that stored all the customers, present
| and pass. They wouldn't give me a password to that, but
| apparently they thought it was safe to enter the password
| without me looking so I could try some queries and test my
| last fixes.
|
| Not sure if it still works the same, but I did some dumb
| query, left the computer on and, next day morning, a
| temporal file was in my %TEMP% with a lot of data of
| millions of persons worldwide. Had I be so inclined, with
| an external hard disk I could have started my homegrown NSA
| project.
|
| Now think of this: how many times have you heard that the
| data of millions of customers were on sale after a data
| breach? Do you doubt that, let's say, China has every
| single person in the West on file?
|
| Our own governments have us legally or semi-legally
| (exchange) anyway.
| pornel wrote:
| Thanks to Snowden's leaks we know one of those is surveillance.
|
| From scanning every message of every person, it's going to
| expand to recognizing every face from every camera, and
| transcribing and analyzing every spoken word recorded.
| kaba0 wrote:
| Hold it in memory? Sure. But actually working with it besides
| copying it around and doing some operation on a "row-by-row"
| basis? Not really.
| hacker_9 wrote:
| Wait until you hear about DNA
| Waterluvian wrote:
| In undergrad I had a bonus GIS lab assignment to complete the
| task using the prof's instructions from the 80s.. maybe 90s? (a
| lot of FORTRAN) and then complete it again using modern GIS
| software. Such an eye opener. The thing that stuck out the most
| to me was how many hoops we jumped through to batch the job and
| commit the incremental results to disk because, bahahahha,
| fitting even 1% of it in RAM was out of the question.
| moeny wrote:
| 's/(human|desktop)/smartphone/g'
| untech wrote:
| Using two 32 bit numbers for coordinates, each record would
| take 8 bytes, which is 64 gigabytes for 8 billion population.
| Don't think many smartphones have this RAM today.
| renonce wrote:
| You can build that phone. Phones made an engineering
| decision to put more battery rather than more RAM, and it's
| just a matter of putting more LPDDR5 chips onto the circuit
| board.
| Dylan16807 wrote:
| The planet has 2^47 square meters of surface, so more like
| 6 bytes.
|
| Plus you can group together people in the same area and/or
| sort positions as integers and store only the deltas
| between them, so you can probably get down to 2-3 bytes per
| person.
|
| And you can get dozens of models of smartphone with 16GB of
| RAM right now. So there might be a gap there but it's a
| very small gap. The phone of tomorrow will have the RAM.
|
| Edit: Thinking about it more, with 2^33 people and 2^47
| locations the average delta would be 2^14, and it's pretty
| easy to guarantee that fits into 2 bytes per person. And
| with a more accurate world population count you'd free up
| at _least_ a gigabyte for your phone to actually operate
| with.
| cellis wrote:
| Don't you need 3 numbers? Unless you believe in a flat
| earth ;). Also you need some slack space for metadata.
| Let's call it 100GB all in.
| toast0 wrote:
| Lat long and assuming at ground gets you most of the way
| there.
| Dylan16807 wrote:
| "Add it all up, and the US has around 340 billion square
| feet of building stock[3]. This is about 12,200 square
| miles, or 0.00032% of US land area"
|
| Judging by that, you need a negligible increase in the
| number of locations you can represent to handle
| everywhere stable and off the ground someone could be.
| Much less than one bit per person.
|
| If you want to deal with people currently in airplanes
| then you could give them an extra couple bytes. It's less
| than a million people so it won't affect your total
| storage at all.
| ilyt wrote:
| Most people don't live in caves or in the sky city, 2 is
| enough.
|
| And in places you care (multi-floor buildings) you aren't
| getting GPS signal inside anyway...
| withinboredom wrote:
| GPS altitude is notoriously bad via consumer devices.
| It's better to store altitude as less than 2 bytes with
| the range from being a few km below sea level to the
| maximum altitude consumer devices will report (3,000 km
| -- though maybe less because there is also a speed limit
| at which they will stop reporting too so you can't just
| buy a GPS device and build a guided missile).
| saalweachter wrote:
| A bacteria has order of 50 billion atoms. (Eukaryotic human
| cells, 100 trillion.)
|
| That's getting to the point you could store 20 bytes per atom
| in a terabyte.
|
| (The big bottleneck is that you need picosecond resolution
| simulation steps and to cover minutes to see a protein fold.)
| varunkmohan wrote:
| Good analysis. Obviously, this doesn't handle cases like
| redundancy and doesn't handle some of other critical workloads
| the company has. However, it does show how much real compute
| bloat these companies actually have -
| https://twitter.com/petrillic/status/1593686223717269504 where
| they use 24 million vcpus and spend 300 million a month on cloud.
| judge2020 wrote:
| On the other hand, Twitter does (or did) handle over 450
| million monthly active users (based on stats websites), with a
| target for 315 monetizable daily active users (based on their
| earnings calls pre-privatization). Handling that amount of
| concurrency and beaming millions of tweets a day to home feeds
| and notifications is going to be logistically hard.
| therealdrag0 wrote:
| Anyone know if MAU includes API clients? Cuz there's a lot of
| apps built on top of twitter that pull data right? Thats add
| a lot of traffic too.
| saagarjha wrote:
| Active users includes those, monetizable active users
| doesn't.
| WJW wrote:
| Is that 315 million monetizable DAUs? That sounds like a lot
| if the total is only 450 MAU. OTOH, 315k DAU seems like it
| wouldn't be enough to pay the bills.
| judge2020 wrote:
| There were some quarters with profit, some without; the
| past few years were mostly without IIRC.
|
| They were targeting 315 mDAUs for Q4 2023, but in the final
| earnings it was only 238 mDAUs. Actual MAU stats weren't
| public iirc but some random stats sites seemed to say 450m
| global MAUs, which likely includes people with no ad
| preferences or who only view NSFW content (which can't be
| shown next to (most?) ads).
|
| https://www.forbes.com/sites/johnkoetsier/2022/11/14/twitte
| r...
| saagarjha wrote:
| This includes users who aren't shown ads, like people who
| are logged out or use third party clients.
| varunkmohan wrote:
| Posted this on a comment above but systems like Whatsapp
| likely sent an insane amount of data as well but used only 16
| servers over 1.5 billion users at time of acquisition. Modern
| NICs can handle millions of requests a second - I still feel
| there is a lot of excess here.
| veec_cas_tant wrote:
| Feels like the comparison is irrelevant. I'm guessing
| WhatsApp would have infrastructure challenges if all of
| their chats were group messages including the entirety of
| their user base, search, moderation, ranking, ads, etc.
| Isn't WhatsApp more comparable to only DMs?
| PragmaticPulp wrote:
| > However, it does show how much real compute bloat these
| companies actually have
|
| No, it doesn't. It's a fun exercise in approaching Twitter as
| an academic exercise. It ignores all of the real-world
| functionality that makes it a business rather than a toy.
|
| A lot of complicated businesses are easy to prototype out if
| you discard all requirements other than the core feature. In
| the real world, more engineering work often goes to ancillary
| features that you never see as an end user.
| varunkmohan wrote:
| Genuinely asking, why do you think Twitter needs 24 million
| vcpus to run?
|
| This is not apples to apples but Whatsapp is a product that
| entirely ran on 16 servers at the time of acquisition (1.5
| billion users). It really begs the question why Twitter uses
| so much compute if there are companies that have operated
| significantly more efficiently. Twitter was unprofitable
| during acquisition and spent around half their revenue on
| compute, maybe some of these features were not really
| necessary (but were just burning money)?
| ignoramous wrote:
| > _This is not apples to apples but Whatsapp is a product
| that entirely ran on 16 servers at the time of acquisition
| (1.5 billion users)._
|
| - 450m DAUs at the time of facebook acquisition [0]
|
| - Twitter is not just DMs or Group Chat.
|
| > _It really begs the question why Twitter uses so much
| compute if there are companies that have operated
| significantly more efficiently._
|
| A fair comparision might have been Instagram: While Systrom
| did run a relatively lean eng org, they never had to
| monetize and got acquired before they got any bigger than
| ~50m?
|
| [0] https://www.sequoiacap.com/article/four-numbers-that-
| explain...
| no_way wrote:
| Chat apps are mostly one on one interaction, it is much
| harder run an open platform where every user can
| potentially interact with every other user, not even
| talking about search and how complex it gets. If Twitter is
| bloated or not is a valid discussion, but comparison it to
| WhatsApp is not.
| jiggawatts wrote:
| Ironically the one-to-many broadcasts are much easier to
| implement on a single box than as a scalable service
| spread across thousands of small container instances.
| xwdv wrote:
| Whatsapp used Erlang.
| metadat wrote:
| Can you expand on why this is worth pointing out in this
| context?
| xwdv wrote:
| WhatsApp used Erlang, a programming language designed for
| building concurrent, distributed systems, to help scale
| its messaging platform to support millions of users.
|
| Erlang is particularly well-suited to building
| distributed systems because it was designed to handle
| failures at the process level rather than the hardware
| level. This means that if one part of a distributed
| system built with Erlang fails, the rest of the system
| can continue to operate without interruption.
|
| This is critical for a messaging platform like WhatsApp,
| which needs to be able to handle millions of users
| sending messages simultaneously without experiencing any
| downtime. Additionally, Erlang's concurrency features
| allow it to support many thousands of simultaneous
| connections on a single machine, which is also important
| for a messaging platform that needs to be able to handle
| a large volume of traffic.
| sangnoir wrote:
| > This is not apples to apples but Whatsapp is a product
| that entirely ran on 16 servers at the time of acquisition
| (1.5 billion users).
|
| WhatsApp is mostly a bent pipe connecting user devices &
| relying on device storage. If WhatsApp had to implement
| Twitter functionality and business model (with same
| Engineers and stack), they'd need a lot more servers too.
| I'd hazard the number of servers would be in the same order
| of magnitude
| Snoozus wrote:
| Because the actual product is not showing people tweets but
| to optimize who to show which ads based on their previous
| interactions with the site. This is many orders of
| magnitude harder.
| danpalmer wrote:
| Being E2E encrypted, WhatsApp can't do much with the
| content, so it is much closer to a naive bitshuffler than
| Twitter.
|
| Twitter, while still not profitable (maybe it was in some
| recent quarters?) was much closer to it, having all the
| components necessary to form a reasonable ad business. For
| ads, analytics is critical, plus all the ad serving, plus
| it's a totally different scale of compute being many to
| many rather than one to ~one.
| peterhunt wrote:
| The real answer is twofold:
|
| 1. Lots of batch jobs. Sometimes it's unclear how much
| value they produce / whether they're still used.
|
| 2. Twitter probably made a mistake early on in taking a
| fanout-on-write approach to populate feeds. This is super
| expensive and necessitates a lot of additional
| infrastructure. There is a good video about it here:
| https://www.youtube.com/watch?v=WEgCjwyXvwc
| mapme wrote:
| They did not spend half their revenue on compute. It's more
| like 20-25% for running data enters/staff for DCs. Check
| their earnings report.
|
| Whats app is not an applicable comparison because messages
| and videos are stored on the client device. Better to look
| at Pinterest and snap, which spend a lot on infra as well.
|
| The issue is storage, ads, and ML to name a few. For
| example, from 2015:
|
| " Our Hadoop filesystems host over 300PB of data on tens of
| thousands of servers. We scale HDFS by federating multiple
| namespaces."
|
| You can also see their hardware usage broken down by
| service as put in their blog.
|
| https://blog.twitter.com/engineering/en_us/topics/infrastru
| c...
|
| https://blog.twitter.com/engineering/en_us/a/2015/hadoop-
| fil....
| therealdrag0 wrote:
| Also Search (the article did says these wouldn't fit to
| be fair but the discussion seems to be ignoring how much
| wouldn't fit and why). Search is pretty expensive
| especially since to have it responsive you need the
| indexes to fit in memory--at least the Lucene variety,
| which at least in old YouTube videos Twitter used.
| marcusae313 wrote:
| Twitter still uses Lucene. They built a custom codec for
| it, but doesn't change the cost too much.
| d23 wrote:
| Presumably there's an entire data engineering / event
| processing pipeline that's being used to track user
| interactions at a fine grained level. These events are
| going to be aggregated and munged by various teams for
| things like business analytics, product / experiment
| feature analysis, ad analysis, as well as machine learning
| model feature development (just to name a few massive ones
| off the top of my head). Each of these will vary in their
| requirements of things like timeliness, how much compute is
| necessary to do their work, what systems / frameworks are
| used to do the aggregations or feature processing, and
| tolerance to failure.
|
| > This is not apples to apples but Whatsapp
|
| And yeah, whatsapp isn't even close to an apt comparison.
| It's a completely different business model with vastly
| different engineering requirements.
|
| Is Twitter bloated? Perhaps, but it's probably driven by
| business reasons, not (just) because engineers just wanted
| to make a bunch of toys and promo projects (though this
| obviously always plays some role).
| [deleted]
| ricardobeat wrote:
| And for the most part, this herculean effort is wasted.
| Most people just want to see latest tweets from people
| they follow. Everything else is fluff to manipulate
| engagement metrics, pad resumes and attempt to turn
| twitter into something it's users never wanted.
| veec_cas_tant wrote:
| Just guessing, but a lot of the resources are probably
| devoted to making money for the business, not padding
| resumes. Others have pointed it out, but showing tweets
| doesn't generate revenue without additional
| infrastructure.
| sofixa wrote:
| Most people probably follow more people than they're
| capable of reading all the latest tweets of, so some sort
| of ranking/prioritisation makes total sense. And Twitter
| is ad funded, so they need to also show relevant ads
| where it makes sense/money.
| vidarh wrote:
| Whatsapp is mostly message delivery from one person to a
| tiny group. It's absolutely trivial compared to Twitter.
|
| In terms of what Twitter uses compute on, I'd guess
| analytics (Twitter measures "everything" for ad serving -
| go explore ads.twitter.com and analytics.twitter.com) and
| non-chronological timeline mixing both takes orders of
| magnitude more resources than the basic functionality.
| threadweaver34 wrote:
| Whatsapp doesn't do ranking.
| Spooky23 wrote:
| Because at that time, Whatsapp didn't do all of the stuff
| that you need to do. No lawful intercept, no abuse
| detection/prevention, etc.
|
| I don't know enough about Twitter to assess their
| infrastructure, but I know that it easy to run lean until
| there's a problem, and then you get trapped.
| BeefWellington wrote:
| I'm going to preface this criticism by saying that I think
| exercises like this are fun in an architectural/prototyping code-
| golf kinda way.
|
| However, I think the author critically under-guesses the sizes of
| things (even just for storage) by a reasonably substantial
| amount. e.g.: Quote tweets do not go against the size limit of
| the tweet field at Twitter. Likely they are embedding a tweet
| reference in some manner or other in place of the text of the
| quoted tweet itself but regardless a tweet takes up more than 280
| unicode characters.
|
| Also, nowhere in the article are hashtags mentioned. For a system
| like this to work you need some indexing of hashtags so you
| aren't doing a full scan of the entire tweet text of every tweet
| anytime someone decides to search for #YOLO. The system as
| proposed is missing a highly critical feature of the platform it
| purports to emulate. I have no insider knowledge but I suspect
| that index is maybe the second largest thing on disk on the
| entire platform, apart from the tweets themselves.
| trishume wrote:
| Quote tweets I'd do as a reference and they'd basically have
| the cost of loading 2 tweets instead of one, so increasing the
| delivery rate by the fraction of tweets that are quote tweets.
|
| Hashtags are a search feature and basically need the same
| posting lists as for search, but if you only support hashtags
| the posting lists are smaller. I already have an estimate
| saying probably search wouldn't fit. But I think hashtag-only
| search might fit, mainly because my impression is people doing
| hashtag searches are a small fraction of traffic nowadays so
| the main cost is disk, not sure though.
|
| I did run the post by 5 ex-Twitter engineers and none of them
| said any of my estimates were super wrong, mainly just brought
| up additional features and things I didn't discuss (which I
| edited into the post before publishing). Still possible that
| they just didn't divulge or didn't know some number they knew
| that I estimated very wrong.
| busymom0 wrote:
| I don't think that hashtags are a search only feature. In the
| posts themselves, the hashtags are clickable to view other
| tweets. I don't think that qualifies as a search.
| mattbillenstein wrote:
| It does strike me as a feature you'd typically serve out of
| some sort of search index since if you had to build search,
| you'd essentially get indexing of hashtags "for free"
| busymom0 wrote:
| You are probably right and I am wrong. I just looked at a
| tweet and clicking the hashtag takes to the search page
| with that hashtag typed in. Probably implemented
| similarly behind the scenes. Though hashtag most likely
| does an exact match search instead of fuzzy searching for
| regular words and phrases.
| herval wrote:
| it does case matching (#hashTag === #hashtag ===
| #HashTag) too
| [deleted]
| mr90210 wrote:
| [flagged]
| Aeolun wrote:
| I'm not sure why you are asking him to do the thing you've
| literally quoted him doing?
| literallyroy wrote:
| I believed he was saying the author should run his idea
| of indexes on disk taking up a lot of space by the
| engineers.
| 867-5309 wrote:
| >Absence of evidence is not evidence of absence.
|
| >possible that they just didn't divulge
| lelandfe wrote:
| If an inspector reviews your house and finds no issues,
| that is indeed evidence of absence.
| quickthrower2 wrote:
| But evidence is not necessarily proof.
| lelandfe wrote:
| Sure, but if someone accuses your house of having issues,
| and you retort that you've had it inspected by
| professionals, a reply of "Hah! That's _evidence_ , not
| proof!" is just a bit smarmy.
| pdpi wrote:
| A few weeks ago there was in incident[0] in Jersey, where
| some people called fire fighters one evening because they
| could smell gas, the fire fighters didn't find any leaks,
| and the building literally blew up the next morning.
| Experts make mistakes, and failing to understand that
| evidence != proof can literally kill people. Sometimes,
| making the distinction is smarmy; other times, it's just
| being sensible.
|
| 0. https://news.sky.com/story/amp/jersey-tower-explosion-
| questi...
| yjftsjthsd-h wrote:
| Okay, but... we're spit balling database sizes. None of
| this is safety critical, or even in the general
| neighborhood of things where it's important enough to go
| and mathematically prove that our numbers are perfect.
| [deleted]
| saagarjha wrote:
| Sure, but it's not definitive evidence.
| ninth_ant wrote:
| This is critically wrong, and misses the point of the
| cliche entirely.
|
| Absence of evidence, in your case via a clean building
| inspection, does not mean the building is safe. It just
| means the checklist of known items was considered and
| nothing bad found.
|
| Ask a building inspector if their clean report proves
| nothing is wrong with the building.
|
| They will be firm and quick to inform you that it's not a
| warranty -- anything not checked was not covered. Items
| not covered could still be significant problems.
|
| That's the whole point of the saying. Absence of evidence
| is not evidence of absence.
| lelandfe wrote:
| I believe you have conflated "proving a negative" with
| "evidence of absence".
| icedchai wrote:
| Not necessarily. The inspector could be corrupt, bad at
| their job, whatever.
| sulam wrote:
| You're missing metadata in your size estimates.
| BeefWellington wrote:
| I think the difficult part would be that tagging and indexing
| the relationship between a single tweet and all of its
| component hashtags (which you would then likely want metrics
| on to avoid needing to count indexes on, etc.) is where it
| would really start to inflate.
|
| Another poster dug into some implementation details that I'm
| not going to go into. I think you could shoehorn it into an
| extremely large server alongside the rest of your project but
| then you're looking at processing overhead and capacity
| management around the indexes themselves starting to become a
| more substantial part of processing power. Consider that for
| each tweet you need to break out what hashtags are in it,
| create records, update indexes, and many times there's
| several hashtags in a given tweet.
|
| When I last ran analytics on the firehose data (ca. 2015/16)
| I saw something like 20% of all tweets had 3 or more
| hashtags. I only remember this fact because I built a demo
| around doing that kind of analytics. That may have changed
| over time obviously, however without that kind of information
| we don't have a good guesstimate even of what storage and
| index management there looks like. I'd be curious if the
| former Twitter engineers you polled worked on the data
| storage side of things. Coming at it from the other end of
| things, I've met more than a few application engineers who
| genuinely have no clue how much work a DBA (or equivalent)
| does to get things stored and indexed well and responsively.
| panarky wrote:
| Twitter has full-text search, not just hashtags.
|
| Also, the big data storage isn't text, it's images and
| videos.
| mrtksn wrote:
| Twitter's "linked" tweets seems to be strangely unattached from
| context.
|
| What I mean is, Twitter seems to be processing data based on
| whatever it is in the tweet and doesn't maintain some grand
| coherent database.
|
| So I changed my Twitter handle and opened a new account with my
| original Twitter handle and to my surprise, I was receiving
| notifications of engagement with tweets my old account sent
| previously.
|
| I also heard that a method for spamming Twitter trending topics
| is to send tweets and delete them quickly.
|
| My impression is that Twitter is big on real time processing.
| They definitely don't search the entire database for #YOLO
| tweets, instead they seem to be searching the almost-live stuff
| and some archived stuff(probably ranked and saved as noteworthy
| tweets or something).
| madeofpalk wrote:
| Old stuff on twitter is weird. Tweets seem to eventually
| forget that _You_ specifically liked a tweet, and will allow
| you to like a tweet again, and the like count will reflect
| you liking it twice.
| kaba0 wrote:
| Like count is almost surely just a locally incremented
| number, your like request will eventually get processed and
| will be dropped if it was already liked. It only has to be
| an eventually consistent value.
| madeofpalk wrote:
| This is beyond optimistic updates (which twitter also has
| plenty of issues with - you'll reply to your tweet and
| it'll jump immediately to two replies, but on refresh
| just show your one).
| walrus01 wrote:
| > I have no insider knowledge but I suspect that index is maybe
| the second largest thing on disk on the entire platform
|
| I really wonder how much of a challenge this is and how much it
| occupies, not even talking about disk, but continuing the
| theoretical exercise in the linked URL, you can get 1U size
| servers with 2TB of RAM these days.
| vidarh wrote:
| Biggest problem with this is the lack of considering
| analytics.twitter.com and ads.twitter.com. Twitter stores event
| data about _everything_ that happens to a tweet, and lets you
| target ads with a lot of precision.
|
| While some of those writes may well be acceptable to lose,
| letting you write to caches, effectively you need to assume
| there are more analytics events triggering writes to
| _something_ than there are tweet views.
| teaearlgraycold wrote:
| Analytics data can easily be 90% of your data.
| social_quotient wrote:
| Phrased differently for effect. Analytics data can be 9x
| larger than core data.
| wtallis wrote:
| A Twitter-like service that fits on a single server could
| probably get by with the reduced revenue that comes with not
| offering obsessively fine-grained analytics and ad targeting.
| throwayyy479087 wrote:
| This is true from the POV of capitalism but not
| institutions. This would be a hyper profitable but
| politically unsustainable business.
| serverholic wrote:
| I think about this often. Specifically, how much bloat
| exists in the world because individuals in our society are
| forced to justify their existence on a daily basis.
|
| Everyone has to be employed so it's better to keep adding
| more crap to products and make those products disposable in
| order to give people a job.
| serial_dev wrote:
| If your team is full of engineers whose specialty is
| running clusters with 1K+ machines, you'll get solutions
| that need clusters with 1K+ machines.
| anticristi wrote:
| Not to mention that your business would end up being more
| profitable by avoiding GDPR fines. Most users in the EU
| would click "Reject all" anyway, so basically that code is
| not needed.
| icedchai wrote:
| True. You'd also save a ton of less operations and
| engineering staff.
|
| Running anything on a single server, however, is really a
| non starter for anything remotely serious. What do you do
| if you need to do an OS update? I suppose you could just
| never do those, like a former employer (1000+ day
| uptimes...)
| kijin wrote:
| Compare the cost of operating multiple servers, on one
| hand, with the lost revenue from having weekly or monthly
| maintenance windows during which you just put up a Fail
| Whale page. Most people overestimate the latter by a huge
| margin.
| IshKebab wrote:
| That's fine if your service is really local - you can do
| it at night. Not really an option for a global site.
| Imagine if Twitter went down for a few hours every month.
| People are addicted to Twitter. It might be at a critical
| time for an entire country (e.g. the Queen dies). Even
| worse you can't guarantee how long the upgrade will take.
|
| You'd definitely need at least two servers. But I think
| you could surely just have simple master/slave
| replication and switch between them.
| kijin wrote:
| Yeah, I was just responding to the "how about OS updates"
| part of the parent comment, for which scheduled downtime
| is a reasonable option. To protect your service from
| _unscheduled_ downtime though, like a failing RAID array,
| you would need at least two servers.
|
| Personally I wouldn't run a critical service on only one
| server, but two servers? Definitely doable. I actually
| have a service running on two servers in different DCs
| 700 miles apart. Zero downtime in 9 years. :)
| glass3 wrote:
| Twitter is not serious enough that there cannot be a
| timeout.
|
| With a second OS partition, the server can alternate
| between the working copy and a copy that is updated in a
| VM. For a free service, customers cannot complain even if
| there is a reboot every day and the service is down for a
| couple of minutes.
|
| Realistically, there would be a mirroring server to be
| prepared for hardware failures. One server can be
| restarted while the other is the main server.
| Sebb767 wrote:
| > With a second OS partition, the server can alternate
| between the working copy and a copy that is updated in a
| VM. For a free service, customers cannot complain even if
| there is a reboot every day and the service is down for a
| couple of minutes.
|
| Are you seriously suggesting that a service (the size of
| Twitter, no less) has an acceptable downtime of a few
| _minutes_ a day?
|
| > Realistically, there would be a mirroring server to be
| prepared for hardware failures. One server can be
| restarted while the other is the main server.
|
| But for that mirroring, you need to replicate disk
| writes, databases, backups etc.. This additional load
| would easily bring the server to a point where a single
| server would no longer suffice, even an insanely spec'ed
| one.
| vidarh wrote:
| > has an acceptable downtime of a few minutes a day?
|
| We kinda know the answer to this: Twitter was struggling
| with harm to its reputation for a _long_ time because of
| regular Fail Whales. It absolutely was a huge problem for
| them at the time.
| glass3 wrote:
| There is a difference between open-ended, unscheduled
| downtime and a known 5 minute reboot window. If I can
| choose between an ad-free service that reboots at noon
| and a downtime-free service with ads, I would choose the
| ad-free service.
|
| Who needs Twitter to be a service without any downtime?
| Sebb767 wrote:
| > You'd also save a ton of less operations and
| engineering staff.
|
| You absolutely would not. The cost of having developers
| put out extremely optimized code (due to the scaling
| limits) and cuddle that single server to never, ever fail
| easily eclipses the cost of a having a multiple servers
| by a few orders of magnitude.
|
| EDIT: To the downvoters, I'd really love to see the
| calculation on how engineering time would be cheaper than
| buying a second server in any reasonable timeframe.
| icedchai wrote:
| I was comparing the costs to building/operating on a
| single server to the real Twitter. Twitter is a massive,
| distributed, hugely complex system, which requires a
| large team to build, maintain, and operate. That costs
| $$$. Both for people and servers.
|
| (You are right, servers are cheap compared to employee
| costs though.)
| femiagbabiaka wrote:
| Twitter actually used to have downtime during certain
| updates IIRC. The state department asked them to skip one
| such maintenance in service of some Iranian protests in
| 2009. I highly doubt that the era of US ops in twitter
| has ended, so downtime is probably a nonstarter.
|
| Source: https://foreignpolicy.com/2009/06/16/state-
| department-interv...
| poslathian wrote:
| Best comment. And one you don't hear nearly enough from
| typical product and eng managers.
| tinyspacewizard wrote:
| Ehhh a large company would much prefer:
| $$$$$$ income on $$$ costs
|
| to $$$ income on $ costs
| gonzo wrote:
| Perhaps. But VCs prefer the first.
| zxcvbn4038 wrote:
| This is absolutely the sort of thing I wish more developers did
| - and I think the good ones already do. Most of what you find
| in blogs will work just fine at 1 request per second (OMG! 1M
| Hits!!) or 10 requests per second (and I think someone did post
| their "how I scaled to 10 million hits per month" blog to
| Hacker News once which sounds impressive until you do the math)
| but when you get into thousands of requests per second you
| really do need to understand the network stack, the different
| storage tiers, your choice of algorithms, how to interact with
| CDNs, etc. a lot more then any blog will tell you.
|
| When interviewing developers I always ask them what is the
| largest public web site they ever worked on and then probe
| about performance issues they encountered and how they resolved
| them in order gague how far along they are in their skill
| development.
|
| I would never plan to run a production service on a single
| server just because coordinating changes in the active dataset
| among two or more production servers often changes your design
| significantly, and you want to plan for that because the
| consumer grade hardware we all use has a nasty habit of not
| working after power cycles (which still tends to be the most
| strain a system goes through, even in a world of SSD storage).
| api wrote:
| I didn't get the impression that this would duplicate the
| _entire_ functionality of Twitter, just what amounts to the MVP
| functionality. If you are only talking about the MVP it 's at
| least somewhat plausible with a lot of careful engineering and
| highly efficient data manipulation.
|
| Adding images, videos, other large attachments, rich search,
| and all the advertising and billing and analytics stuff would
| blow this out of the water, but... maybe not by as much as
| people think...? I would not be surprised if a very
| performance-engineered version of Twitter could run on a few
| dozen racks full of beefy machines like this with HPC-grade
| super-fast fabric interconnects.
|
| I have a strong sense that most large scale systems are way
| less efficient than what's possible. They trade development
| ease, modularity, and velocity for performance by using a lot
| of microservices, flabby but easy and flexible protocols (e.g.
| JSON over HTTP), slow dynamic languages, and layers of
| abstraction that could be integrated a lot more tightly.
|
| Of course that may be a rational trade-off if velocity,
| flexibility, and labor costs and logistics matter more than
| hardware, power, or data center floor space costs.
| BeefWellington wrote:
| I didn't get the impression that this would duplicate the
| entire functionality of Twitter, just what amounts to the MVP
| functionality. If you are only talking about the MVP it's at
| least somewhat plausible with a lot of careful engineering
| and highly efficient data manipulation.
|
| I agree mostly. Where I differ in that I would argue that
| hashtags were _THE_ thing that Twitter is most known for but
| that could be a perspective from having been on the platform
| for forever and a day and I recognize not everyone may make
| that same association anymore.
| saagarjha wrote:
| FWIW my opinion of hashtags (relatively new to Twitter) is
| that they're only used by brands and mostly cringe people
| gorgoiler wrote:
| Isn't a hashtag just another kind of user account -- an account
| from which anyone can post?
|
| The data structures for the @BeefWellington timeline of tweets
| and the one for the #BeefWellington timeline of tweets could
| look roughly the same.
| [deleted]
| bithaze wrote:
| Hashtags aren't like user accounts, no - they're strings that
| are part of a tweet. In theory, a separate data structure
| shouldn't be needed since you can just search the full text
| of tweets, but in practice, I don't know how that scales for
| the number of all-time tweets.
| HyperSane wrote:
| I think he was talking about how hashtags are implemented.
| knorker wrote:
| I would be surprised if indexes were not larger than the raw
| tweets.
|
| Text search, hashtag index, some structured data for popular
| tweets, etc...
|
| In order to deliver search results I would not be surprised if
| tweets are duplicated/denormalized, for quick search/lookup.
| dspillett wrote:
| The hashtag index is not going to be any bigger than the tweets
| storage though, and may be significantly smaller, so this part
| is not if by an order of magnitude (even a binary one).
| Assuming something like a common SQL database is used for
| storage there would be a tags table (one row per unique tag,
| tag string plus a numeric identifier, indexed by both which
| bloats the size a bit but it'll still be small) and a link
| table (one row per tag in a message, containing the tag and
| message ids). Even if using 64-bit IDs because you don't want
| to fall over at 2 thousand million messages (or 4, if your DB
| supports unsigned types or you start at MININT instead of zero
| or one) then that structure is going to consume about 32 bytes
| per tag per message (plus some static overheads and a little
| more for non-leaf index pages of course). _In theory_ this
| could be the same size as the messages table or even larger (if
| most messages contain many small tags), but in practise it is
| going to be significantly smaller.
|
| Yes, this would be big enough to need specifically factoring
| into a real implementation design. But it would not be big
| enough to invalidate the proposed idea so I understand leaving
| it off, at least initially, to simplify the thought experiment.
|
| Similarly to support a message responding to, or quoting, a
| single other you only need one or two NULLable ID references,
| 16 bytes per message, which will likely be dwarfed by the
| message text in the average case. Given it likely makes sense
| to use something like SQL Server's compression options for data
| like this the average load imposed will be much smaller than 16
| bytes/message.
|
| We are fiddling, fairly insignificantly, measurable but to
| massively, with constants a & b in O(a+bN) here, so the storage
| problem is still essentially of order O(N) [where N is total
| length of the stored messages].
| Xeoncross wrote:
| > I think the author critically under-guesses the sizes of
| things (even just for storage) by a reasonably substantial
| amount.
|
| While true, and not to take away from the parent comment, I've
| noticed that the size of things is often partially the result
| of scaling out horizontally. Most companies I've worked at end
| up with a lot of duplicate records as each subsystem might want
| a copy or to cache a copy.
| vidarh wrote:
| This is indeed a problem, and one of the reasons to be
| careful about _knowing_ you can fit things in one machine
| before taking that approach, because the moment you 're
| forced to move to a distributed model it's rarely one machine
| to two but one machine to a dozen coupled with a major
| rearchitecturing effort at just the wrong moment.
|
| It's often fine to start without a fully decoupled system
| (net present value of the time and money needed to scale out
| might be far too high), but you need to know whether or not
| it's likely to come and what to look for so you can start
| preparing in time.
| redbell wrote:
| > _I think the author critically under-guesses the sizes of
| things (even just for storage) by a reasonably substantial
| amount._
|
| I want to add another concept that may impact, considerably,
| the storage, which is "threads". I'm not sure what is the
| percentage of threads/tweets but what I consider an important
| factor is that threads do not have a maximum number of
| characters.
| chippiewill wrote:
| > I have no insider knowledge but I suspect that index is maybe
| the second largest thing on disk on the entire platform, apart
| from the tweets themselves.
|
| I'd probably go as far to say that the indexes _generally_ at
| twitter could be larger than the tweets
| truth_seeker wrote:
| How to subdue the cost of abstraction. Very well explained !
| mcqueenjordan wrote:
| Fun thought experiment! I can't help but be reminded of the Good
| Will Hunting quote, though:
|
| SEAN: So if I asked you about art you'd probably give me the
| skinny on every art book ever written. Michelangelo? You know a
| lot about him. Life's work, political aspirations, him and the
| pope, sexual orientation, the whole works, right? But I bet you
| can't tell me what it smells like in the Sistine Chapel. You've
| never actually stood there and looked up at that beautiful
| ceiling. Seen that.
| drewg123 wrote:
| How much bandwidth does Twitter use for images and videos? Less
| than 1.4Tb/s globally? If so, we could probably fit that onto a
| second machine. We can currently serve over 700Gb/s from a dual-
| socket Milan based server[1]. I'm still waiting for hardware, but
| assuming there are no new bottlenecks, that should directly scale
| up to 1.4Tb/s with Genoa and ConnectX-7, given the IO pathways
| are all at least twice the bandwidth of the previous generation.
|
| There are storage size issues (like how big is their long tail;
| quite large I'd imagine), but its a fun thing to think about.
|
| [1] https://people.freebsd.org/~gallatin/talks/euro2022.pdf
| cortesoft wrote:
| It is way more than 1.4TBs a second globally.
| xyzzy123 wrote:
| I wonder how much is api traffic and how much is assets &
| images.
| koolba wrote:
| I wonder how much of that is crypto spam bots replying to
| each other.
| skywhopper wrote:
| Can you detect and block those spam bots with less effort
| than it would take to process them?
| [deleted]
| koolba wrote:
| That's a not so simple calculation as comparing the raw
| request process time isn't the complete picture. Spam bot
| content must be persistently stored, indexed, archived,
| etc so the long term cost is much more than the one off
| POST to create the entity.
|
| You'd also have to quantify the improved user experience
| from seeing less spam v.s. inflated ad revenue for
| garbage views / content.
| _zoltan_ wrote:
| in this specific discussion it's very important to use Tb and
| TB correctly.
| Gigachad wrote:
| This is never going to happen. If I want to indicate it's
| correct I'd write Tbit/s or Tbyte/s. Otherwise its a coin
| flip if TB and Tb has been used correctly.
| jrockway wrote:
| I like /s for bytes and ps for bits. 100Mbps = 100
| million bits per second. 100MB/s = 100 million bytes per
| second. (The capitalization is important of course. I
| tried writing some examples where it's not preserved and
| it's too weird. I tend to not use capital letters on
| things like Slack, but for bytes, you just have to. The
| difference between milli and Mega can also be important,
| but since nobody talks about the negative powers with
| bandwidth, you are probably OK if your shift key breaks.)
| JosephRedfern wrote:
| I suppose that in practice you'd need to consider burst
| bandwidth and not just 95/99 percentiles.
| quickthrower2 wrote:
| ... HN thread that reinvents CDN ...
| thorncorona wrote:
| Twitter is big on real time though. Each user gets served
| their own feed in real time.
| icedchai wrote:
| This doesn't seem like an issue. You can store images /
| other assets on the CDN, and serve the real time stuff
| from the origin without caching.
| britneybitch wrote:
| > colo cost + total server cost/(3 year) => $18,471/year
|
| Meanwhile the company I just left was spending more than this for
| dozens of kubernetes clusters on AWS before signing a single
| customer. Sometimes I wonder what I'm still doing in this
| industry.
| ummonk wrote:
| Was it more than the salary of even a single software engineer?
| britneybitch wrote:
| We wasted 5-6 figures of salary dollars lovingly building
| those clusters and the automation surrounding them. We had
| blue-green zero downtime deploys, but no customers who would
| notice any downtime to begin with. I think the CTO just
| wanted k8s on his resume.
|
| In one afternoon (at most), I could have written a script to
| deploy our demo with docker compose over ssh. Sure, docker
| compose won't scale forever, but their runway didn't last
| forever either.
| ummonk wrote:
| Fair enough. To be honest, of the abstraction tools out
| there, Kubernetes is the one I'd hold off on as long as
| possible. Load balancing proxies, docker compose, auto-
| scaling, cloud databases, etc. are things I'd do relatively
| early, but not Kubernetes.
| yazzku wrote:
| Ah, the classic excuse for writing shitty software systems.
| tluyben2 wrote:
| Very often. I work with companies spending 10x as much on bad
| (bizarrely complex; indeed kubernetes, lambda, gateway, rds
| etc) setup and bad code on aws. Almost no traffic (b2b). Makes
| no sense at all.
| threeseed wrote:
| If they were a startup like you suggest then it's possible they
| were running on AWS credits.
|
| You can get up to $100k and it's a big reason many startups go
| in that direction.
|
| Also $20k is nothing when you factor in developer time etc.
| Existenceblinks wrote:
| Techies in tech industry are basically eating the rich .. A lot
| of buzzword to suck investment money in.
| paulryanrogers wrote:
| Or rather cloud providers are eating the rich. Techies are
| carrying the plates.
| Existenceblinks wrote:
| It's a big meal. Those old-industry tycoons (oil/real
| estate/finance/alcohol/export/etc.) from gold rush era have
| too much money. There are a few places for money to land.
| Cloud seems to be the first wave, social media, messaging,
| big data, ai, smart everything. $200k-400k + $80k-$150k
| ranges are like hyenas army!
| [deleted]
| traceroute66 wrote:
| > spending more than this for dozens of kubernetes clusters on
| AWS before signing a single customer
|
| Yup.
|
| Cloud is 21st century Nickel & Diming.
|
| Sure _it sounds_ cheap, everything is priced in small sounding
| cents per unit.
|
| But then it _very quickly_ becomes a compounding vicious circle
| ... a dozen different cloud tools, each charged at cents per
| unit, those units often being measured in increments of
| hours....next thing you know is your cloud bill has as many
| zeros on the end of it as the number of cloud services you are
| using. ;-)
|
| And that's before we start talking about the data egress costs.
|
| With colo you can start off with two 1/4 rack spaces at two
| different sites for resilience. You can get quite a lot of bang
| for your buck in a 1/4 rack with today's kit.
| richwater wrote:
| > You can get quite a lot of bang for your buck in a 1/4 rack
| with today's kit.
|
| Until very recently, while money was still very cheap, the
| time overhead it would take to manage this just was not worth
| the cost savings.
|
| Even with the market falling out from under VC, I think it
| still is a good tradeoff for many shops.
| toast0 wrote:
| > Until very recently, while money was still very cheap,
| the time overhead it would take to manage this just was not
| worth the cost savings.
|
| You can also rent a whole server. There's not much
| difference in time in managing a VM in a cloud or a whole
| server you rent from someone. Depending on the vendor,
| maybe some more setup time, since low end hosts don't
| usually have great setup workflows, so maybe you need to
| fiddle with the ipmi console once or twice to get it
| started, but if you go with a higher tier provider, you can
| fully automate everything if that floats your boat. It's
| just bare metal rather than a VM, and typically much lower
| cost for sustained usage (if you're really scaling up
| signfigantly and down throughout the day, cloud costs can
| work out less, although some vendors offer bare metal by
| the hour, too)
| Tepix wrote:
| The new EPYC servers can be filled with 6TB of RAM and 96 cores
| per socket. Fun times.
| summerlight wrote:
| I think many people in this thread are making the mistake of
| ignoring evolutionary factors in system engineering. If a system
| doesn't need to adopt or change, lots of things can be much more
| efficient, easier and simpler, likely the order of 10x~100x. But
| you gotta appreciate that we're all paid because we need to swap
| wheels on running trains (or even engines in flying airplanes). A
| large fraction of demand for redundancy, introspection,
| abstraction and generalization comes from this.
|
| Why do we want to apply ML at the cost of a significant fleet
| cost increase? Because it can make the overall system
| consistently perform against external changes via generalization,
| thus the system can evolve more cheaply. Why do we want to
| implement a complex logging layer although it doesn't bring
| direct gains on system performance? Because you need to inspect
| the system to understand its behavior and find out where it needs
| to change. The list can go on and I can give you hundreds of
| reasons why we need all these apparently unnecessary complexities
| and overheads can be important for systems' longevity.
|
| I don't deny the existence of accidental complexities (probably
| Twitter can become 2~3x simpler and cheaper given sufficient eng
| resource and time), but in many cases you probably won't be able
| to confidently say if some overheads are accidental or essential
| since system engineering is essentially a highly
| predictive/speculative activity. To make this happen, you gotta
| have a precise understanding of how the system "currently works"
| to make a good bet rather than re-imagination of the system with
| your own wish list of how the system "should work". There's a
| certain value on the latter option, but it's usually more
| constructive to build an alternative rather than complaining
| about the existing system. This post is great since the author
| actually tried to build something to prove its possibility, this
| knowledge could turn out to be valuable for other Twitter
| alternatives later on.
| ilyt wrote:
| > A large fraction of demand for redundancy, introspection,
| abstraction and generalization comes from this.
|
| Sure, you need to invest into it but those are things you can
| reuse for every app and feature you build.
|
| And those are not the _reason_ why those systems are so
| complex, those are just ways to keep complex systems running
| and manageable. In most they also do not stand in the way of
| making system better but help in it.
|
| They need to exist _because_ the architecture of system grew
| organically from smaller system over and over again and big
| restructurization was deemed not worth it. It 's "just have a
| bunch more hardware and engineers" vs "we're not delivering
| features and we might not get rewrite right".
|
| And every time you throw money at the problem the problem
| becomes a bigger problem and potential benefits from "getting
| it right" are also getting bigger. But nobody wants to be
| herald that tells management "we 're going to spend 6-12
| months" for somethinkg that have few years of pay-off
| jideel wrote:
| [dead]
| wonnage wrote:
| This doesn't seem to support fetching a specific tweet by id?
| sgt wrote:
| No need, for those you can display a "Twitter is experiencing
| issues" or something similar. That will encourage the user to
| return to the main app and enjoying infinite scroll.
| mattbillenstein wrote:
| Interesting abstract system design type problem - I think it
| becomes difficult if you have to shard the data though because
| all the assumptions about the hot set being in RAM break all the
| performance guarantees now I think... Which is I think basically
| what Twitter's existing backend has to deal with.
| mgaunard wrote:
| All web and cloud technologies are inherently inefficient, and
| most programmers don't know networking or even how hardware works
| sufficiently well to optimize for high througput and low-latency.
|
| There was an article just yesterday about how Jane Street had
| developed an internal exchange way faster than any actual
| exchange by building it from the ground up, thinking about how
| the hardware works and how agents can interact with it.
|
| Modern software like Slack or Twitter are just reinventing what
| IRC or BBS did in the past, and those were much leaner, more
| reliable and snappier than their modern counterparts, even if
| they didn't run at the same scale.
|
| It wouldn't be surprising at all that you could build something
| equivalent to Twitter on just one beefy machine, maybe two for
| redundancy.
| therealdrag0 wrote:
| It's "easy" to optimize for speed when you build from the
| ground up with no real customers or feature requirements that
| you can't just conveniently ignore.
| [deleted]
| drowsspa wrote:
| I'm always reminded of that tweet from @SwiftOnSecurity
| https://mobile.twitter.com/SwiftOnSecurity/status/1485822027...
|
| > Once you understand your computer has 16 cores running at
| 3GHz and yet doesn't boot up in .2 nanoseconds you understand
| everything they have taken from you.
|
| With their infinite VC money at their disposal, and with their
| programmers having 100 GHz machines with thousands of cores,
| 128 TB of RAM and FTL internet connections, tech companies
| don't really have any incentive to actually reduce bloat.
|
| Edit: it's still quite sad. I feel like we had languages with a
| way better future, and more promising programming
| architectures, back in the 80s.
| kaba0 wrote:
| Booting has fundamental hardware limits, though. Will that
| harddrive or whatever really reply all in 0.2 ns?
| drowsspa wrote:
| I mean, it's just a hyperbole but they definitely shouldn't
| need to take seconds to boot either.
| ilyt wrote:
| And most machines don't even touch that limits. I remember
| some presentation about it [1] and actual limits aren't
| even touched, especially on servers.
|
| From what I remember "hard" limits are CPU/DRAM-memory
| initialization and speed of read from flash chip storing,
| and sources of lag include firmware from add-on cards just
| being slow (if RAID controller takes 30 seconds to return,
| and firmware is not running initialization in parallel,
| that's your extra boot time. Or stuff out of left field
| like "IPMI controller logs stuff via serial so if you print
| too much text it slows down". Most BIOSes do things
| painfully parallel too.
|
| * [1] https://www.youtube.com/watch?v=fTLsS_QZ8us
| saagarjha wrote:
| > It wouldn't be surprising at all that you could build
| something equivalent to Twitter on just one beefy machine,
| maybe two for redundancy.
|
| The blog post kind of gets a very cut-down version of Twitter
| running on a single machine. Actual Twitter absolutely would
| not work.
| mgaunard wrote:
| Whatever you need to tell yourself to justify the bloat.
|
| Of course it's a proof-of-concept, it's not a drop-in
| replacement for Twitter.
| jacobsenscott wrote:
| Nice. While there may be some impracticalities to actually doing
| this for twitter, 99% of the software out there could run on a
| fraction of a single commodity server. People complain about the
| carbon burn of crypto, and they are right, but I bet it is
| dwarfed by the carbon burn of all the shitty over-provisioned and
| over-architected CRUD apps running interpreted languages.
| Unfortunately with universities teaching python of all things
| we'll have (or maybe already do have) a whole generation of
| developers that actually have no idea how powerful a modern
| computer is.
|
| I suppose there's a chance AI will get to the point where we can
| feed it a ruby/python/js/whatever code base and it can emit the
| functionally equivalent machine code as a single binary (even a
| microservices mess).
| [deleted]
| etaioinshrdlu wrote:
| I've been experimenting with GPT3 as a "compiler" and it's
| always amazing when it works. Extrapolating from here, I think
| this is right on the nose -- we feed in any high level language
| and get reasonably efficient assembly out the other end,
| furthermore, the assembly is more human-like than a compiler
| would give you.
|
| There's some big problems with this approach today, namely,
| it's not always right, and it may sometimes be half right (miss
| edge cases).
|
| But think of where this AI technology is headed -- it stands to
| reason it will eventually work pretty much perfect.
|
| And then I think we'll see another very strong trend -- large
| AI models replacing other forms of software. Why write a
| compiler when GPT3 can compile C to asm? Why write an
| interpreter when GPT3 can "compile" python to C?
|
| The AI model is hilariously less-efficient than traditional
| software, but it will be far far cheaper and faster to create
| than the traditional equivalent.
|
| What other types of software will be replaced by AI models?
| jbotdev wrote:
| I think people overestimate how much CPU time a typical CRUD
| app spends on actual business logic, even with an interpreted
| language like Ruby or Python. What I've seen is the bottleneck
| is largely memory, such that you can pack a ton of these apps
| on a machine with a few cores and a lot of RAM.
|
| The stuff that actually is CPU-bound often ends up being
| written in an appropriate language, or uses C extensions (e.g.
| ML and data science libraries for Python).
| jacobsenscott wrote:
| Sometimes. But in a rails app, and probably any app using an
| orm, transforming database rows to activerecord objects makes
| the cpu go brrr. And thats all ruby code. A port of AR to
| rust would be amazing, but idk how feasible given all the
| metaprogramming.
| siliconc0w wrote:
| Enjoyed the write up, would be curious to see the twitter spend
| broken down by functionality given all the extra stuff they do. I
| imagine it's a non-linear relationship where the company has to
| burn more and more cash with every new feature (and esp things
| like Advertising which you need once your spend surpasses what a
| simple subscription can offer), more scale adds more complexity,
| bureaucracy and overhead (management, hr & recruiting,
| legal&accounting, etc). While it's likely there is waste (some of
| which is inevitable, see 'overhead' above) a super bareboens
| twitter can maybe run within one beefy machine but a 'real'
| twitter ends up needing millions + lots of people.
| spullara wrote:
| When I was working there I implemented my patent during a hack
| week (given a set of follows return the list of matching tweet
| ids, very similar to his prototype):
|
| https://patents.google.com/patent/US20120136905A1/en (licensed
| under Innovators Patent Agreement,
| https://github.com/twitter/innovators-patent-agreement)
|
| I could have definitely served all the chronological timeline
| requests on a normal server with lower latency that the 1.1 home
| timeline API. There are a bunch of numbers in the calculations
| that he is doing that are off but not by an order of magnitude.
| The big issue is that since I left back then Twitter has added ML
| ads, ML timeline and other features that make current Twitter
| much harder to fit on a machine than 2013 Twitter.
| steviesands wrote:
| A few thoughts. The first is, are we asking the wrong
| questions? Should it be, "If I spend 10m on hardware for
| predicting ads (storage/compute) that generates 25m in revenue,
| should I buy the hardware?". Sure, we can "minify" twitter, and
| it's a wonderful thought experiment, but it seems devoid of the
| context of revenue generation.
|
| The second is, it's interesting to understand social media
| industry wide infra cost per user. If you look at FB, Snap,
| etc. they are within all within an order of magnitude in cost
| per DAU (DAU / Cost of revenue) of each other. This can be
| verified via 10-ks which show Twitter at $1.4B vs. SNAP 1.7B
| Cost of Revenue. The major difference between the platforms is
| revenue per user, with FB being the notable exception.
|
| Also would you summarize the patent/architecture? The link is a
| bit opaque/hard to read.
|
| Note: Cost of Revenue does also include TAC and revenue sharing
| (IIRC) and not just Infra costs but in theory they would also
| be at similar levels.
|
| eg. SNAPs 10-k
| https://d18rn0p25nwr6d.cloudfront.net/CIK-0001564408/da8288a...
| spullara wrote:
| The basic idea of the system was to scan a reverse
| chronologically ordered list of "user id, tweet id",
| filtering out any tweet whose user wasn't in the follow set
| (or sets in the case of scan sharing) until you retrieved
| enough tweets for the timeline request. There are a bunch of
| variants in the patent, but that is the basic idea. At the
| time, I estimated that Twitter was spending 80% of its CPU
| time in the DC doing thrift/json/html
| serialization/deserialization and mused about merging all the
| separate services into a single process. Lot's of opportunity
| for optimization.
| pg314 wrote:
| Could you give an insight into the reasons that such a
| system never replaced the existing implementation?
| steviesands wrote:
| Interesting, 80% seems a bit on the higher end nowadays
| though? For example, Google quantified this as the
| "datacenter tax" and through their cluster wide profiling
| tooling saw that it was 22-27% of all CPU cycles (still a
| huge amount). They go a different route and suggest
| hardware accelerators for common operations. Datacenter tax
| was defined as:
|
| "The components that we included in the tax classification
| are: protocol buffer management, remote procedure calls
| (RPCs), hashing, compression, memory allocation and data
| movement."
|
| https://static.googleusercontent.com/media/research.google.
| c...
| bitbckt wrote:
| Regarding tweet distribution, I was one of the folks who built
| the first scalable solution to this problem at Twitter (called
| Haplocheirus). We used the Yahoo "Feeding Frenzy" design, pushing
| tweets through a Redis-backed caching layer.
|
| Feel free to continue using that (historically-correct) answer in
| interviews. :P
| jasonhansel wrote:
| If you really wanted to run Twitter on one machine at any cost,
| wouldn't an IBM mainframe be much more practical?
|
| You can even run Linux on them now. The specs he cites would
| actually be fairly small for a mainframe, which can reach up to
| 40TB of memory.
|
| I'm not saying this is a _good_ idea, but it seems better than
| what the OP proposes.
| wmf wrote:
| No. A Genoa server is probably faster than a z16 and a
| Superdome Flex is definitely faster.
| bob1029 wrote:
| If it's good enough for the payment card industry, I don't know
| why it can't work for tweets. The amount of data per
| transaction is very similar.
| trishume wrote:
| My friend mentioned this just before I published and I think
| that probably is the fastest largest thing you can get which
| would in some sense count as one machine. I haven't looked into
| it, but I wouldn't be surprised if they could get around the
| trickiest constraint, which is how many hard drives you can
| plug in to a non-mainframe machine for historical image
| storage. Definitely more expensive than just networking a few
| standard machines though.
|
| I also bet that mainframes have software solutions to a lot of
| the multi-tenancy and fault tolerance challenges with running
| systems on one machine that I mention.
| jasonhansel wrote:
| Incidentally, a lot of people have argued that the massive
| datacenters used by e.g. AWS are effectively single large
| ("warehouse-scale") computers. In a way, it seems that the
| mainframe has been reinvented.
| dekhn wrote:
| I wouldn't really agree with this since those machines
| don't share address spaces or directly attached busses.
| Better to say it's a warehouse-scale "service" provided by
| many machines which are aggregated in various ways.
| sterlind wrote:
| I wonder though.. _could_ you emulate a 20k-core VM with
| 100 terabytes of RAM on a DC?
|
| Ethernet is fast, you might be able to get in range of
| DRAM access with an RDMA setup. cache coherency would
| require some kind of crazy locking, but maybe you could
| do it with FPGAs attached to the RDMA controllers that
| implement something like Raft?
|
| it'd be kind of pointless and crash the second any
| machine in the cluster dies, but kind of a cool idea.
|
| it'd be fun to see what Task Manager would make of it if
| you could get it to last long enough to boot Windows.
| MayeulC wrote:
| You absolutely can, but the speed of light is still going
| to be a limitting factor for RTT latencies, acquiring and
| releasing locks, obtaining data from memory, etc.
|
| It's relatively easy to have it work slowly (reducing
| clocks to have a period higher than max latency), but
| becomes very hard to do at higher freqs.
|
| Beowulf clusters can get you there to some extent,
| although you can always do better with specialized
| hardware and software (by then you're building a
| supercomputer...)
| dekhn wrote:
| it's been done:
| https://www.bleepingcomputer.com/news/microsoft/monster-
| azur...
| trishume wrote:
| I have fantasized about doing this as a startup,
| basically doing cache coherency protocols at the page
| table level with RDMA. There's some academic systems that
| do something like it but without the hypervisor part.
|
| My joke fantasy startup is a cloud provider called
| one.computer where you just have a slider for the number
| of cores on your single instance, and it gives you a
| standard linux system that appears to have 10k cores.
| Most multithreaded software would absolutely trash the
| cache-coherency protocols and have poor performance, but
| it might be useful to easily turn embarrassingly parallel
| threaded map-reduces into multi-machine ones.
| hinkley wrote:
| Oxide is basically building a minicomputer.
| sterlind wrote:
| to me the line between machine and cluster is mostly about
| real-time and fate-sharing. multiple cores on a single
| machine can expect memory accesses to succeed, caches to be
| coherent, interrupts to trigger within a deadline, clocks
| not to skew, cores in a CPU not to drop out, etc.
|
| in a cluster, communication isn't real-time. packets drop,
| fetches fail, clocks skew, machines reboot.
|
| IPC is a gray area. the remote process might die, its
| threads might be preempted, etc. RTOSes make IPC work more
| like a single machine, while regular OSes make IPC more
| like a network call.
|
| so to me, the datacenter-as-mainframe idea falls apart
| because you need massive amounts of software infrastructure
| to treat a cluster like a mainframe. you have to use Paxos
| or Raft for serializing operations, you have to shard data
| and handle failures, etc. etc.
|
| but it's definitely getting closer, thanks to lots of
| distributed systems engineering.
| sayrer wrote:
| It's a neat thought exercise, but wrong for so many reasons
| (there are probably like 100s). Some jump out: spam/abuse
| detection, ad relevance, open graph web previews, promoted
| tweets that don't appear in author timelines, blocks/mutes,
| etc. This program is what people think Twitter is, but
| there's a lot more to it.
|
| I think every big internet service uses user-space networking
| where required, so that part isn't new.
| trishume wrote:
| I think I'm pretty careful to say that this is a simplified
| version of Twitter. Of the features you list:
|
| - spam detection: I agree this is a reasonably core feature
| and a good point. I think you could fit something here but
| you'd have to architect your entire spam detection approach
| around being able to fit, which is a pretty tricky
| constraint and probably would make it perform worse than a
| less constrained solution. Similar to ML timelines.
|
| - ad relevance: Not a core feature if your costs are low
| enough. But see the ML estimates for how much throughput
| A100s have at dot producting ML embeddings.
|
| - web previews: I'd do this by making it the client's
| responsibility. You'd lose trustworthiness though so users
| with hacked clients could make troll web previews, they can
| already do that for a site they control, but not a general
| site.
|
| - blocks/mutes: Not a concern for the main timeline other
| than when using ML, when looking at replies will need to
| fetch blocks/mutes and filter. Whether this costs too much
| depends on how frequently people look at replies.
|
| I'm fully aware that real Twitter has bajillions of
| features that I don't investigate, and you couldn't fit all
| of them on one machine. Many of them make up such a small
| fraction of load that you could still fit them. Others do
| indeed pose challenges, but ones similar to features I'd
| already discussed.
| sayrer wrote:
| "web previews: I'd do this by making it the client's
| responsibility."
|
| Actually a good example of how difficult the problem is.
| A very common attack is to switch a bit.ly link or
| something like that to a malicious destination. You would
| also DoS the hosts... as the Mastodon folks are
| discovering (https://www.jwz.org/blog/2022/11/mastodon-
| stampede/)
|
| For blocks/mutes, you have to account for retweets and
| quotes, it's just not a fun problem.
|
| Shipping the product is much more difficult that what's
| in your post. It's not realistic at all, but it is fun to
| think about.
| ilyt wrote:
| Well, Mastodon is criminally slow
| sayrer wrote:
| Here are some pointers:
|
| "Our approach to blocking links"
| https://help.twitter.com/en/safety-and-security/phishing-
| spa...
|
| "The Infrastructure Behind Twitter: Scale" https://blog.t
| witter.com/engineering/en_us/topics/infrastruc...
|
| "Mux" https://twitter.github.io/finagle/guide/Protocols.h
| tml#mux
|
| I do agree that some of this could be done better a
| decade later (like, using Rust for some things instead of
| Scala), but it was all considered. A single machine is a
| fun thing to think about, but not close to realistic. CPU
| time was not usually the concern in designing these
| systems.
| sayrer wrote:
| Here's the Twitter edge server from years ago: https://co
| urses.cs.washington.edu/courses/cse551/15sp/notes/...
| NavinF wrote:
| I'll go ahead and quote that blog post because they block
| HN users using the referer header.
|
| ---
|
| "Federation" now apparently means "DDoS yourself." Every
| time I do a new blog post, within a second I have over a
| thousand simultaneous hits of that URL on my web server
| from unique IPs. Load goes over 100, and mariadb stops
| responding.
|
| The server is basically unusable for 30 to 60 seconds
| until the stampede of Mastodons slows down.
|
| Presumably each of those IPs is an instance, none of
| which share any caching infrastructure with each other,
| and this problem is going to scale with my number of
| followers (followers' instances).
|
| This system is not a good system.
|
| Update: Blocking the Mastodon user agent is a workaround
| for the DDoS. "(Mastodon|http\\.rb)/". The side effect is
| that people on Mastodon who see links to my posts no
| longer get link previews, just the URL.
|
| ---
|
| I personally find this absolutely hilarious. Is that blog
| hosted on a Raspberry Pi or something? "Over a thousand"
| requests per second shouldn't even show up on the
| utilization graphs on a modern server. The comments
| suggest that he's hitting the database for every request
| instead of caching GET responses, but even with such a
| weird config a normal machine should be able to do over
| 10k/second without breaking a sweat.
| ilyt wrote:
| > I personally find this absolutely hilarious. Is that
| blog hosted on a Raspberry Pi or something? "Over a
| thousand" requests per second shouldn't even show up on
| the utilization graphs on a modern server.
|
| Mastodon is written on Ruby on Rails. That should really
| answer all your questions about the problem but if you're
| unfamiliar Ruby is slow compared to any compiled
| language, Rails is slow compared to near-every framework
| on the planet and it isn't written that well either.
| vidarh wrote:
| While that may be funny, the number of Mastodon instances
| is growing rapidly, to the point where it will need to
| eventually be dealt with (not least because hosting on a
| Pi or having a badly optimized setup both happens in real
| life). But more to this example, it shows passing preview
| responsibility _to end user clients_ is a far bigger
| problem. Eg not many would be able to handle the
| onslaught of being linked to from a highly viral tweet if
| previews weren 't cached.
| dpkirchner wrote:
| FWIW, jwz uses referer checking to redirect links from HN
| ... for "DoS" reasons.
| mschuster91 wrote:
| > I haven't looked into it, but I wouldn't be surprised if
| they could get around the trickiest constraint, which is how
| many hard drives you can plug in to a non-mainframe machine
| for historical image storage.
|
| Netapp is at something > 300TB storage per node IIRC, but in
| any case it would make more sense to use some cloud service.
| AWS EFS and S3 don't have any (practically reachable) limit
| in size.
| threeseed wrote:
| Have you actually used EFS/S3 before ?
|
| Because both are ridiculously slow to the point where they
| would be completely unusable for a service such as Twitter
| whose current latency is based off everything largely being
| in memory.
|
| And Twitter already evaluated using the cloud for their
| core services and it was cost-prohibitive compared to on-
| premise.
| toast0 wrote:
| > I wouldn't be surprised if they could get around the
| trickiest constraint, which is how many hard drives you can
| plug in to a non-mainframe machine for historical image
| storage.
|
| Some commodity machines use external SAS to connect to more
| disk boxes. IMHO, there's not a real reason to keep images
| and tweets on the same server if you're going to need an
| external disk box anyway. Rather than getting a 4u server
| with a lot of disks and a 4u additional disk box, you may as
| well get 4u servers with a lot of disks each, use one for
| tweets and the other for images. Anyway, images are fairly
| easy to scale horizontally, there's not much simplicity
| gained by having them all in one host, like there is for
| tweets.
| trishume wrote:
| Yah like I say in the post, the exactly one machine thing
| is just for fun and as an illustration of how far vertical
| scaling can go, practically I'd definitely scale storage
| with many sharded smaller storage servers.
| jiggawatts wrote:
| > which is how many hard drives you can plug in to a non-
| mainframe machine for historical image storage.
|
| You would be _surprised_. First off, SSDs are denser than
| hard drives now if you 're willing to spend $$$.
|
| Second, "plug in" doesn't necessarily mean "in the chassis".
| You can expand storage with external disk arrays in all sorts
| of ways. Everything from external PCI-e cages to SAS disk
| arrays, fibre channel, NVMe-over-Ethernet, etc...
|
| It's fairly easy to get several petabytes of fast storage
| directly managed by one box. The only limit is the total
| usable PCIe bandwidth of the CPUs, which for a current-gen
| EPYC 9004 series processors in a dual-socket configuration is
| something crazy like 512 GB/s. This vastly exceeds typical
| NIC speeds. You'd have to balance available bandwidth between
| _multiple_ 400 Gbps NICs and disks to be able to saturate the
| system.
|
| People really overestimate the data volume put out by a
| service like Twitter while simultaneously underestimating the
| bandwidth capability of a single server.
| trishume wrote:
| That's really cool! Each year of historical images I
| estimate at 2.8PB, so it would need to scale quite far to
| handle multiple years. How would you actually connect all
| those external drive chassis, is there some kind of
| chainable SAS or PCIe that can scale arbitrarily far? I
| consider NVMe-over-fabrics to be cheating and just using
| multiple machines and calling it one machine, but "one
| machine" is kinda an arbitrary stunt metric.
| wtallis wrote:
| Stuff like [0] exists, allowing you to fan out a single
| server's PCIe to quite a few PCIe JBOD chassis.
| Considering that SSDs can get you ~1PB in 1U these days,
| you can get pretty far while still technically sticking
| with PCIe connectivity rather than NVMeoF.
|
| [0] https://www.liqid.com/products/liqid-
| elements/liqid-48-port-...
| metadat wrote:
| I was skeptical about 1PB in 1U, but searched and learned
| this was showcased by Supermicro as far back as 2018:
|
| https://www.supermicro.com/en/pressreleases/supermicro-
| unlea...
| ksec wrote:
| It depends on how you think of "one machines". :) You can
| fit 1PB in 1U without something like NVMe-over-fabrics.
| So in a 4U unit gives you plenty of room.
|
| We have Zen4c 128 Core with DDR5 now. We might get a 256
| Core Zen6c with PCI-E 6.0 and DDR6 by 2026.
|
| I really like these exercise of trying to shrink the
| amount of server needed, especially those on Web usage.
| And the mention of Mainframe. Which dont get enough
| credit for. I did something similar with Netflix
| 800Gbps's post. [1] Where they could serve every single
| user with less than 50 Racks by the end of this decade.
|
| [1] https://news.ycombinator.com/item?id=33451430
| NavinF wrote:
| > I consider NVMe-over-fabrics to be cheating
|
| Is an infiniband switch connected to a bunch of machines
| that expose NVMe targets really that different from a SAS
| expander connected to a bunch of JBOD enclosures? Only
| difference is that the former can scale beyond 256 drives
| per controller and fill an entire data center. You're
| still doing all the compute on one machine so I think it
| still counts.
| ilyt wrote:
| > People really overestimate the data volume put out by a
| service like Twitter while simultaneously underestimating
| the bandwidth capability of a single server.
|
| It's outright comical. Above we have people thinking
| somehow amount of TLS connections single server can handle
| is _a problem_ , in service where there would be hundreds
| of thousands lines of code to generate the content served
| over it, all while using numbers from what seems like 10+
| years old server hardware
| snotrockets wrote:
| I ask candidates I interview to design a certain service. Most
| ask about scale, to which I like to direct the question back at
| them: it's going to be huge. As big as Twitter. How big would
| that be, do you think?
|
| Most then suggest scale that would make the service run
| comfortable from a not-too powerful machine, and then go to
| design data-center spanning distributed service.
| justapassenger wrote:
| Saying this is production Twitter is like saying that rsync is a
| Dropbox.
| betaby wrote:
| rsync is way better than Dropbox for large volumes / high speed
| file transfers . So, yes.
| gravypod wrote:
| As a side note:
|
| > I did all my calculations for this project using Calca (which
| is great although buggy, laggy and unmaintained. I might switch
| to Soulver) and I'll be including all calculations as snippets
| from my calculation notebook.
|
| I've always wanted an {open source, stable, unit-aware} version
| of something like this which could be run locally or in the
| browser (with persistence on a server). I have yet to find one.
| This would be a massive help to anyone who does systems design.
| pengaru wrote:
| This post reminds me of an experience I had in ~2005 while @
| Hostway Chicago.
|
| Unsolicited story time:
|
| Prior to my joining the company Hostway had transitioned from
| handling all email in a dispersed fashion across shared hosting
| Linux boxes with sendmail et al, to a centralized "cluster"
| having disparate horizontally-scaled slices of edge-SMTP servers,
| delivery servers, POP3 servers, IMAP servers, and spam scanners.
| That seemed to be their scaling plan anyways.
|
| In the middle of this cluster sat a refrigerator sized EMC
| fileserver for storing the Maildirs. I forget the exact model,
| but it was quite expensive and exotic for the time, especially
| for an otherwise run of the mill commodity-PC based hosting
| company. It was a big shiny expensive black box, and everyone
| involved seemed to assume it would Just Work and they could keep
| adding more edge-SMTP/POP/IMAP or delivery servers if those
| respective services became resource constrained.
|
| At some point a pile of additional customers were migrated into
| this cluster, through an acquisition if memory serves, and things
| started getting slow/unstable. So they go add more machines to
| the cluster, and the situation just gets worse.
|
| Eventually it got to where every Monday was known as Monday
| Morning Mail Madness, because all weekend nobody would read their
| mail. Then come Monday, there's this big accumulation of new
| unread messages that now needs to be downloaded and either
| archived or deleted.
|
| The more servers they added the more NFS clients they added, and
| this just increased the ops/sec experienced at the EMC. Instead
| of improving things they were basically DDoSing their overpriced
| NFS server by trying to shove more iops down its throat at once.
|
| Furthermore, by executing delivery and POP3+IMAP services on
| separate machines, they were preventing any sharing of buffer
| caches across these embarrassingly cache-friendly when colocated
| services. When the delivery servers wrote emails through to the
| EMC, the emails were also hanging around locally in RAM, and
| these machines had several gigabytes of RAM - only to _never_ be
| read from. Then when customers would check their mail, the POP3
| /IMAP servers _always_ needed to hit the EMC to access new
| messages, data that was _probably_ sitting uselessly in a
| delivery server 's RAM somewhere.
|
| None of this was under my team's purview at the time, but when
| the castle is burning down every Monday, it becomes an all hands
| on deck situation.
|
| When I ran the rough numbers of what was actually being performed
| in terms of the amount of real data being delivered and
| retrieved, it was a trivial amount for a moderately beefy PC to
| handle at the time.
|
| So it seemed like the obvious thing to do was simply colocate the
| primary services accessing the EMC so they could actually profit
| from the buffer cache, and shut off most of the cluster. At the
| time this was POP3 and delivery (smtpd), luckily IMAP hadn't
| taken off yet.
|
| The main barrier to doing this all with one machine was the
| amount of RAM required, because all the services were built upon
| classical UNIX style multi-process implementations (courier-pop
| and courier-smtp IIRC). So in essence the main reason most of
| this cluster existed was just to have enough RAM for running
| multiprocess POP and SMTP sessions.
|
| What followed was a kamikaze-style developed-in-production
| conversion of courier-pop and courier-smtp to use pthreads
| instead of processes by yours truly. After a week or so of
| sleepless nights we had all the cluster's POP3 and delivery
| running on a single box with a hot spare. Within a month or so
| IIRC we had powered down most of the cluster, leaving just spam
| scanning and edge-SMTP stuff for horizontal scaling, since those
| didn't touch the EMC. Eventually even the EMC was powered down,
| in favor of drbd+nfs on more commodity linux boxes w/coraid.
|
| According to my old notes it was a Dell 2850 w/8GB RAM we ended
| up with for the POP3+delivery server and identical hot spare,
| replacing _racks_ of comparable machines just having less RAM.
| >300,000 email accounts.
| BenjiWiebe wrote:
| Thank you for writing this.
| icedchai wrote:
| Sounds reasonable. Back in the 90's, at an early ISP, we had
| about 6,000 POP3/SMTP accounts on a FreeBSD box with 128 megs
| of RAM and a decent (for 1996) SCSI disk. I recall we did some
| kernel tuning (max processes and file handles, maybe?)
| firstSpeaker wrote:
| This is one of the most interesting part of the whole post for
| me:
|
| Through intense digging I found a researcher who left a notebook
| public including tweet counts from many years of Twitter's 10%
| sampled "Decahose" API and discovered the surprising fact that
| tweet rate today is around the same as or lower than 2013! Tweet
| rate peaked in 2014 and then declined before reaching new peaks
| in the pandemic. Elon recently tweeted the same 500M/day number
| which matches the Decahose notebook and 2013 blog post, so this
| seems to be true! Twitter's active users grew the whole time so I
| think this reflects a shift from a "posting about your life to
| your friends" platform to an algorithmic content-consumption
| platform.
|
| So, the number of writes has been the same for a good long while.
| lightlyused wrote:
| It is all fun running on one machine until a capacitor leaks or
| something else goes south.
| systemvoltage wrote:
| It's worth noting Stackoverflow's production arch:
| https://stackexchange.com/performance
| ricardobeat wrote:
| > super high performance tiering RAM+NVMe buffer managers which
| can access the RAM-cached pages almost as fast as a normal memory
| access are mostly only detailed and benchmarked in academic
| papers
|
| Isn't this exactly what modern key value stores like RocksDB,
| LMDB etc are built for?
| surume wrote:
| Ops: "One of our instances went down" Everyone else: "Gaaaahhh"
| kureikain wrote:
| Not to the extreme of fitting everything into one machine but I
| have explorer the idea of separate stateless workload into its
| own machine.
|
| However, the stateless workload can still operate in a read-only
| manner if the stateful component failed.
|
| I run an email forwarding service[1], and one of challenge is how
| can I ensure the email forwarding still work even if my primary
| database failed.
|
| And I come up with a design that the app boot up, and load entire
| routing data from my postgres into its memory data structure, and
| persisted to local storage. So if postgres datbase failed, as
| long as I have an instance of those app(which I can run as many
| as I can), the system continue to work for existing customer.
|
| The app use listen/notify to load new data from postgres into its
| memory.
|
| Not exactly the same concept as the artcile, but the idea is that
| we try to design the system in a way where it can operate fully
| on a single machine. Another cool thing is that it easiser to
| test this, instead of loading data from Postgres, it can load
| from config files, so essentially the core biz logic is isolated
| into a single machine.
|
| ---
|
| https://mailwip.com
| sitkack wrote:
| I am both embarrassed and disappointed with the negativity this
| post has attracted.
|
| A litanny of "gotchas", where someone attempts to best the OP.
| What about x, y and z? It can't possibly scale. Twitter is so
| much more than this, etc.
|
| The OP isn't making the assertion that Twitter should replace
| their current system with a single large machine.
|
| The whole thread paints a picture of HN like it is full of a
| bunch of half-educated, uncreative negative brats.
|
| To the people that encourage a fun discussion, thank you! Great
| things are not built by people who only see how something cannot
| possibly work.
| kissgyorgy wrote:
| If he would have called it anything else but Twitter and stated
| it is theoretical or a hobby project or something for fun, I
| would be fine with it, but don't mention Twitter anywhere near
| because it has nothing to do with Twitter.
| bcrosby95 wrote:
| It has nothing to do with Twitter, except if you described
| the features of the project, the vast majority of people
| would say "oh, its like Twitter?"
| idealmedtech wrote:
| A post like this without Twitter in the title isn't nearly as
| provocative, and would probably have died in /new. It's
| because of the authors decision to scope out a Twitter clone
| that we're having this discussion!
| LastTrain wrote:
| Yours is by far the most negative post in this thread - in tone
| and content. I'm sorry but this kind of proposal is
| intentionally controversial and invites picking through it like
| this. Glad the author has thicker skin than you.
| [deleted]
| mcenedella wrote:
| Amen!
| Xeoncross wrote:
| I must admit, as a bystander I'm torn here.
|
| I love that Tristan put out this post and made it so detailed
| with plenty of assumptions to cover. I also like to hear about
| possible issues and assumptions which the crowd calls out. Even
| naysayers can be helpful.
|
| I want both, but I don't want to crowd to go to far and kill
| the desire to produce this kind of content.
| trishume wrote:
| To me most interesting are factors I didn't consider in
| features I did cover. Next most interesting are features I
| didn't cover which are kinda core to Twitter being good, and
| also pose interesting performance problems, like the person
| who mentioned spam/abuse detection. After that are non-core
| features which pose interesting performance problems that are
| different from problems I already covered.
|
| The comments that I think aren't contributing much are ones
| that mention features that I didn't cover but make no attempt
| to argue that they're actually hard to implement efficiently,
| or that assert that because I didn't implement something it
| isn't feasible to make as fast as I calculate, without
| arguing what would actually stop me from implementing
| something that efficient. Or ones who repeat that this isn't
| practical, which I say at length in the post.
| racingmars wrote:
| > I want both, but I don't want to crowd to go to far and
| kill the desire to produce this kind of content.
|
| I think it's easy to have both. It's all about the tone of
| the responses.
|
| For example, instead of "your assumptions are wrong, this
| would collapse because X" or "this is dumb because real
| Twitter does Y which yours doesn't handle," I think responses
| could be framed as:
|
| "Wow, neat thought experiment! If I were to approach this
| same problem, I might make an allowance of more than 280
| bytes of storage per tweet to allow for additional metadata
| that is probably needed to make everything work together; I
| wonder if that can be accommodated with an even beefier big
| computer?"
|
| Or "What a great writeup of building a simplified Twitter!
| After the features you've accounted for, the next most
| important feature of Twitter for me personally is Y. What
| kinds of things would we have to do to stretch your idea to
| handle that? [or, I bet with the addition of X we could make
| that happen in this setup too!]"
|
| I think many criticisms could be turned into constructive
| positive additions to the original article versus attacks
| against the idea of the article.
| nickstinemates wrote:
| Being nice takes up a lot of space. Straight and to the
| point without the judgment would be a good compromise.
| jameshart wrote:
| Getting everything onto one machine works great until... it no
| longer fits on one machine.
|
| You add another feature and it requires a little bit more RAM,
| and another feature that needs a little bit more, and..
| eventually it doesn't all fit.
|
| Now you have to go distributed.
|
| And your entire system architecture and all your development
| approaches are built around the assumptions of locality and cache
| line optimization and all of a sudden none of that matters any
| more.
|
| Or you accept that there's a hard ceiling on what your system
| will ever be able to do.
|
| This is like building a video game that pushes a specific
| generation of console hardware to its limit - fantastic! You got
| it to do realtime shadows and 100 simultaneous NPCs on screen!
| But when the level designer asks if they can have water in one
| level you have to say 'no', there's no room to add screenspace
| reflections, the console can't handle that as well. And that's
| just a compromise you have to make, and ship the game with the
| best set of features you can cram into that specific hardware.
|
| You certainly _could_ build server applications that way. But it
| feels like there 's something fundamental to how service
| businesses operate that pushes away from that kind of
| hyperoptimized model.
| Ingaz wrote:
| I think that the main point of OP is that it's possible to
| serve production load using just one server.
|
| I did not looked into source code yet but I suppose that OP if
| not implemented already than there should ideas for
| implementation.
|
| In addition: from my POV implementation of scaling for such
| service should be trivial: - sharding of data between instances
| by a criteria (e.g. regional) or by hash - configure network
| routing
|
| I think it should work
| Winsaucerer wrote:
| How about smaller projects that might be hugely successful, but
| will never be remotely close to a Twitter level of success?
|
| It's interesting to see how much can be done with a single
| machine, because most projects will never be this big.
|
| Though there will still be other concerns like redundancy to
| deal with.
| Ingaz wrote:
| Can you explain me what exactly meant by "success of
| Twitter"?
|
| It's not sarcasm, I have twitter account but I never
| understood hype about twitter.
|
| I see nothing in twitter from technical POW, closed twitter
| protocol looks very strange, they banned Trump, they were
| profitable in 2021, Elon Musk bought them for ~44 bln.
|
| Maybe sellout of company with problems for such price is
| success.
| vidarh wrote:
| Many of us will remember that Twitter in fact did start out
| with a monolithic database and had to rewrite a bunch of stuff
| when they couldn't make that work anymore.
|
| Of course they could fit a much larger dataset on one machine
| today.
|
| (But I will note the article is also assuming a chronological
| timeline by default, but that of course hasn't been true for
| years - the ranking Twitter does now is far more complex)
| gnuvince wrote:
| There's a story by Bryan Cantrill [1] about how he went to
| Twitter to help them understand why it would take 400
| milliseconds of compute to process a request (I'll leave the
| reveal to Bryan). Scaling horizontally is probably necessary
| for something the size of Twitter, but that doesn't mean that
| we can half-ass the code and just throw more machines at the
| problem. If we write code with a bit more mechanical sympathy
| and avoid the latest non-proven fads, we can surely write
| software that is 10x faster and requires much less scaling.
|
| [1] https://www.youtube.com/watch?v=LjFM8vw3pbU&t=3240s
| gnuvince wrote:
| Fixed timestamp: https://youtu.be/LjFM8vw3pbU?t=1613
| vidarh wrote:
| Oh, absolutely. I think in terms of making people think
| about putting things in-process in RAM and being a bit more
| imaginative at looking at machines sizes as worth exploring
| is a good thing, and the article is interesting as a look
| at what you _can_ do.
|
| Many things _can_ plausibly fit on a single machine
| irrespective of uptake (e.g. I worked on a system not long
| ago where even if we cornered the entire global market and
| the market expanded several magnitudes in size, our entire
| working set would still fit comfortable in memory).
|
| But even when you need to scale, it absolutely can scale
| _better_ if you 're willing to not automatically resort to
| a standard database stack for everything for example.
| kaba0 wrote:
| Twitter used to (not sure how much did that idiot fck it
| up) be quite great from this perspective from what I
| gathered. They had technical blog posts and are even
| featured in the famous data intensive book as an example
| for many scaling problems. Sure, they didn't write it in
| assembly but instead use Java/Scala with Graal, but
| architecturally they had a sound system (plus what people
| routinely forget, the system should handle the general
| load, _but_ it worths nothing if it fails at peaks, and
| twitter can easily hit a billion daily users during big
| global events)z
| nyanpasu64 wrote:
| Video games have had 3D water for decades before screen-space
| reflections, and many look serviceable decades later (Super
| Mario Sunshine looks great at 480p though dated at higher
| resolutions).
| jameshart wrote:
| Curses. That undermines my argument completely. It wasn't
| merely a random example of how you have to compromise on
| features to fit in the available hardware budget, but
| actually the premise upon which my entire argument rested.
| vidarh wrote:
| > And your entire system architecture and all your development
| approaches are built around the assumptions of locality and
| cache line optimization and all of a sudden none of that
| matters any more.
|
| Indeed, with the hyperoptimized version here, the moment you
| tip over into two machines each machine will need two copies of
| every tweet from anyone who has followers sharded to both
| machines, so the capacity of two machines is going to be far
| less than twice the capacity of one as a large proportion of
| tweets will cause writes on both shards. This inefficiency will
| now always be with you - the average number of writes per user
| per tweet will go up until your number of shards approaches
| average follower counts.
|
| This is why it's common to model this with fan-out on write,
| because the moment you accept that there is a risk you'll tip
| over into a sharded model you need to account for that. If
| asked the question of such a design, it's worth pointing out
| that _if_ you can guarantee it fits on one machine, and this is
| true for many more problems than people expect, then you can
| save a lot, but then I 'd set out the more complex model and
| contrast it to the single-machine model.
|
| You don't need to fan-out to every account even in such a
| distributed system, certainly. You can fan-out to every
| shard/instance, and keeping that cached in RAM would still
| allow you to be far more efficient than e.g. Mastodon (which
| does fan-out to every instance for the actual post data, but
| relies on a Postgres database)
| joshspankit wrote:
| Weird criticism but o.k.
|
| Edit: Unless I missed something, the author never argued that
| Twitter _should_ be hosted on one machine and therefore
| criticizing the "fun stunt" like this makes no sense to me
| jameshart wrote:
| There are certainly people reading this and nodding and
| thinking 'yeah, this makes sense! Why _don 't_ we build
| services like this?' And adding it to their mental list of
| arguments against microservices or whatever - and I wanted to
| make sure people hear the reasons why this kind of
| performance maximization tends not to be the norm.
| p-e-w wrote:
| > You certainly _could_ build server applications that way. But
| it feels like there 's something fundamental to how service
| businesses operate that pushes away from that kind of
| hyperoptimized model.
|
| That "fundamental" thing is the cultural expectation that SaaS
| offerings constantly grow in features, rather than in
| reliability or performance. As your example from the world of
| video games demonstrates, there is no industry-wide belief that
| things must be able to do ever-more, forever. It's really
| mostly SaaS and desktop software that has this weird and
| unreasonable culture around it. That's why your word processor
| can now send emails, and your email provider now does
| translations as well.
| nkellenicki wrote:
| You're not taking into account data, you're only talking
| about features. What about when the data no longer fits on
| the one machine? Or processing the data exceeds the capacity
| of the machine?
|
| Data growth through user growth or just normal day-to-day
| usage is expected.
| p-e-w wrote:
| If Twitter's data can fit on one machine, then the data of
| 99.99% of companies can. Not every product needs a billion
| users with Gigabytes of storage each. The assumption that
| if your startup's tech isn't scalable enough to become the
| next Google then it's the wrong tech is hilarious nonsense
| driven largely by ego fantasies.
| mapme wrote:
| Twitters data cannot fit on one machine. In 2015 their
| Hadoop cluster was 30 PB per earlier comments/their blog.
| How do you fit that on one machine?
| nkellenicki wrote:
| It does not fit on one machine. Tweets alone generate
| petabytes of data a year, and other events are petabytes
| per day.
|
| https://blog.twitter.com/engineering/en_us/topics/infrast
| ruc...
|
| https://ankush-chavan.medium.com/twitter-data-storage-
| and-pr...
| dmitriid wrote:
| > Tweets alone generate petabytes of data a year
|
| Nope. It's not Tweets that generate that data. It's the
| insane amount of (mostly unnecessary) noise that gets
| thrown into the mix: analytics, logs, metrics, you name
| it.
|
| Every time you scroll Twitter sends multiple events to
| the server. That alone will generate a large chunk of
| those petabytes.
| nkellenicki wrote:
| No, that's the second link - generated data, separate
| from tweets.
|
| Tweets alone generate petabytes of data a year.
|
| https://ankush-chavan.medium.com/twitter-data-storage-
| and-pr...
|
| Also, many people would disagree that stuff required to
| run a business is "mostly unnecessary".
| eismcc wrote:
| You'd end up synchronizing feature releases to Moores law.
| Which while it sounds untenable there are large corporations
| that continue to use monolithic approach and vertical scaling.
| seanhunter wrote:
| It's sort of strange you have to make these points, but as an
| industry we seem to have an extremely short memory.
|
| Vertical scaling was absolutely the way most big applications
| were built up until well into the 90s. Companies like Oracle
| were really built on the fact that getting performance and
| reliability out of a single highly-contested massive server is
| hard but important if that's the way you're going. Linux became
| dominant primarily because horizontal scaling won that argument
| and it won it pretty much exactly because of:
|
| 1)what you said - you hit a hard cap on how big you can make
| your main server at which point you are really screwed.
| Scalability pain points become a hard wall.
|
| 2) when I say "server" I mean "servers" of course because you'd
| need an H/A failover, at which point you've eaten the cost of
| replication, handling failover etc and you may as well
| distribute
|
| 3) cost. Because hardware cost vs capability is exponential, as
| your requirements become bigger you pretty rapidly hit a point
| where lots of commodity hardware becomes cheaper for a given
| performance point than few big servers
|
| So there's a reason that distributed systems on commodity
| hardware became the dominant architectural paradigm. It's not
| the only way to do it, but it's a reasonable default for many
| use cases. For a very high-throughput system like twitter it
| seems a very obvious choice.
|
| Clearly there are costs to distribution, so if you can get away
| with a simpler architecture then as always Occam's razor
| applies. Also if you can easily distribute later then it
| probably makes sense to leave that option open and explore it
| when you need it rather than overcomplicate too early.
| Joeri wrote:
| The thing is that hardware scales faster than humanity. When
| the internet boom happened there was no choice except to
| scale horizontally to reach a global audience, but as this
| article points out that assumption might no longer hold true
| for many services. It might make sense to return to
| vertically scaled highly reliable servers to achieve software
| simplicity and a lower overall cost.
|
| I'm always reminded of how stackoverflow essentially runs off
| a single database server. If they can do it, most web
| properties can do it.
| saagarjha wrote:
| Stack Overflow happens to go down fair maintenance a fair
| bit. Now, it's not really a service that might need
| reliability like Twitter does*, but it's important to keep
| in mind.
|
| *Some engineers may disagree
| brazzledazzle wrote:
| Also worth pointing out that Stack Overflow's Microsoft-
| centric architecture may also incentivize them to
| maximize vertical scaling to save on the licensing cost
| overhead horizontal scaling would incur for at least part
| of their stack.
| kaba0 wrote:
| Most, sure. Twitter? No way on Earth.
| jameshart wrote:
| When the hardware scales, the tricks to wring maximum
| performance out of it change.
|
| When you come back to build the new version of your game on
| the next gen console sure you can now add all those
| features but the processing pipelines are different now and
| the disk performance and memory to cache ratios have
| changed - getting your hyper optimized code to work in the
| new platform takes a ton of effort - so you either run it
| in some kind of emulation mode, sacrificing some of the
| performance for productivity, or you rewrite it.
|
| Same happens with new generations of server hardware. Your
| clever hack to maximize NUMA locality of data to each core
| becomes a liability when the next hardware gen comes out
| and on-die caches are bigger. Decisions about what should
| use RAM get invalidated by faster SSDs.
|
| Maybe you can build a service this way - hardware first.
| Pick a server platform for a couple of years; build to that
| capacity; ship; then start designing the next gen service
| to run on a new set of hardware?
|
| To some extent this _is_ how database or virtualization
| systems software is written. And if I'm not mistaken
| Twitter actually did develop their own database stack to
| optimally handle their particular data storage model, and I
| assume that was done pretty close to the metal.
| popcorncowboy wrote:
| It's less "short memory" than the fact that you can be a
| "senior software engineer" after just 5 years or so
| experience. There is a significant cohort of (particularly
| web-tech) developers who were young children in the 90s, and
| whose professional careers started in the 2010s and have only
| ever known "the cloud", big-tech and big-tech tech (k8s,
| etc).
|
| It's a similar phenomenon to the observation that tech
| "innovations" tend to recapitulate research that had its
| roots in the 50-60-70s.
|
| "The industry" doesn't seem to put much stock in generational
| knowledge transfer.
| britannio wrote:
| I'm a CS undergrad, do you have any recommendations for
| effectively combating this?
| wrigby wrote:
| No approach is a silver bullet here, but what I've found
| effective is to seek out friendships with / mentorship
| from senior and staff-level engineers. They'll have tons
| of war stories from 10-20 years ago, and may even have
| some snarky opinions on what "new technologies" are just
| re-inventions of something that the industry had already
| solved decades ago.
|
| As a current undergrad, you can also look to your
| professors for this (especially those with industry
| experience before they went into teaching).
|
| After graduation, this may mean working at a company
| _with_ those older engineers, as opposed to a 5-20 person
| startup with a homogeneous group of 20-somethings.
| jameshart wrote:
| It's gotta be tough though for young devs to tell the
| difference between old fogeys just dismissing new tech
| because it's new, vs because it's a recapitulation of an
| old mistake.
|
| And the thing is that what was a bad idea in 2000 might
| now be an idea whose time has come, because the
| surrounding context has changed - be it browser
| technology or the size of machine memory or the
| capabilities of programming languages.
|
| So, like, when I point out that kubernetes is just DCOM
| all over again I'm not actually dismissing kubernetes
| (just because we don't use DCOM any more doesn't mean it
| wasn't a good choice then); less still suggesting that we
| should go back to using DCOM; I'm just saying 'maybe
| there are some lessons we can learn from how people used
| DCOM back in the day about what cases kubernetes is
| suited for and what the pitfalls might be'. And, also,
| maybe raising the possibility that in a few years time we
| will look back at kubernetes as a bloated outdated
| approach and be glad to see the back of it - even though
| right now it might be a great technology to use.
|
| But I'm not sure how a new junior dev can possibly pick
| up all that nuance from just listening to old farts like
| me talking about how this reminds us of how we used to do
| things back in the old days.
| switchbak wrote:
| Wouldn't DCOM be more akin to Corba or RMI? I'm failing
| to see the analogy to Kubernetes.
| jameshart wrote:
| Independently deployed components, connected up to
| discoverable queues and data providers, relying on a
| registry for discovery and load balancing... there's a
| lot more in common than you'd think.
| switchbak wrote:
| Not the OP but the courses I took on operating systems
| (writing a simple one, in C) have yielded lessons that
| I've used throughout my career. If you can find a course
| too challenge you, and it's taught well, that should
| provide a ton of the learning that us old neckbeards
| couldn't avoid back in the day.
|
| Forcing yourself to use barebones languages and
| environments is good too. Hacking on ancient machines or
| targeting embedded hardware is another good way to get a
| better intuition on the order of magnitude performance
| differences of various approaches.
|
| Reading about promising tech of the past is also useful:
| prolog, Smalltalk, etc. Lots of inspiring and fruitful
| lessons to mine there.
| [deleted]
| swellguy wrote:
| You have violated the number one rule in Silicon Valley: If it
| doesn't take at least "N" "engineers" to "solve" a problem who
| report directly to moi, then how am I relevant? So I agree this
| is entirely possible, but no one would build this with any
| funding.
| z3t4 wrote:
| A Twitter clone could probably run in a teenagers closet, but not
| after it has been iterated by 10000 monkeys.
| morphle wrote:
| Why not a single FPGA with 100Gbps ethernet or pcie with NVM
| attached? Around $5K for the hardware and $5K for the traffic per
| month. The software would be a bit trickier to write, but you now
| get 100x performance for the same price
| tluyben2 wrote:
| That would be quite a nice project for fun.
| morphle wrote:
| I have seen a number of papers of people doing such a fun
| project as their PhD thesis. I'll try to find some more [1]
| examples with links on Google Scholar. Try searching for p4
| programmable dataplane content address".
|
| [1] Flexible Content-based Publish/Subscribe over
| Programmable Data Planes https://pure.rug.nl/ws/files/2060934
| 41/Flexible_Content_base...
| mpoteat wrote:
| Let's spend multi-million dollars a year on a team of highly
| specialized FPGA engineers writing assembly and HDL so that we
| can save 5k a month. Feature velocity will be 100x slower as
| well, but at least our application is efficient.
|
| I think that this may make sense for some applications, but I
| also think that if you can utilize software abstractions to
| improve developer efficiency, it reduces risk in the long run.
| nimish wrote:
| >Let's spend multi-million dollars a year on a team of highly
| specialized FPGA engineers
|
| I agree with you specialists are expensive but even a team of
| software engineers runs into the multi million dollar
| territory.
|
| Why not spend the same amount AND cut down resource use?
| Hyperscalers have shifted to custom hardware already.
| morphle wrote:
| Those millions of dollars have already been spent. For
| example the P4 [1] language (a HDL language) and the Tofino 3
| chip. It started out as FPGA (NetFPGA) to do programmable
| packet routing at linespeed. You now have the P4 language to
| define your packet routers with and generate your code. Ten
| years later we have 25 Tbps software defined packet routers.
|
| [1] https://en.wikipedia.org/wiki/P4_(programming_language)
|
| [2] https://www.intel.com/content/www/us/en/products/network-
| io/...
| tylerhou wrote:
| A bit trickier is a huge understatement.
| morphle wrote:
| That depends. If you would use our asynchronous runtime
| reconfigurable array called Morphle Logic [1] instead of FPGA
| (Field Programmable Gate Array), you could program this
| hardware Twitter in a 1000 hours and have it run at 50 Tbps.
| You would not need to know much about hardware but you would
| get a thousandfold speedup of your software.
|
| [1] https://github.com/fiberhood/MorphleLogic/blob/main/READM
| E_M...
| 0837560358 wrote:
| [flagged]
| viraptor wrote:
| This is in no way a criticism of the analysis. But what I think
| is a hidden cost of an idea like this (that hasn't been pointed
| out) is the ability to extend the features. With a tightly
| integrated system like that you may want to add a frobnicator as
| a test - now that whole system would need to change to
| accommodate that, because all the timeline processing happens
| more or less in memory. Making things external / network based
| adds overhead, but makes plugging in / removing an extra feature
| much easier. If you count the cost of work required for making
| changes, then burning money on the "unnecessary" horizontal
| scaling may not be a bad idea. Wanna add new ads analytics? Just
| plug into this common firehouse/summary endpoint without worrying
| about the internals. Wanna test a new implementation of some
| component? Run both in parallel, plugging into same inputs. Etc.
| [deleted]
| sethev wrote:
| John Carmack tweeted something that made me noodle on this too:
|
| >It is amusing to consider how much of the world you could serve
| something like Twitter to from a single beefy server if it really
| was just shuffling tweet sized buffers to network offload cards.
| Smart clients instead of web pages could make a very large
| difference. [1]
|
| Very interesting to see the idea worked out in more detail.
|
| [1] https://twitter.com/id_aa_carmack/status/1350672098029694998
| varjag wrote:
| Isn't that what an OPA sorta kinda does.
| threeseed wrote:
| > just shuffling tweet sized buffers to network offload cards
|
| Except that's not what it is doing at all.
|
| It assembles all the Tweets internally, applies an ML model to
| produce a finalised response to the user.
| nimish wrote:
| Great, staple a few ML accelerators to your NIC. Nvidia sells
| them! You could build an entire supercomputer style setup
| 100% optimized for Twitter data movement and computation with
| COTS hardware IMO.
|
| I strongly doubt that entire datacenters need to be used if
| and only if Twitter obsessively optimized for hardware usage
| efficiency over everything else. In reality they don't and
| make some pretty big compromises to actually get stuff built.
| Hardware is cheap, people are not.
| threeseed wrote:
| a) No one has said that Twitter doesn't use ML
| accelerators.
|
| b) No one has said Twitter operates entire data centres.
|
| c) You need more than just NICs and ML accelerators to
| built a Twitter timeline. You need to rank the content,
| determine appropriate ads and combine them together. You
| can't do that in your network card.
| pg314 wrote:
| > No one has said Twitter operates entire data centres.
|
| They do though. 3 in the US alone
| (https://www.datacenterdynamics.com/en/news/report-elon-
| musk-...).
| seritools wrote:
| > if it really was [which it isn't]
| to11mtm wrote:
| Maybe it should be, though?
|
| I sometimes wonder how much value ML provides vs a proper
| sort function for anything but advertising.
| threeseed wrote:
| It's an easy thought experiment.
|
| Do you think that the most successful web companies in
| the world with arguably the best people i.e. Amazon,
| Facebook, Instagram, TikTok, LinkedIn, Pinterest,
| Youtube, Netflix, Snapchat etc. have no idea what they
| are doing. That the highly complex, expensive and latency
| impacting recommendation systems could be replaced by
| trivial sorting.
|
| Or maybe they do work, do translate to increased usage
| and do significantly impact revenue.
| fzingle wrote:
| Maybe they are considering that we shouldn't build and
| optimize our society solely for the purposes of
| maximizing revenue.
|
| Would it be better to live in a world where Twitter (for
| example) existed because it is a useful thing and not
| because it might make lots of money?
| afrodc_ wrote:
| Doesn't it lose a lot of money and its usefulness is
| directly correlated with its current massive usage.
| saagarjha wrote:
| It ran about break even until very recently.
| brazzledazzle wrote:
| A lot of people don't realize that Dunning-Kruger can
| catch any of us unaware. It's easy to look at a problem,
| think about the surface level challenges you'd have
| building it and come to the conclusion that you could do
| it better or simpler.
| vidarh wrote:
| Try changing your ordering on Twitter to chronological
| and see how much you miss.
|
| It's not just ads, it means the set of people you follow
| becomes extremely critical for your experience in a way
| that makes it far less engaging.
|
| That may be good or bad for you as a user depending on
| what you want, but for Twitter having most people stick
| to the ML augmented timeline is essential to keep you
| hooked.
| threeseed wrote:
| But many people including the OP think it is.
|
| It's like me running a web crawler on my phone and saying I
| can replace Google.
| Dylan16807 wrote:
| If you were able to index the same amount of content
| you'd have a damn good alternative. And that's the level
| of this experiment.
| threeseed wrote:
| It is trivial to index the same amount of content as
| Google as the web is mostly static.
|
| The hard part is in being able to translate a search
| query into a list of pages.
|
| And that requires a level of sophistication that far
| exceeds a laptop.
| Dylan16807 wrote:
| It is not trivial at all. That is a huge amount of
| scraping.
|
| Any machine that can do that will be a similar spec to
| what you need for serving queries. Not as fast as google
| does it, but a good amount of them.
| threeseed wrote:
| It is trivial compared to the task of translating a user
| query into search results.
|
| It's akin to saying the magic behind OpenGPT is the
| dataset.
| Dylan16807 wrote:
| Keyword search is not that hard. Pagerank is not that
| hard.
|
| The comparable goal to the article is to be a search
| engine, not to fight google for best results.
| threeseed wrote:
| > Keyword search is not that hard. Pagerank is not that
| hard.
|
| George Hotz, is that you ?
| Dylan16807 wrote:
| Keyword search and pagerank had working solutions decades
| ago.
|
| Hotz was trying to make a car controller that had never
| been done before, by himself, and then he wanted to
| """improve""" search with no explanation of what that
| meant that I saw.
|
| I think if he was tasked with taking twitter from no
| search to "has a search" he probably could have managed
| it. A team of five people definitely could have managed
| it.
| ilyt wrote:
| I think you severely underestimate the effort search
| engines have to put to filter out all the SEO garbage and
| come up with some sensible results
| Dylan16807 wrote:
| I don't underestimate the effort, I claim that it's not
| strictly necessary to have more than a couple people
| working on it for a minimum viable product, and less for
| a beta.
| seritools wrote:
| What? The OP covers much more than "just shuffling tweet
| sized buffers to network offload cards"
| PaulDavisThe1st wrote:
| which arrives at a browser running Tweak New Twitter as a
| browser extension, and that strips the response to just what
| the user actually wants to see. Effficiency!
| andrewstuart wrote:
| Most projects I encounter these days instantly reach for
| kubernetes, containers and microservices or cloud functions.
|
| I find it much more appealing to just make the whole thing run on
| one fast machine. When you suggest this tend to people say "but
| scaling!", without understanding how much capacity there is in
| vertical.
|
| The thing most appealing about single server configs is the
| simplicity. The more simple a system easy, likely the more
| reliable and easy to understand.
|
| The software thing most people are building these days can easily
| run lock stock and barrel on one machine.
|
| I wrote a prototype for an in-memory message queue in Rust and
| ran it on the fastest EC2 instance I could and it was able to
| process nearly 8 million messages a second.
|
| You could be forgiven for believing the only way to write
| software is is a giant kablooie of containers, microservices,
| cloud functions and kubernetes, because that's what the cloud
| vendors want you to do, and it's also because it seems to be the
| primary approach discussed. Every layer of such stuff add
| complexity, development, devops, maintenance, support,
| deployment, testing and (un)reliability. Single server systems
| can be dramatically mnore simple because you can trim is as close
| as possible down to just the code and the storage.
| traceroute66 wrote:
| > I find it much more appealing to just make the whole thing
| run on one fast machine.
|
| Indeed.
|
| Lots of examples out there, one being Let's Encrypt[1] who run
| off one MySQL server (with a few read replicas but only one
| write).
|
| [1] https://letsencrypt.org/2021/01/21/next-gen-database-
| servers...
| yazzku wrote:
| Damn, that was a good and sober read, thanks for sharing.
| anamexis wrote:
| That doesn't really seem like an example, since the whole
| thing doesn't run one machine. The database alone has
| multiple machines.
| traceroute66 wrote:
| > That doesn't really seem like an example, since the whole
| thing doesn't run one machine.
|
| It is an example. It shows you how you can run a service
| that issues a few hundred million SSL certs a year off
| relatively few pieces of hardware, i.e. no need to go
| drinking the cloud Kool aid.
|
| There will never be a "perfect" example. The overall point
| here is demonstrating that the first answer to everything
| doesn't have to include the word "cloud".
|
| > The database alone has multiple machines.
|
| As I said, and the blog says ... there is only one writer.
| The other nodes are smaller read replicas.
|
| Which again shows you don't need to go with the cloud
| buzzword-filled database services.
| erik wrote:
| Last I read, Hacker News was still running on one big
| machine. And still uses text files as its database.
| threeseed wrote:
| Twitter runs ads and generates billions in revenue.
|
| It can't just tolerate being down or having under-load
| issues like HN often is.
| ilyt wrote:
| > It is an example. It shows you how you can run a
| service that issues a few hundred million SSL certs a
| year off relatively few pieces of hardware, i.e. no need
| to go drinking the cloud Kool aid.
|
| "100 million certs a year" is only like ~3 certs a
| second. Maybe twice in peak. That's not much. And you're
| doing same error, focusing on one core feature and
| ignoring everything around it
| threeseed wrote:
| > there is only one writer
|
| And what happens if that writer goes down. Then the
| service just stops.
|
| > buzzword-filled
|
| I love how your buzzwords e.g. read replicas are okay but
| everyone else's are bad.
| tambourine_man wrote:
| >... without understanding how much capacity there is in
| horizontal.
|
| I think you mean vertical, right?
| andrewstuart wrote:
| Ha ha yes I do! (corrected)
| tambourine_man wrote:
| Freudian slip :)
| vasco wrote:
| Kubernetes is useful if you have many teams working on things
| in parallel and you want them to deploy in similar ways to not
| have to reinvent the same wheel in 5 different ways by 5
| different teams. If you don't have multiple teams, you don't
| need it.
| jupp0r wrote:
| It's also useful if you want your app to update without being
| down, which even a single team might want to do.
| dinosaurdynasty wrote:
| You don't need k8s for that, teams have been doing that for
| decades before k8s was ever a thing
| jupp0r wrote:
| Sure, but whatever they built themselves to accomplish
| this is also complicated. I know because I have built
| such systems (and replaced them by k8s).
| erulabs wrote:
| Hah, exactly. It's not that you can't accomplish all the
| same things as k8s with your own bash scripts - it's that
| k8s exists _to replace all your custom bash scripts_!
| jupp0r wrote:
| But you can still land on top of HN with a "I replaced
| kubernetes with this 500 line bash script without unit
| tests" blog article ;)
| vbezhenar wrote:
| Projects are optimized to be developed by so-called ordinary
| developers.
|
| We have python service which consumes gigabytes of RAM for
| quite simple task. I'm sure that I'd rewrite it with Rust to
| consume tens of megabytes of RAM at most. Probably even less.
|
| But I don't have time for that, there are more important things
| to consider and gigabytes is not that bad. Especially when you
| have some hardware elasticity with cloud resources.
|
| I think that if you can develop world-scale twitter which could
| run on a single computer, that's a great skill. But it's a rare
| skill. It's safer to develop world-scale twitter which will run
| on Kubernetes and will not require rare developer skills.
| yodsanklai wrote:
| > The thing most appealing about single server configs is the
| simplicity. The more simple a system easy, likely the more
| reliable and easy to understand.
|
| What if your unique machine crash?
| readonlybarbie wrote:
| [dead]
| andrewstuart wrote:
| Well you gotta have a backup strategy. I'm talking about the
| primary machine here, I assumed that would be obvious but
| maybe not. You build your failover strategy into your
| architecture - there's lots of ways to do it - I use Postgres
| so I would favor something based around log shipping.
| ffssffss wrote:
| And uptime is important, so you want to have that secondary
| running and ready, with a proxy in front of everything so
| you can switch as soon as you detect a failure. That's
| three hosts, plus your alerting has to be separate too, so
| that's four. Now, to orchestrate all this, we'll first get
| out Puppet...
| toast0 wrote:
| If you're going from one machine to two, and you add an
| automatically failover mechansim, chances are your load
| switching mechanism is going to cause more downtime than
| just running from your single machine, and manually
| switching on failure (after being paged).
| pclmulqdq wrote:
| You spin up a backup as the new "unique machine."
| fbdab103 wrote:
| I think you should always plan for failures, but modern
| enterprise hardware is quite reliable. I would even posit
| that if you stood up a brand new physical server today, it
| has a good chance of beating AWS uptime (well, not the AWS
| dashboard numbers) over a one year period.
| jupp0r wrote:
| "hardware is quite reliable" is not a valid strategy.
| Hardware fails with some non-zero probability. You need to
| have a plan in place what to do if that happens, taking
| into account service disruption, backups etc.
|
| Having a system in place that handles most of this
| gracefully (like kubernetes) is one way of having such a
| plan, there are others. Which one works best is dependent
| on your app, cost of downtime, your team that's tasked with
| bringing everything back up in the middle if the night,
| etc.
|
| People who leave details like this out when they say
| "kubernetes is complicated" just haven't seen the
| complexities of operating a service well.
| fbdab103 wrote:
| My first sentence was to always plan for failures.
| jupp0r wrote:
| Yeah sorry it wasn't meant to dispute your argument, just
| as an addition.
| Nextgrid wrote:
| Keep in mind that the complexity of your distributed
| system is a liability itself. Most cloud or large company
| downtimes aren't due to hardware failure per-se but
| either operator error or unexpected interactions between
| components of their distributed system. You might just be
| trading one source of downtime for another.
|
| > the complexities of operating a service well.
|
| Keep in mind that in a lot of business applications,
| downtime isn't the end of the world and might be an
| accepted and priced-in "cost" of doing business.
| Operating it as you consider "well" would just cost them
| more with no benefit.
| pixl97 wrote:
| It doesn't matter if hardware is 99.99% reliable if you're
| the .01% that day
| yazzku wrote:
| If I remember correctly, Lichess runs on a single server.
| brightball wrote:
| I tend to agree on simplicity. Really just depends on whether
| you can tolerate downtime for either outages or deployments.
|
| As soon as you start accounting for redundancy you have to fan
| out anyway.
| kbumsik wrote:
| > The thing most appealing about single server configs is the
| simplicity.
|
| In my experience this ended up with more complicated.
|
| Those systems are typically developed by people who already
| left and are undocumented, and they become extremely difficult
| to figure out the config (packages, etc files... oh, where even
| the service files are located?) and almost impossible to
| reproduce.
|
| It might be okay to leave it there, but when we need to modify
| or troubleshoot the system a nightmare begins...
|
| Maybe I was just unlucky, but at least k8s configs are more
| organized and simpler than dealing with a whole custom
| configured Linux system.
| Thaxll wrote:
| Because OP example is very simplistic and left on the table
| very important details, you would base 250M on a single
| machine? What about backups, obervability, how do you update
| that stack without bringing down everything ... Also this is
| napkin maths, this could be off by 10 or 100x which would
| change everything.
|
| It's very simple to make a PoC on a very powerful machine, make
| it ready from production serving hunderd of millions of users
| is completely different.
| raverbashing wrote:
| > What about backups
|
| Several ways of doing this without relying on k8s
|
| > observability
|
| This doesn't require k8s neither and it's more on your app.
| Systemd can restart systems by itself
|
| > how do you update that stack without bringing down
| everything
|
| That's probably where redundancy helps the most. I wouldn't
| run a big service without it (but again it found be at server
| level)
| threeseed wrote:
| > but again it found be at server level
|
| Can you educate us on how to have a resilient app with no-
| downtime updates at the server level.
|
| Because if you're doing this via software then it's no
| different to Kubernetes.
| raverbashing wrote:
| What I meant there is 2 servers + load balancer in front.
|
| Yeah, k8s might be the easiest way of doing this today if
| you want only 1 physical server
| PragmaticPulp wrote:
| It's worth noting that the author's example doesn't do
| anything like HTTP. It was purely an algorithmic benchmark.
|
| Nobody should be looking at this and thinking that it's
| realistic to actually serve a _functional website_ at this
| scale on a single machine with actual real world
| requirements.
| mpoteat wrote:
| As well, you should be creating regional servers to minimize
| latency for folks in other geographic regions. Can't beat c!
| sitkack wrote:
| It is a BoE system design. How is it off by 100x?
| jupp0r wrote:
| In addition, you should worry about what happens to your app
| if a hardware error, network problem or natural disaster
| makes your machine unavailable.
| eddsh1994 wrote:
| Split the DB from the app and replicate with a load
| balancer?
| threeseed wrote:
| That database is going to need to be clustered as well
| for resiliency.
|
| Sounds like you already have quite a number of different
| containers already.
| eddsh1994 wrote:
| Four vms isn't too bad to manage, I was the first dev at
| a startup that went prelaunch to acquisition from 2
| App/DB servers in EC2 for a mobile app that hit #1 on
| both stores with a fairly high user tx rate and some ML
| stuff.
| jupp0r wrote:
| Now you have "Production Twitter on three machines".
| Follow the rabbit and you end up with something like what
| Twitter looks like today.
| pixl97 wrote:
| Heh, also praying that everything stays in the fast path.
| If a small portion of the workload uses a higher portion of
| machine resources then the moment an attacker figures it
| out they have a great way of DDOSing your service
| resources.
| [deleted]
| fifilura wrote:
| I dont think k8s is what you are shooting at but the IPC that
| is required to run a set of microservices.
|
| Like kafka.
|
| My impression is that it is the serialisation that comes with
| each service-to-service communication that is really expensive.
| judge2020 wrote:
| Kubernetes and containers are a means to service architecture;
| It enabled scalability but does not require it. You should
| still be containerizing your applications to ensure a
| consistent environment, even if you only throw it in a docker-
| compose file on your production server.
| ownagefool wrote:
| So the goal is basically being able to do builds whilst
| running as few setup steps as possible.
|
| Containers are a good common denominator because you
| essentially start with the OS, and then there's a file that
| automates installing further dependencies and building the
| artifact, which typically includes the important parts of the
| runtime environment. - They're stupidly
| popular, so it basically nullifies the setup steps. -
| Once setup, they by combinding both OS layers and App, they
| solve more of the problem and are therefore slightly more
| reliable. - They're self-documenting as long as you
| understand bash, docker, and don't do weird shit like build
| an undocumented intermediary layer.
|
| Infrastructure as Code does the same thing for the underlying
| infra layers and kuberenetes is one of the nicer / quicker
| implementations of this, but requires you have kubernetes
| available.
|
| Together they largely solve the "works on my PC" problem.
| KronisLV wrote:
| > You should still be containerizing your applications to
| ensure a consistent environment, even if you only throw it in
| a docker-compose file on your production server.
|
| I'll say that this is a good point, especially because if you
| don't use containers or a similar solution (even things like
| shipping VM images, for all I care), you'll end up with
| environment drift, unless your application is a statically
| compiled executable with no system dependencies, like a
| JDK/.NET/Python/Ruby runtime or worse yet, an application
| server like Tomcat, all of which can have different versions.
| Worse yet, if you need to install packages on the system, for
| which you haven't pinned specific versions (e.g. needing
| something that's installed through apt/yum, rather than
| package.json or Gemfile, or requirements.txt and so on).
|
| That said, even when you don't use containers, you can still
| benefit from some pretty nice suggestions that will help make
| the software you develop easier to manage and run:
| https://12factor.net/
|
| I'd also suggest that you have a single mechanism for
| managing everything that you need to run, so if it's not
| containers and an orchestrator of some sort, at least write
| systemd services or an equivalent for every process or group
| of processes that should be running.
|
| Disclaimer: I still think that containers are a good idea,
| just because of how much of a dumpsterfire managing different
| OSes, their packages, language runtimes, application
| dependencies, application executables, port mappings,
| application resource limits, configuration, logging and other
| aspects is. Kubernetes, perhaps a bit less so, although when
| it works, it gets the job done... passably. Then again,
| Docker Swarm to me felt better for smaller deployments (a
| better fit for what you want to do vs the resources you
| have), whereas Nomad was also pretty nice, even if HCL sadly
| doesn't use the Docker Compose specification.
| vbezhenar wrote:
| When it comes to Java, everything could be used as a
| directory installation. Like you need JDK, maven and
| tomcat? Download and extract it somewhere. Modify your
| current PATH to include java and that's about it. You can
| build big tar.gz instead of OCI container which will work
| just fine.
|
| So IMO it's perfectly possible to run Java applications
| without containers. You would need to think about network
| ports, about resource limits, but those are not hard
| things.
|
| And tomcat even provides zero-downtime upgrades, although
| it's not that easy to set up, but when it works, it does
| work.
|
| After I've got some experience with Kubernetes, I'd reach
| for it always because it's very simple and easy to use. But
| that requires to go through some learning curve, for sure.
|
| The best and unbeatable thing about containers is that
| there're plenty of ready ones. I have no idea how would I
| install postgres without apt. I guess I could download
| binaries (where?), put them somewhere, read docs, craft
| config file with data dir pointing to anotherwere and so
| on. That should be doable but that's time. I can docker run
| it in seconds and that's saved time. Another example is
| ingress-nginx + cert-manager. It would take hours if not
| days from me to craft set of scripts and configs to
| replicate thing which is available almost out of the box in
| k8s, well tested and just works.
| KronisLV wrote:
| > When it comes to Java, everything could be used as a
| directory installation. Like you need JDK, maven and
| tomcat? Download and extract it somewhere. Modify your
| current PATH to include java and that's about it. You can
| build big tar.gz instead of OCI container which will work
| just fine.
|
| I've seen something similar in projects previously, it
| never worked all that well.
|
| While the idea of shipping one archive with _everything_
| is pretty good, people don 't want to include the full
| JDK and Tomcat installs with each software delivery,
| unlike with containers, where you get _some_ benefit out
| of layer re-use when they haven 't changed, while having
| the confidence that what you tested is what you'll ship.
| Shipping 100 app versions with the same JDK + Tomcat
| version will mean reused layers instead of 100 copies in
| the archives. And if you _don 't_ ship everything
| together, but merely suggest that release X should run on
| JDK version Y, the possibility of someone not following
| those instructions at least once approaches 100% with
| every next release.
|
| Furthermore, Tomcat typically will need custom
| configuration for the app server, as well as
| configuration for the actual apps. This means that you'd
| need to store the configuration in a bunch of separate
| files and then apply (copy) it on top of the newly
| delivered version. But you can't really do that directly,
| so you'd need to use something like Meld to compare
| whether the newly shipped default configuration doesn't
| include something that your old custom configuration
| doesn't (e.g. something new in web.xml or server.xml).
| The same applies to something like cacerts within your
| JDK install, if you haven't bothered to set up custom
| files separately.
|
| Worse yet, if people aren't really disciplined about all
| of this, you'll end up with configuration drift over time
| - where your dev environment will have configuration A,
| your test environment will have configuration B (which
| will _sort of_ be like A), and staging or prod will have
| something else. You 'll be able to ignore some of those
| differences until everything will go horribly wrong one
| day, or maybe you'll get degraded performance but without
| a clear reason for it.
|
| > So IMO it's perfectly possible to run Java applications
| without containers. You would need to think about network
| ports, about resource limits, but those are not hard
| things.
|
| This is only viable/easy/not brittle when you have self-
| contained .jar files, which admittedly are pretty nice!
| Though if shipping JDK with each delivery isn't in the
| cards (for example, because of the space considerations),
| that's not safe either - I've seen performance degrade
| 10x because of a JDK patch release was different between
| two environments, all because of JDK being managed
| through the system packages.
|
| Resource limits are generally doable, though Xms and Xmx
| lie to you, you'd need systemd slices or an equivalent
| for hard resource limits, which I haven't seen anyone
| seriously bother with, although they're at a risk of the
| entire server/VM becoming unresponsive should their
| process go rogue for whatever reason (e.g. CPU at 100%,
| which is arguably worse than OOM because of bad memory
| limits).
|
| Ports are okay when you are actually in control of the
| software and nothing is hardcoded. Then again, another
| aspect is being able to run multiple versions of software
| at the same time (e.g. different MySQL/MariaDB releases
| for different services/projects on the same node), which
| most nix distributions are pretty bad at.
|
| > And tomcat even provides zero-downtime upgrades,
| although it's not that easy to set up, but when it works,
| it does work.
|
| I've seen this attempted, but it never worked properly -
| the codebases might not have been good, but those
| redeployments and integrating with Tomcat always lead to
| either memory leaks or odd cases of the app server
| breaking. That's why personally I actually enjoy the
| approach of killing the entire thing alongside the app
| and doing a restart (especially good with embedded
| Tomcat/Jetty/Undertow), using health checks for routing
| traffic instead.
|
| I think doing these things at the app server level is
| generally just asking for headaches, though the idea of
| being able to do so is nice. Then again, I don't see
| servers like Payara (like GlassFish) in use anymore, so I
| guess Spring Boot with embedded Tomcat largely won, in
| combination with other tools.
|
| > After I've got some experience with Kubernetes, I'd
| reach for it always because it's very simple and easy to
| use. But that requires to go through some learning curve,
| for sure.
|
| I wouldn't claim that Kubernetes is simple if you need to
| run your own clusters, though projects like K3s, K0s and
| MicroK8s are admittedly pretty close.
|
| > The best and unbeatable thing about containers is that
| there're plenty of ready ones. I have no idea how would I
| install postgres without apt. I guess I could download
| binaries (where?), put them somewhere, read docs, craft
| config file with data dir pointing to anotherwere and so
| on. That should be doable but that's time. I can docker
| run it in seconds and that's saved time. Another example
| is ingress-nginx + cert-manager. It would take hours if
| not days from me to craft set of scripts and configs to
| replicate thing which is available almost out of the box
| in k8s, well tested and just works.
|
| This is definitely a benefit!
|
| Though for my personal needs, I build most (funnily
| enough, excluding databases, but that's mostly because
| I'm lazy) of my own containers from a common Ubuntu base.
| Because of layer reuse, I don't even need tricks like
| copying files directly, but can use the OS package
| manager (though clean up package cache afterwards) and
| pretty approachable configuration methods:
| https://blog.kronis.dev/articles/using-ubuntu-as-the-
| base-fo...
|
| In addition, my ingress is just a containerized instance
| of Apache running on my nodes, with Docker Swarm instead
| of Kubernetes: https://blog.kronis.dev/tutorials/how-and-
| why-to-use-apache-... In my case, the distinction between
| the web server running inside of a container and outside
| of a container is minimal, with the exception that Docker
| takes care of service discovery for me, which is
| delightfully simple.
|
| I won't say that the ingress abstraction in Kubernetes
| isn't nice, though you can occasionally run into
| configurations which aren't as easy as they should be:
| e.g. configuring Apache/Nginx/Caddy/Traefik certs which
| has numerous tutorials and examples online vs trying to
| feed your wildcard TLS cert into a Traefik ingress, with
| all of the configuration so that your K3s cluster would
| use it as the default certificate for the apps you want
| to expose. Not that other ingresses aren't great (e.g.
| Nginx), it's just that you're buying into additional
| complexity and I've personally have also had cases where
| removing and re-adding it hangs because of some resource
| cleanup in Kubernetes failing to complete.
|
| I guess what I'm saying is that it's nice to use
| containers for whatever the strong parts are (for
| example, the bit about being able to run things easily),
| though ideally without ending up with an abstraction that
| might eventually become leaky (e.g. using lots of Helm
| charts that have lots of complexity hiding under the
| hood). Just this week I had CI deploys starting to
| randomly fail because some of the cluster's certificates
| had expired and kubectl connections wouldn't work. A
| restart of the cluster systemd services helped make
| everything rotate, but that's another thing to think
| about, which otherwise wouldn't be a concern.
| andrewstuart wrote:
| I don't even use containers - I aim primarily for simplicity
| and so far I have found I am able to build entire
| sophisticated systems without a single container. Containers
| I find make things much more complex.
| John23832 wrote:
| That just means you don't know how to architect a machine
| with containers (and if you're effective in what you do,
| that's ok).
|
| But it's a pretty objective notation that manually scaled
| single machines don't scale as well as automation.
| fragmede wrote:
| Are you at least using Ansible or Chef or something?
| TillE wrote:
| Docker can be super simple. Like, if I want to run a Python
| service, that's just a few lines in a Dockerfile and a
| docker-compose.yml stub. Then I can trivially deploy that
| anywhere.
| paulryanrogers wrote:
| Simple to start. Yet it is more complex at run time,
| which can complicate (or simplify) debugging, depending
| on the problem.
| judge2020 wrote:
| Observability in production is where APM solutions like
| Datadog, Elastic, and Sentry come in; you can go from
| just logging errors all the way up to continuously
| profiling your application and beaming log files to them
| to correlate with metrics and database query timings.
|
| If you're just doing a simple application, Sentry really
| is the way to go, while Datadog and ELK are agent-based
| and more intended for complex setups and big enterprises
| (especially in their pricing structure/infra costs).
| tluyben2 wrote:
| By far most indeed. And if you want failover, just run 2 two
| machines. More fun (for me) also than tying together all that
| complexity.
| John23832 wrote:
| Kubernetes just orchestrates containers. You can still run
| beefy machines and scale (if necessary) accordingly.
|
| If anything, Kubernetes allows you to save cost by going with a
| scalable number of small, inexpensive, fully utilized machines,
| vs one large, expensive, underused one.
| sitkack wrote:
| I would wager that the majority of users of k8s do so on a
| cloud where they could provision VMs of the proper size to
| begin with. The utilization argument is specious.
| John23832 wrote:
| > The utilization argument is specious.
|
| It's not. Utilization is a key metric in capacity planning
| of large scalable apps.
|
| Capacity is based upon max utilization. A scaled web app is
| does not have constant utilization. The parent I was
| responding to suggested running on one large/face instance.
| Ok... if you're capacity planning, are you planning for
| peak rps or min rps? Obviously peak. Peak times are always
| a fraction of your total server uptime. This means one
| big/fast server would be underutilized most of the time.
|
| How do you expect to dynamically vertically scale in cloud
| to fit demand while using a single server? Re-provision
| another server (either smaller or larger), redeploy all
| apps to the server, and then route traffic? Great, you're
| doing kubernetes job by hand.
| pikdum wrote:
| I go with k8s even on a single server nowadays, it just makes
| everything so much more convenient.
|
| https://k3s.io/ makes it really easy to set up, too.
| vbezhenar wrote:
| I never tried k3s, but what's wrong with kubeadm? I think
| that's literally two commands to run single server k8s:
| kubeadm init and kubectl taint something.
|
| The only thing bad about single server kubernetes is that
| it'll eat like 1-2 GB of RAM by itself. When you whole server
| could be 256 MB, that's a lot of wasted RAM.
| pikdum wrote:
| Nothing wrong with kubeadm, but k3s should be a bit more
| lightweight.
| samsquire wrote:
| I recommend this table of latency figures for any software
| engineer:
|
| https://gist.github.com/jboner/2841832
|
| Essentially IO is expensive except within a datacenter but even
| in a data center, you can do a lot of loop iterations in a hot
| loop in the time it takes to ask a server for something.
|
| There is a whitepaper which talks about the raw throughput and
| performance of single core systems outperforming scalable
| systems. These should be required reading of those developing
| distributed systems.
|
| http://www.frankmcsherry.org/assets/COST.pdf A summary:
| http://dsrg.pdos.csail.mit.edu/2016/06/26/scalability-cost/
| eatonphil wrote:
| This is a great exercise in napkin math, even with constraints
| you've set for yourself that don't fully approximate Twitter
| (yet). Thanks!
| habibur wrote:
| He will be up for surprise.
|
| HTTP with connection: keep-open can serve 100k req/sec. But
| that's for one client being served repeatedly over 1 connection.
| And this is the inflated number that's published in webserver
| benchmark tests.
|
| For more practical down to earth test, you need to measure
| performance w/o keep-alive. Request per second will drop to 12k /
| sec then.
|
| And that's for HTTP without encryption or ssl handshake. Use
| HTTPS and watch it fall down to only 400 req / sec under load
| test [ without connection: keep-alive ].
|
| That's what I observer.
| [deleted]
| trishume wrote:
| I agree most HTTP server benchmarks are highly misleading in
| that way, and mention in my post how disappointed I am at the
| lack of good benchmarks. I also agree that typical HTTP servers
| would fall over at much lower new connection loads.
|
| I'm talking about a hypothetical HTTPS server that used
| optimized kernel-bypass networking. Here's a kernel-bypass HTTP
| server benchmarked doing 50k new connections per core second
| while re-using nginx code: https://github.com/F-Stack/f-stack.
| But I don't know of anyone who's done something similar with
| HTTPS support.
| jck wrote:
| I once built a quick and dirty load testing tool for a public
| facing service we built. The tool was pretty simple -
| something like https://github.com/bojand/ghz but with traffic
| and data patterns closer to what we expected to see in the
| real world. We used argo-workflows to generate scale.
|
| One thing which we noticed was that there was a considerable
| difference in performance characteristics based on how we
| parallelized the load testing tool (multiple threads,
| multiple processes, multiple kubernetes pods, pods forced to
| be distributed across nodes).
|
| I think that when you run non-distrubuted load tests you
| benefit from bunch of cool things which happen with http2 and
| Linux (multiplexing, resource sharing etc) which might make
| applications seem much faster than they would be in the real
| world.
| sayrer wrote:
| Userspace networking is pretty common. The chair of the IETF
| even wrote one: https://github.com/NTAP/quant
|
| "Quant uses the warpcore zero-copy userspace UDP/IP stack,
| which in addition to running on on top of the standard Socket
| API has support for the netmap fast packet I/O framework, as
| well as the Particle and RIOT IoT stacks. Quant hence
| supports traditional POSIX platforms (Linux, MacOS, FreeBSD,
| etc.) as well as embedded systems."
| lossolo wrote:
| TLS handling would dominate your performance, kernel
| bypassing would not help here unless you would also do TLS
| NIC offloading, you still need to process new TLS sessions
| from OP example and they would dominate your http processing
| time (excluding application business logic processing).
| ilyt wrote:
| using what ? That numbers are on low side even for my old
| desktop
| [deleted]
| pixl97 wrote:
| And I would say real life Twitter involves mostly cell phone
| use where we see companies like Google try to push HTTP/3 to
| deal with head of line issues on lossy connections. Serving at
| the millions of hits per day on lossy networks is going to
| leave you with massive numbers of connections that have been
| abandoned but you don't know it yet. Or connections that are
| behaving like they are tar pitted and running at bits per
| second.
| saagarjha wrote:
| I don't believe Twitter ever got around to rolling out HTTP/3
| to their clients.
| ilyt wrote:
| That's so low overhead compared to everything else needed
| that it is near-irrelevant
| hinkley wrote:
| Vertical scaling doesn't have to be a single machine. You can
| do a lot with a half dozen machines split for different
| responsibilities, like we did in the 90's and 00's. Database,
| web servers, reverse proxy.
| lossolo wrote:
| > Use HTTPS and watch it fall down to only 400 req / sec under
| load test [ without connection: keep-alive ].
|
| I'm running about 2000 requests/s in one of my real-world
| production systems. All of the requests are without keep-alive
| and use TLS. They use about one core for TLS and HTTP
| processing.
| brazzledazzle wrote:
| Is this static or dynamic content? Are the simulated load
| test clients requesting the exact same pages/resources?
| lossolo wrote:
| All dynamic content, all hitting data storage. There are no
| simulated clients, this is all real traffic from real
| clients, a lot of requests do writes, some do only reads.
| habibur wrote:
| Fascinating. Any special optimization you are using, or is it
| from off the shelf software and with standard configuration?
| lossolo wrote:
| For offloading SSL I use haproxy with some custom settings
| and a few non standard kernel settings.
| kijin wrote:
| Sounds totally off-the-shelf.
|
| I have a basic LAMP server running on a 4-core VM on a
| laptop. I just threw ApacheBench at it (not the fastest
| benchmarking tool, either -- it eats up 1 core all by
| itself), and it handles 1200 req/s TLS with no keepalive,
| and 3400 req/s with keepalive. This stuff scales linearly
| with core count, so I wouldn't be surprised to see much
| higher numbers in real servers.
| tpetry wrote:
| Are these all _new_ TLS connections? Because most
| benchmarks use TLS resumption which means the TLS
| handshake was only done once!
| lossolo wrote:
| In my case, all of TLS connections are new and from real
| clients (not a benchmark/test)
| kijin wrote:
| AFAIK apachebench is so old it doesn't support TLS
| resumption, but I might be wrong.
| brazzledazzle wrote:
| Is AB running on the same machine during the tests?
| kijin wrote:
| Same machine, pegging a CPU core the whole time. So the
| server side only has 3 cores left to use.
| JanisErdmanis wrote:
| I guess that the biggest chunk of a slowdown from TLS comes due
| to group operations alone. So wouldn't it be practical to
| configure TLS for session resumption and limit the number of
| handshakes per second it could do?
| keewee7 wrote:
| In the coming years we will probably see a lot of complicated
| microservice architectures be replaced by well-designed and
| optimized Rust (and modern C++) monoliths that use simple
| replication to scale horizontally.
| pixl97 wrote:
| Replication and simple never belong in the same sentence. DNS
| which is one of the simplest replication systems I know of has
| its own complex failure modes.
| qaq wrote:
| It can be outsourced complexity as in just using horizontally
| scalable database like Spanner, CockroachDB etc.
| pixl97 wrote:
| All complexity must be paid for and typically the more
| complex the problem the higher the cost to manage.
| TideAd wrote:
| CockroachDB is nice to use but every database has
| complexity you have to deal with.
|
| Here's one I ran into recently: if a range has only 1 of 3
| replicas online then it will stop accepting traffic for
| that range until it has 3 replicas again.
|
| (for the folks at home, "range" is a technical term for 512
| bit slice of the data - CRDB replicates at the range level)
|
| So, in some code I wrote, I had account for not only 1) the
| whole DB being unavailable but also 2) just one replica
| being unavailable (they're different failure modes that say
| different things about the health of the system).
|
| It's a good behavior! Good for durability. But I had to do
| some work to deal with it, spend an hour coming up with a
| solution, etc. There are databases that work at Twitter
| scale but no there are no silver bullets among those that
| do. You need full time engineers to manage the complexity
| and keep it online, or else it could cost the company
| shitloads of money - I've seen websites of similar scale
| where a two-hour outage cost them $20 million.
| riku_iki wrote:
| You could have more replicas,if availability is
| critical?..
| Nextgrid wrote:
| You don't even need Rust or highly optimised code. Just moving
| the existing code from "vCPUs" and networked storage to _real_
| CPUs with direct-attach NVME storage will be enough for most
| purposes. (btw you can do that now, just get yourself a beefy
| server at OVH /Hetzner and play around with it)
| hinkley wrote:
| Yes, but it won't be because they're better, it'll be because
| they're different.
|
| Like last time, and the time before that, and the time before
| that, and the time before that.
| Hasu wrote:
| You're right, no progress has ever been made in software, no
| new ideas are better than any old ideas, and it's fads all
| the way down. The only difference between software today and
| software in 1980 is that today's software is hip and software
| from 1980 is square.
|
| I understand the frustration with flavor of the week "best
| practices" and the constant churn of frameworks and ideas,
| but software engineering as a practice IS moving forward. The
| difficulty is separating the good ideas (CI/CD, for example)
| from the trends (TDD all the things all the time) ahead of
| time.
| Cyph0n wrote:
| More like "barebones, in-memory, English-only Twitter clone on
| one machine".
|
| Edit: Still a nice writeup!
| Dylan16807 wrote:
| What strikes you as English-only about it?
| Cyph0n wrote:
| My bad - not English, but ASCII. The assumed max tweet size
| is in bytes rather than (UTF) characters.
| trishume wrote:
| I specifically assumed a max tweet size based on the
| maximum number of UTF-8 bytes a tweet can contain (560),
| with a link to an analysis of that, and discussion of how
| you could optimize for the common case of tweets that
| contain way fewer UTF-8 bytes than that. Everything in my
| post assumes unicode.
| Tepix wrote:
| Did you consider URLs? They don't seem to count against
| tue size and can be very large indeed (like 4k)
| modeless wrote:
| URLs are shortened and the shortened size counts against
| the tweet size. The URL shortener could be a totally
| separate service that the core service never interacts
| with at all. Though I think in real twitter URLs may be
| partially expanded before tweets are sent to clients, so
| if you wanted to maintain that then the core service
| would need to interact with the URL shortener.
| KingLancelot wrote:
| UTF-8 is 1-4 bytes per codepoint dude, not 1-2.
|
| 280 * 4 = 1120, not 560.
| Dylan16807 wrote:
| Twitter doesn't allow 280 of any codepoint. The more
| complicated ones count double against your limit.
| KingLancelot wrote:
| Ah, so Twitter's Unicode implementation is fucked.
| Dylan16807 wrote:
| Nothing is wrong with it.
|
| If it counted UTF-16 code units that would be dumb. It
| doesn't. The cutoff was deliberately set to keep the 140
| character limit for CJK but increase it to 280 for the
| rest. And they did that based on observational data.
|
| https://cdn.cms-twdigitalassets.com/content/dam/blog-
| twitter...
|
| https://blog.twitter.com/en_us/topics/product/2017/Giving
| -yo...
| Cyph0n wrote:
| Thanks for clarifying. I missed the max vs. average
| analysis because I was focused on the text. Still, as
| noted in the Rust code comment, the sample implementation
| doesn't handle longer tweets.
| Dylan16807 wrote:
| That size in bytes is based on the max size in UTF-8 and
| UTF-16. Codepoints below U+1100 are counted as one
| "character" by twitter and will need at most 2 bytes.
| Codepoints above it are counted as two "characters" by
| twitter and will need at most 4 bytes. Therefore 560 bytes,
| and it supports all languages.
|
| Side note, this is more pessimistic than it needs to be, if
| you're willing to transcode. The larger codepoints fit into
| 20-21 bits, and the smaller ones fit into 12-13 bits.
| Cyph0n wrote:
| I was referring to the comment in the Rust code, not the
| analysis.
| halfmatthalfcat wrote:
| As in the instance will only be tuned to serve one language?
| mizzao wrote:
| While you might not want to do this with actual Twitter, any sort
| of high-performance computing workload can run substantially
| faster on a single optimized machine than on a distributed
| computing environment.
|
| I learned this the hard way when I was running a medium-sized
| MapReduce job in grad school that was over 100x faster when run
| as a local direct computation with some numerical optimizations.
| robluxus wrote:
| Obligatory "you don't need Hadoop":
| https://news.ycombinator.com/item?id=14401399
| castratikron wrote:
| Does 4chan fit on one machine?
| SilverBirch wrote:
| I think one of the under-estimated interesting points of twitter
| as a business is that this is the core. Yes, Twitter is 140
| characters, it's got "300m users" which is probably 5m real heavy
| users. So yes, you could do a lot of "140 characters, a few
| tweets per person, few million users" on very little hardware.
| But that's why Twitters a shit business!
|
| How much RAM did your advertising network need? Becuase _that_ is
| what makes twitter a business! How are you building your
| advertiser profiles? Where are you accounting for fast roll out
| of a Snapchat /Instagram/BeReal/Tiktok equivalent? Oh look, your
| 140 characters just turned into a few hundreds megs of video that
| you're going to transcode 16 different ways for Qos. Ruh Roh!
|
| How are your 1,000 engineers going to push their code to
| production _on one machine_?
|
| Almost always the answer to "do more work" or "buy more machines"
| is "buy more machines".
|
| All I'm saying is I'd change it to "Toy twitter on one machine"
| not Production.
| Nextgrid wrote:
| If it actually takes you a single machine to run this, you
| don't really need an advertising network to fund it. Out of 5M
| users (let alone the theoretical 300M) there will be enough
| people who'd be happy to pay for verification or an exclusive
| badge on their profile.
|
| > How are your 1,000 engineers going to push their code to
| production on one machine?
|
| That might actually be the reason why Twitter barely keeps
| afloat. 1k engineers for a product that's already built and
| hasn't fundamentally changed nor evolved in years makes me
| wonder what business value those engineers actually provide.
| reacharavindh wrote:
| The author claimed early on, and very clearly that this was a
| fun exercise of thought and engineering rather than saying
| "Look this is how Twitter should be run". After all this is
| Hacker News. Such exercises, and engaging other hackers to pick
| something out of there is how we progress(and get our tickles).
| So, may be instead think about how one could tackle the
| advertising/indexing needs in a similar fashion(could it be
| done in just another server? 5 more servers?)..
| SilverBirch wrote:
| Yeah I completely get that, but I think a lot of hacker news
| tends to think of companies as the sum of their engineering
| resources, rather than what they really are. Which is weird,
| because it's hosted by YC which is meant to be the polar
| opposite of that. The point of my comment was that the
| decision making that lead to Twitter's design actually make
| sense when you understand the business model behind them.
| It's not good enough to just run a webpage.
| irq wrote:
| Excellent article! I wish the font size on mobile was bigger.
| kierank wrote:
| This is as realistic as the moon rocket in my back garden.
| fleddr wrote:
| "Through intense digging I found a researcher who left a notebook
| public including tweet counts from many years of Twitter's 10%
| sampled "Decahose" API and discovered the surprising fact that
| tweet rate today is around the same as or lower than 2013! Tweet
| rate peaked in 2014 and then declined before reaching new peaks
| in the pandemic. Elon recently tweeted the same 500M/day number
| which matches the Decahose notebook and 2013 blog post, so this
| seems to be true! Twitter's active users grew the whole time so I
| think this reflects a shift from a "posting about your life to
| your friends" platform to an algorithmic content-consumption
| platform."
|
| I know it's not the core premise of the article, but this is very
| interesting.
|
| I believe that 90% of tweets per day are retweets, which supports
| the author's conclusion that Twitter is largely about reading and
| amplifying others.
|
| That would leave 50 million "original" tweets per day, which you
| should probably separate as main tweets and reply tweets. Then
| there's bots and hardcore tweeters tweeting many times per day,
| and you'll end up with a very sobering number of actual unique
| tweeters writing original tweets.
|
| I'd say that number would be somewhere in the single digit
| millions of people. Most of these tweets get zero engagement.
| It's easy to verify this yourself. Just open up a bunch of rando
| profiles in a thread and you'll notice a pattern. A symmetrical
| amount of followers and following typically in the range of
| 20-200. Individual tweets get no likes, no retweets, no replies,
| nothing. Literally tweeting into the void.
|
| If you'd take away the zero engagement tweets, you'll arrive at
| what Twitter really is. A cultural network. Not a social network.
| Not a network of participation. A network of cultural influencers
| consisting of journalists, politicians, celebrities, companies
| and a few witty ones that got lucky. That's all it is: some tens
| of thousands of people tweeting and the rest leeching and
| responding to it.
|
| You could argue that is true for every social network, but I just
| think it's nowhere this extreme. Twitter is also the only
| "social" network that failed to (exponentially) grow in a period
| that you might as well consider the golden age of social
| networks. A spectacular failure.
|
| Musk bought garbage for top dollar. The interesting dynamic is
| that many Twitter top dogs have an inflated status that cannot be
| replicated elsewhere. They're kind of stuck. They achieved their
| status with hot take dunks on others, but that tactic doesn't
| really work on any other social network.
| yodsanklai wrote:
| > Musk bought garbage for top dollar
|
| Totally out of topic here, but could be he just wants the
| ability to amplify his own ideas. Also, why measure Twitter
| value (arbitrarily?) by number of unique tweets, rather than by
| read tweets?
| fleddr wrote:
| Doing irrational things for kicks is very much on brand for
| Musk, but in this case he did want to back out of the deal,
| but couldn't.
___________________________________________________________________
(page generated 2023-01-08 23:02 UTC)