[HN Gopher] GitHub - system-design-primer: Learn how to design l...
___________________________________________________________________
GitHub - system-design-primer: Learn how to design large-scale
systems
Author : msolujic
Score : 96 points
Date : 2023-10-29 07:41 UTC (15 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| alok-g wrote:
| Meta comment:
|
| We have come to very narrow definition of a 'system' when using
| terms like system design, system architect, etc., which here
| seems to mean a distributed computer system. That, as compared to
| a more open-ended definition found on, say, Wikipedia [1]. What
| about biological systems like a cell or a human body, a road
| system, a sound system, ...
|
| [1] https://en.m.wikipedia.org/wiki/System
| qznc wrote:
| Working in the automotive sector, I find it even worse.
| Sometimes "system" means an ECU (e.g. a radar sensor),
| sometimes the whole car, sometimes even includes infrastructure
| (cloud, other cars, etc). Sometimes it is just a software
| component within an ECU (e.g. the operating system).
|
| If I could I would taboo the term. It conveys practically zero
| information at this point.
| charcircuit wrote:
| >which here seems to mean a distributed computer system.
|
| No, it can also be a design between multiple services on the
| same machine. Or it can be a design between multiple subsystems
| of a single application.
| petepete wrote:
| Slightly confusingly, this is nothing to do with GitHub's Primer
| design system.
|
| https://primer.style/
| esafak wrote:
| I think the "large scale" qualifier should dispel confusion.
| bananapub wrote:
| the Google NALSD guides are also worth reading for concrete-ising
| these sort of designs, too. there's some stuff in the SRE
| Workbook: https://sre.google/workbook/non-abstract-design/ and
| much more online.
| rollulus wrote:
| This thing has 233k stars. I dare to say, without evidence, that
| the vast majority of devs does not know what e.g. the CAP theorem
| is----and they'll never need it. The vast majority of "systems"
| are small-scale. Who's this massive audience?
| brunooliv wrote:
| People who live for short tenured stays at FAANG companies and
| optimize their time and skills exclusively for interviews?
| robervin wrote:
| Still seems quite high, but I imagine a lot of tech founders
| reach for tools like this if they've not been tasked with
| designing full systems before.
| mediascreen wrote:
| I would guess that it's probably aspirational to some degree.
| People hope to work with larger systems.
|
| Personally I think it looks interesting and I work at a small
| web agency. We usually only work with a few parts of these
| kinds of systems, but occasionally we have clients like a large
| sports ticket vendor, where a few more of these patterns are
| helpful.
| nunez wrote:
| Job hoppers trying for staff+ roles at FAANGs or VC-backed mid-
| stage startups.
|
| Also, to add to your comment about CAP. knowing CAP
| (consistency, availability, partition tolerance, choose two)
| can be useful when choosing which database to use for a
| production system, I.e. can you get away with a fancy
| distributed gossip database or do you need the robustness of a
| rdbms.
|
| 99.9% of the time a RDBMS is fine, but if you're vying for a
| senior staff role, you need to sound like you can talk about
| the 0.01% of the time where Riak or something like that makes
| sense as a primary data store. That's why this guide mentions
| it.
| lifeisstillgood wrote:
| I fundamentally distrust these sort of guides. It was brought
| home reading something on the google SRE book - basically they
| talked about how google started with two data centres and a bunch
| of python scripts as automation.
|
| Did they do "up-front" design? Of course. Did they get it
| horribly wrong at times ? of course.
|
| but the large working system evolved from a small working system.
|
| Thinking you can cunningly plan a future system to hang together
| like some giant zipper with pieces as yet to be built pulling
| together perfectly is ludicrous.
| rajaman0 wrote:
| You are comparing apples and oranges.
|
| These interviews are for engineers interviewing at companies
| which have scale ; not for a 0-1 growth of a product like your
| google example.
|
| When you're launching products at scale, you absolutely need to
| design large systems that are durable and can operate at said
| scale. And when I'm hiring, I need to filter for folks who are
| able to rationalize at that scale.
| lifeisstillgood wrote:
| Well as you are the FAANG hiring manager I expect I am going
| to lose the argument but I wonder if my oranges are more like
| apples than you suspect.
|
| I see there being two options here
|
| apples: please take the existing set of distributed
| components that evolved in tandem at this company and design
| a new application on top of and using those components
|
| oranges: please start from scratch and explain how you would
| build a set of components that will work as well as those in
| 1.
|
| Apples is what we mostly look for - in any org we have a
| scale and ecosystem and we don't want to throw it all out. If
| it must evolve it evolves in tandem with rest of the
| ecosystem. So if we have some scalable data layer now that
| has chosen one side of CP/AP and then we come along and say
| actually throw that out We need full concistency all the time
| for this HR app, we are going to build our own different CP
| data layer globally, hold my beer, then ...
|
| The oranges part is fine - it tells us if someone has
| actually understood the principles behind apples. But
| honestly it's a fake out of we think oranges can actually be
| _done_. And this sort of stuff gives the impression you can
| design the application solution by also designing the
| ecosystem at the same time. That 's the bit Inwant to
| emphasise - you build the platform then design the
| application on the quirks of what you built
|
| And anyone who has not used the apples components won't
| understand their quirks and can come unstuck when the well
| known but non obvious behaviour strikes.
|
| I am being too vague here. I feel there is an interesting set
| of discussions to come out of it - Inwill reread the article
| nunez wrote:
| This is super helpful as a primer (especially the bits about
| knowing which kinds of infra go where and doing capacity planning
| BEFORE architecting anything) but is not a replacement for
| actually having done it somewhere, since there are so many tools
| and tricks that are difficult to convey in a single guide.
|
| I say this because this guide is often used by engineers trying
| to job hop their way into staff+ roles without having the
| requisite experience to back that up.
|
| Put another way, this guide will help you answer "design
| Netflix," (I.e. narrow the scope, capacity plan, diagram the
| ideal path, replace with infrastructure, discuss tradeoffs) but
| will NOT help you answer "tell me about a time where you designed
| a large-scale system and how you would've made it better given
| the chance"
| marginalia_nu wrote:
| I had a look at the "design a crawler" part, since that's
| something I have experience with. It makes some pretty sketchy
| assumptions and weird design choices.
|
| First in the back-of-the-envelope calculation it overestimates
| the size of a document by about two orders of magnitude. It's
| closer to 7.5 KB than 500 KB on average, _uncompressed_. 500 KB
| is closer to the size of Homer 's Odyssey, which I will state is
| much longer than most websites. I'm willing to give them the
| benefit of the doubt though, maybe I'm misunderstanding what
| these 500 KB is supposed to consist of.
|
| I also don't think the crawl frontier design would work at the
| sort of scales they imagine. This is a design that will get you
| to maybe 10 million documents.
|
| Apart from the profoundly questionable choice of using a nosql
| database for this, apart from the race condition in the crawling
| algorithm, Redis can only store 4 billion (2^32-1) items in a
| set, and let me tell you, crawl frontiers grow very quickly when
| you design them like this one has been designed. By the time
| you're indexing a billion documents, your crawl frontier will be
| of many orders of magnitude more.
|
| The biggest mistake is to think too link-oriented. You probably
| want to retrieve links one domain at a time as this permits
| connection pooling (otherwise your biggest crawling bottleneck
| will be waiting for the remote end to acknowledge the
| connection's closure), helps with deduplication and means you
| don't need to re-fetch robots.txt for every single document you
| retrieve. It also solves many problems limiting how much you
| fetch from a single domain, since you can just set a hard time
| and link limit.
|
| You don't have to re-fetch every document on that domain when you
| refresh it, just get the new ones and sample the existing ones.
| This also means you don't need a billion-entry priority queue,
| you can get away with a manageable queue of a few million
| instead.
| lokesh1729 wrote:
| This guide gives the overview of breadth of concepts. This is a
| good starting point.
|
| There's one more github repo called "awesome-scalability" which
| has the good blog posts by concept.
___________________________________________________________________
(page generated 2023-10-29 23:01 UTC)