[HN Gopher] Caching is an abstraction, not an optimization
___________________________________________________________________
Caching is an abstraction, not an optimization
Author : samuel246
Score : 129 points
Date : 2025-07-01 10:42 UTC (3 days ago)
(HTM) web link (buttondown.com)
(TXT) w3m dump (buttondown.com)
| ckdot2 wrote:
| "I think now caching is probably best understood as a tool for
| making software simpler" - that's cute. Caching might be
| beneficial for many cases, but if it doesn't do one thing then
| this is simplifying software. There's that famous quote "There
| are only two hard things in Computer Science: cache invalidation
| and naming things.", and, sure, it's a bit ironical, but there's
| some truth in there.
| bell-cot wrote:
| (You forgot off-by-1 errors.)
|
| All software has to name things, and count. Caching (including
| invalidation) is best understood as a liability. If you can
| foist it off on your CPU and OS and DB, good for you.
| Programming whatever you're actually trying to get done is
| already hard enough.
| yxhuvud wrote:
| Off by 1-errors is not part of the original quote, but is
| just a later addon to make it funny.
|
| They also tend not to be very hard.
| TeMPOraL wrote:
| Except when they're part of some base assumptions in the
| domain or dozen of layers of abstractions below you. They
| are hard to _prevent from happening_.
| tombert wrote:
| They're not hard but I will say that when I was writing an
| app that was using both JavaScript and Julia, I kept
| getting off-by-one errors because Julia starts at 1 instead
| of 0.
|
| Really the only time in my entire professional career that
| off-by-one errors have actually given me headaches.
| bobthepanda wrote:
| I think that is from a time when popular programming
| languages varied in their behavior and also when people
| were writing for loops with incrementation all the time.
|
| A lot of languages have just settled on zero indexing,
| and many now have some variation of for/each or for/of
| that would eliminate a lot of potential ways to encounter
| this error.
| tombert wrote:
| Yeah, I mostly will do map/reduce/filter when possible,
| and obviously in those cases indexes don't matter. It
| could start at index 12345 for all I care with that
| stuff.
|
| Occasionally, though, I need to use the same index across
| multiple items, there's not a trivial means in which to
| zip, and at that point I have to use an old school for
| loop. That's when the 1-index vs 0-index bites me.
| Cthulhu_ wrote:
| If you omit the off-by-1 error from the two hard things joke,
| you're still off by 1, right? Kind of?
| whateveracct wrote:
| caching often does simplify software though when done well
|
| and - as the OP suggests - it works best when the cache is a
| well-defined abstraction with properties and rules about how it
| works
|
| just because "caching" is mentioned in a meme doesn't mean it
| can't be true that it can simplify software
| meesles wrote:
| > caching often does simplify software though when done well
|
| I have to push back here, I think this is objectively untrue.
| By definition a system or piece of code on where you add a
| condition where something else happens (cache) that behaves
| differently than the uncached path increases complexity.
|
| I'm not saying it's wrong to cache things or that they aren't
| useful, but I think they absolutely are an abstraction and an
| optimization at the cost of complexity. Good code bases hide
| complexity from the devs all the time, so it's not a question
| of whether you can code it away, but rather how difficult is
| it to troubleshoot the internals of the system.
| PaulHoule wrote:
| Trying some other way to explicitly manage multiple storage
| tiers could get pretty complicated.
| jameshart wrote:
| If you hide caching away as an implementation detail behind
| an abstraction, it comes back and bites you as a leaky
| abstraction later.
|
| Look at how CPU cache line behaviors radically change the
| performance of superficially similar algorithms.
|
| Look at how query performance for a database server drops off
| a cliff the moment the working cache no longer fits in
| memory.
|
| Hiding complexity can be a simplification, until you exceed
| the bounds of the simplification and the complexity you hid
| demands your attention anyway.
| atq2119 wrote:
| CPUs are still a great example for how caching simplifies
| things.
|
| There's a long history in computer architecture of cores
| and accelerators that don't have a cache but instead rely
| on explicitly programmed local scratchpads. They are
| universally more difficult to program than general purpose
| CPUs because of that.
| hmottestad wrote:
| I'm sure the CPU designers would love it if they didn't
| need several different layers of cache. Or no cache at
| all. Imagine if memory IOPS were as fast as L1 cache, no
| need for all that dedicated SRAM on the chip or worry
| about side channel attacks.
| atq2119 wrote:
| Sure, but we were talking about the perspective of
| software developers. The hardware designers take on
| complexity so that the software developer's work can be
| simpler.
| aswanson wrote:
| I guess simplification needs to include "at what level" as a
| qualifier.
| ckdot2 wrote:
| That abstraction is another layer though. And additional
| layers are additional complexity. So, if you add another
| layer, the software is less simple than before. You might
| need to have caching in your software. I don't doubt that.
| But there's simply no way it makes the software more simple
| except if you assume some unfortunate starting point where
| you could get rid of any high-complex performance
| optimizations in your existing code by replacing them with a
| more simple cache solution. But then the statement should be
| "refactoring makes your code simpler".
| whateveracct wrote:
| additional layers (or software in general) are not
| inherently additional complexity
| TeMPOraL wrote:
| In some sense they are, since establishing an abstraction
| is strictly additive. Abstractions help _manage
| complexity_.
| moritzwarhier wrote:
| Getting cache keys or caching events wrong is easy and a
| nightmare.
|
| But getting them right can easily cross the boundary of
| purely optimizing performance towards simplifying public API
| of something. I think this is true.
|
| I'd imagine an involved example where semantics and caching
| really start to offer a trade-off.
|
| Imagine that somehow querying the actual meteorological data
| is quite expensive, and consider this badly written
| pseudocode (equals sign denoting default parameters):
|
| - measureCurrentTemparature()
|
| - retrieveAccurateTemperatureForNanoSecond(momentInTime)
|
| -> cached abstractions which would access cached data:
|
| - getTempearature(moment = now(), tolerance = 1min)
|
| - getCurrentTemperature(tolerance = MIN_TOLERANCE)
|
| I know, reality is much more complicated, and using time
| (seeing it as quasi-continuous) as a caching parameter is
| already stretching it so far.
|
| Just a stupid example that came to my mind.
|
| I've bitten myself in the ass with caching rasterized
| reprentations of images more than once, where the input were
| SVG images or limited formats that convert to SVG.
| fastball wrote:
| Caching is a performance improvement. There is no software
| that _requires_ caching, therefore it is always something
| being added on top of the business logic that is
| fundamentally required. As such, a cache is increasing
| complexity by nature of its existence.
|
| The only scenario where it would simplify software is if a
| bunch of complex (non-cache) things are being done to improve
| perf, and a cache would be the simpler solution. But in that
| case the simplifying step is not _adding a cache_ , it is
| _removing complex things that aren 't actually required_.
| After that you add a cache to improve performance (which
| increases complexity but is worth it for this imagined use-
| case). But maybe you remove the complex perf shenanigans, and
| realize that perf is still "good enough" even without a
| cache, keeping your software even simpler.
| EGreg wrote:
| I never understood about cache invalidation or naming things
|
| Both are not that difficult, honestly.
|
| Aren't there a lot harder things out there
| gryfft wrote:
| It's a little bit tongue in cheek; no one is seriously
| suggesting it's harder than P=NP or the problem of
| consciousness. But there's something a bit "death and taxes"
| to the inevitability that any large enough project is going
| to have some corner cases involving these old chestnuts.
|
| Heck you can probably prove that any system for naming things
| is either inconsistent or incomplete.
| TeMPOraL wrote:
| > _no one is seriously suggesting it 's harder than P=NP or
| the problem of consciousness._
|
| Well, I for one feel that "naming things" ultimately boils
| down to the latter, which may or may not be harder than the
| former.
| Valodim wrote:
| In my experience, the larger the software you write, the
| truer these become. At some point all obvious names will have
| collisions, and getting caching right is crucial to do but
| difficult to achieve because it transcends the entire stack.
|
| You could group these two things into "getting the data model
| right" as the single hard thing, perhaps that rings more true
| to you :)
| quuxplusone wrote:
| For "only two hard problems," read "two candidates for among
| the hardest problems (but we feel strongly that these are
| indeed good candidates)," or something along those lines,
| more or less.
|
| It's also possible that these _used_ to be the only two hard
| problems at the time the aphorism was first recorded, but the
| underlying state of the world has changed since then and the
| aphorism, as recorded, is no longer current.
| IshKebab wrote:
| Cache invalidation isn't hard _in theory_. It 's just one of
| those things that is very easy to get subtly wrong and
| difficult to test.
|
| Think about all those times your program isn't building and
| `make clean` fixes it.
| throwaway150 wrote:
| > Think about all those times your program isn't building
| and `make clean` fixes it.
|
| I don't think that's a good example. I've worked with more
| than 20 different build tools by now, and I cannot recall a
| single instance where the problem actually came down to
| cache invalidation. Every time I dug into it, the real
| cause was something else: a mistake in the build script, an
| incorrectly declared dependency, or something similar.
|
| So when you say "think about all those times", not a single
| such time comes to my mind!
| degamad wrote:
| I think the idea is that if make clean allows the build
| to complete correctly, then the underlying issue is that
| the "mistake in the build script, incorrectly declared
| dependency, or similar" was causing the cached build
| results to not be invalidated when they should have.
| IshKebab wrote:
| > an incorrectly declared dependency
|
| That literally _IS_ cache invalidation. Incremental build
| systems are caching previous artefacts. They 're supposed
| to invalidate the artefact if a dependency changes. But
| you forgot to declare the dependency... so it doesn't get
| invalidated when it should.
| gpderetta wrote:
| Namings things is of course a bit tongue in cheek. But cache
| invalidation is hard. For example, allegedly MESI is one of
| the hardest things to validate in processor design.
| TOGoS wrote:
| There is a secret technique, called content-addressing[1],
| which elegantly solves both of them at once.
|
| A lot of people haven't caught on, and try to cache things
| using ambiguous names, hence the struggle to invalidate their
| caches when the meaning changes.
|
| [1] This can be applied even if you don't know the content
| yet; you just have to unambiguously name the inputs to the
| function that produces it. You might not _know_ what all the
| inputs are, and then you have to start adding stuff like
| "unknown-unknown-2025-07-03T16", but it'll still basically
| work.
| ninalanyon wrote:
| Really? Have you tried building any substantial program that
| makes use of caching and succeeded in invalidating the cache
| both correctly and efficiently? It's not all about simple
| things like disk access, caching is also useful in software
| that models complex hardware where properties depend on
| multitudes of interconnected calculated values that are time
| consuming to calculate and where you cannot predict which
| ones the client will ask for next.
| EGreg wrote:
| Yes, I have. I've actually built a general-purpose cache
| system. Several, in fact:
|
| Here's one for PHP: https://github.com/Qbix/Platform/blob/m
| ain/platform/classes/...
|
| And here is for the Web: https://github.com/Qbix/Platform/b
| lob/dc95bd85fa386e45546b0b...
|
| I also discuss caching in the context of HTTP:
| https://community.qbix.com/t/caching-and-sessions/192
| WRITING ABOUT CACHING AND INVALIDATION
|
| You should especially start here:
| https://community.qbix.com/t/qbix-websites-loading-
| quickly/2...
|
| And then you can read about incremental updates:
| https://community.qbix.com/t/streams-plugin-messages-
| subscri...
| necovek wrote:
| Your implementations are not cache systems: they are key-
| value store abstractions which can be used for caching.
| There is a simpler one in most languages (IIRC, named
| "hashes" in PHP).
|
| Caching becomes hard when such a stores are used in
| distributed or concurrent contexts.
|
| An example: imagine Qcache being used when fetching data
| from an SQL database. And data in the database changes
| with a direct SQL query from another process.
|
| How will your key-value store know that the value in it
| is stale and needs refreshing?
| EGreg wrote:
| Yes, they are cache systems. And as you say, they can be
| used for caching.
|
| Caching systems know when something needs updating
| several ways. One is pubsub: When the information is
| loaded from the cache, you want to set up a realtime
| websocket or webhook for example, to reflect any changes
| since the cached info was shown. If you can manage to
| have a server running, you can simply receive new data
| (deltas) and update your cached data -- in that case you
| can even have it ready and never stale by the time it is
| shown.
|
| If you can't run a server and don't want to reveive
| pushes then another approach simply involves storing the
| latest ordinal or state for EACH cached item locally
| (e-tags does this) and then before rendering (or right
| after you do) bulk-sending the tags and seeing what has
| been updated, then pulling just that. The downside is
| that you may have a flash of old content if you show it
| optimistically.
|
| If you combine the two methods, you can easily get an up-
| to-date syndication of a remote mutable data store.
|
| My whole framework is built around such abstractions, to
| take care of them for people.
| necovek wrote:
| We can agree to disagree on this being a "caching system"
| and it "knowing when something needs updating" (from the
| API, I am guessing it's a "set" method).
|
| Phrases like "simply" and "never stale" are doing a lot
| of heavy lifting. Yet you haven't answered a very simple
| question I posted above: how would Q_Cache structure
| handle a direct SQL write query from another process on
| the database it is being used as a cache for?
|
| SQL databases don't run websockets to external servers,
| nor do they fire webhooks. If you are running a server
| which you are hitting with every update _instead_ of
| doing direct SQL on the DB, there 's always a small
| amount of time where a cache user can see stale data
| before this server manages to trigger the update of the
| cache.
| EGreg wrote:
| Simple. Use the principles I said above, for distributed
| and asynchronous systems.
|
| The SQL server should have a way to do pubsub. It can
| then notify your PHP webhook via HTTP, or run a script on
| the command line, when a row changes. It should have an
| interface to subscribe to these changes.
|
| If your SQL data store lacks the ability to notify others
| that a row changed, then there's your main problem. Add
| that functionality. Make a TRIGGER in SQL for instance
| and use an extension.
|
| If MySQL really lacks this ability then just put node.js
| middleware in front of it that will do this for you. And
| make sure that all mysql transactions from all clients go
| through that middleware. Now your combined MySQL+Node
| server is adequate to help clients avoid stale data.
|
| As I already said, if you for some reason refuse to
| handle push updates, then you have to store the latest
| row state (etags) in your PHP cache (apcu). And then when
| you access a bunch of cached items on a request, at the
| end collect all their ids and etags and simply bulk query
| node.js for any cached items that changed. You can use
| bloom filters or merkle trees or prolly trees to optimize
| the query.
|
| Joel Gustafson had a great article on this and now uses
| it in gossiplog:
| https://joelgustafson.com/posts/2023-05-04/merklizing-
| the-ke...
| logical42 wrote:
| Adding change data capture to a database isn't exactly
| trivial.
| EGreg wrote:
| Well then maybe consider making a sql replication client
| that would ingest the changes (like most databases, mysql
| writes to a straming append-only log, before it is
| compacted). Just parse the log and act on them.
|
| Not trivial, really? Here, enjoy:
| https://github.com/krowinski/php-mysql-replication
| tsimionescu wrote:
| You're just pushing much of the complexity to that
| notification layer.
|
| What if an SQL update changes a million rows, and you had
| cached the sum of those million rows? Should it send a
| million notifications? Should it re-run the sum for you?
| What if it's a complex operation and it takes a while to
| re-compute, and another update arrives before the re-
| compute finishes?
|
| And of course, you will always have some partitions, so
| you will occasionally need to re-query the data anyway
| and re-establish any kind of pubsub system. And the
| complexity in efficiently reconciling two views of a
| large amount of data is a major issue, that you are just
| brushing off as "you can do various things to maybe
| optimize the query".
| EGreg wrote:
| Yes, if you really want to show something as ambitious as
| a sum of a million rows, then you should absolutely move
| to an event-based, push-based architecture. Because once
| you've calculated the sum, the least work you would need
| to do is to periodically look at batches of UPDATES to
| rows, and update your sum based on that (rather than,
| say, running SUM again on the whole entire table). You
| can either pull a digest periodically, or -- you can
| handle incremental updates by listening for pushes.
|
| All the difficulties you cite, including the
| "optimizations" I mentioned, stem from not using push,
| and insisting on periodically polling something.
|
| If you step back, your whole problem is that you are
| using a system that only has pull and not push
| architecture. You're trying to put a bandaid on that core
| fact. You chose a relational database system that you
| refuse to build extensions for (as opposed to pretty much
| any other system) in order to insist "nyeh nyeh you
| havent solved cache invalidation".
|
| Caching is just one part of sync. The part that stores a
| non-authoritative, slightly out of date copy locally. And
| sync, or replication, is just one part of an eventually
| consistent system. I mean, even MySQL supports
| replication, so just hook your client up to that protocol
| (encrypted in transit) and boom, now you can update your
| cache.
|
| Here's the thing. If you use any network protocol that is
| open, eg HTTP, then yea it's solved. Because you can have
| webhooks and just spin up a server to handle them, and
| update your cache. A row is removed? Decrement your sum.
| A row is added? Increment it.
|
| You are just insisting that "no, our system has to be
| clunky and we refuse to implement webhooks / websockets /
| sockets / push / mysql replication client / updates of
| any kind, now solve it for me to be as good as a system
| which has push capabilities."
| EGreg wrote:
| I will answer your question directly, after having
| explained the core issue.
|
| Mysql and Postgresql does in fact have push. It is called
| replication. The protocol is documented. So, much like a
| webhook with HTTP protocol, you can ingest changes on a
| table with a php long-running client and then simply
| update your apcu cache entries if they are still being
| stored, and matching that row. And all your PHP processes
| (managed by php-fpm or FrankenPHP or whatever) will
| benefit from that atomic apcu update.
|
| There is nothing that says only another mysql server can
| be a mysql replication client.
|
| Boom, solved. Even in your case. Note that the SQL
| database did in fact expect you to open a socket, and
| then it writes its log to that socket. That's all the
| streaming events you need.
|
| But now you're gonna say "but but no you cant have a long
| running PHP client, we're talking about a shared hosting
| environment where the host disables your ability to run a
| long PHP service that opens a socket". Well, look. Most
| of the difficultu is that you're intentionally trying to
| use the most clunky architectures just to "prove" that
| "cache invalidation is really hard."
|
| Shall we continue with Naming next? :)
| necovek wrote:
| I wasn't going to say any of that ;) Yes, Postgres does
| have WAL log you can subscribe to, but you are creating a
| replicated slave really. There are many nuances to doing
| that right and you are downplaying how hard that is (it's
| not about API, it's about ACID).
|
| More importantly, you are pushing all the complex logic
| to this imaginary DB replication client: your "caching
| system" is still nothing more than a KV store.
| EGreg wrote:
| Well what are the "real" caching systems, if my system
| can be used for caching but isnt "real"? This feels like
| a no true scotsman argument.
| necovek wrote:
| Caching system would be what you described as a response
| to my example case: your key-value store _combined_
| together with the entire mechanism to watch for DB writes
| (eg with triggers or the replication log) and tie them
| back to cached values which need to have their lifetime
| managed.
| logical42 wrote:
| I agree with you, but you are omitting a key aspect of the
| problem which makes this hard. If you have a single server
| serving from a local sqlite database, caching and
| invalidating the cache is trivially easy.
|
| It becomes way more difficult when you have N servers all
| of which could potentially serve the same data. Local
| caches could then, yes easily become stale, and even if you
| have a propagation mechanism, you couldn't guarantee
| against network failures or latency issues.
|
| But suppose you have an HA redis instance that you store
| your cached data in. Even with a write-through cache, you
| basically need to implement a 2-phase commit to ensure
| consistency.
| EGreg wrote:
| See the thread below. Caches only become stale if you
| refuse to implement any sort of push architecture. And
| even then, frequent batch-polling with etags will quickly
| resolve it. (eg right after you used N entries, poll to
| see which of the N have changed).
|
| Bloom filters, Prolly trees etc can speed the batch
| polling up. But push architecture obviates the need for
| it prevents caches from being stale, it's called eventual
| consistency.
|
| ( _Well, of course unless there is a network partition
| and you can't communicate with the machine holding the
| actual source of truth then yeah, your local cache will
| be stale. And to catch up on missed updates you will need
| to request all updates since a certain marker you saved.
| If the upstream had a ton of updates since your marker
| and can't send the entire history since your marker then
| you might have to purge your cache for that specific
| table / collection, yeah._).
| bloppe wrote:
| "Two programs could have similar behaviour but structured very
| differently, the difference being that one utilizes caching as
| an abstraction and one explicitly has the concept of different
| tiers of storage."
|
| The author is comparing "off-the-shelf" caching with custom
| caching. They're coming from the assumption that you must be
| caching _somehow_ and arguing that the word "caching" should
| be understood to mean only particular approaches to the general
| idea of caching. And obviously the whole point of the general
| idea is to optimize things.
|
| It's a rhetorical mess
| heikkilevanto wrote:
| Caching is simple, yes. The hard part is in the last word,
| invalidation. Even that is manageable for a single process. But
| as soon as you have multiple (threads / processes / nodes /
| data centers) updating the data, it does get quite complex,
| pretty fast.
|
| Likewise, naming things is simple as long as you alone, or a in
| a small team. But as soon as there are multiple organizations
| with all their own traditions, it gets tricky. Just witness the
| eternal flame wars about camelCase, PascalCase, snake_case,
| kebab-case, and UPPER_CASE. It is almost as hopeless culture
| clash as Emacs vs Vi vs PowerPoint...
|
| (I leave the off-by-one errors as an exercise for the reader)
| TeMPOraL wrote:
| I'd say this is not the "naming things" that's hard. Beyond
| picking a common identifier format in the team, there are at
| least two dimensions that are _much harder_ :
|
| - The language dimension - choice of words, that are good
| enough for the purpose, and not confusing. For example,
| "Manager" is as ambiguous as it gets, it can mean many thing,
| except we've been using it long enough that there's a more
| specific shape of meaning[0] for that word in code/program
| architecture contexts - so you still would use it instead of,
| say "Coordinator", which would raise all kinds of questions
| that "Manager" no longer does.
|
| - The epistemological dimension - whether the word you chose
| correctly names the concept you meant, and whether the
| concept you meant is actually the right one to describe the
| thing you're trying to describe. Ultimately, this is _the_
| hard thing at the root of philosophy. In practice, it
| manifests like e.g. choice between digging into some obscure
| branches of mathematics to correctly name the thing
| "endofunctor" or something, or calling it "Square" and saying
| "fuck it, we'll clarify the exceptions in the comments".
|
| --
|
| [0] - I mean "more specific" in the sense it's distinct from
| the other meanings and somewhat narrow - but still it's fuzzy
| as heck and you can't describe it fully in words; it's
| basically tacit knowledge.
| Xss3 wrote:
| I try to name things descriptively in simple terms and
| often end up with NamesAboutThisLong, once they get too
| long i know the thing is doing too much and some
| refactoring is needed for readability.
|
| I also avoid letting the reader make assumptions.
| HasPlayerJumpedRecently is bad. What does recently mean?
| HasPlayerJumpedInLastTenMs is better, even if it's a bit
| long...Which highlights that it should probably be
| refactored into a more flexible value;
| MsSincePlayerLastJumped.
|
| If you arent assuming a time var wth Ms is milliseconds you
| aren't doing games dev so that one slides with me.
| dmkolobov wrote:
| Wow cool, you just summed up something I've found myself
| doing subconsciously in the past few years. Thanks!
|
| I use to be quite fond of short identifiers, especially
| ones the make the signs "line up"... until I worked with
| code long enough that I forgot what I did and had to read
| it again.
| TeMPOraL wrote:
| That's a great example. Personally, even beyond gamedev,
| units are an exception for me - they _need_ to be
| somewhere, whether in type or in identifier name (or
| ideally both), or else bad things happen.
|
| But that's what I meant by the epistemological aspect -
| what is "recently"? Still, "LastTenMs" is arguably even
| worse - _why ten_? Did you really mean "since last
| frame"? Then say "SinceLastFrame". Did you mean "since 2x
| update time delta"? Then maybe make a PlayerJumpCooldown
| constant, but then maybe it's not about cooldowns, so...
| here's when I'd probably say screw this, let's just track
| MsSincePlayerLastJumped and add a
| HasPlayerJumpedRecently() -> bool as a helper, with a fat
| interface comment explaining what "recently" means - and
| go back to talk with the game designers to give a more
| specific name for the "recently" thing here.
|
| Point being, this is deceptively hard - doubly so because
| going all perfectionist about it is _also bad_.
| Xss3 wrote:
| I admit my example is slightly contrived and LastTen
| still raises questions re frames and time handling,
| especially as its so small, oops! Lets pretend i said
| LastHundred and its using a framerate independent method.
|
| If i were to helperise it with a bool I'd use the intent
| for the bool, CanPlayerJumpAgain or something
| (again...contrived example).
| gblargg wrote:
| I figured the naming issue is deciding how much context. A
| name might begin inside an organization but need to endure a
| wider area. If you make all names so long and context-free
| that they can work in any context, they become unwieldy. Also
| it can be hard to realize some of the implicit context and
| what needs to be differentiated with the name. Where server
| used to suffice, now you need server-a and server-b.
| yashasolutions wrote:
| Don't bring a PowerPoint to a Vi/Emacs fight...
| Pet_Ant wrote:
| Even caching is not simple as the resources get consumed and
| you need an eviction policy. Like in a Maven cache, keep no
| more than one old version around of library to allow for
| upgrade windows.
| Traubenfuchs wrote:
| I never understood this meme.
|
| We use caching a lot, anything that gets cached can only be
| written by one service each. The writing services emit cache
| invalidation messages via SNS that cache users must listen to
| via SQS, to clear/update their cache.
|
| Alternatively we cache stuff with just a TTL, when immediate
| cache invalidation is not important.
|
| Where's the struggle?
| porridgeraisin wrote:
| Here's one: everybody invalidating and refreshing their cache
| at the same time can cause a thundering herd problem.
| hmottestad wrote:
| Does SQS guarantee delivery to all clients? If it does then
| that's doing a lot of heavy lifting for you.
|
| If it doesn't guarantee delivery, then I believe you will at
| some point have a client that reads a cached value thinking
| it's still valid because the invalidation message got lost in
| the network.
| maccard wrote:
| Eventually. The problem is that eventually delivering that
| message will result in clients assuming that it will always
| be the same, when it's not.
| williamdclt wrote:
| You don't support read-your-own-write and your cache data
| might be stale for arbitrarily long. These relaxed
| consistency constraints make caching a lot easier. If that's
| acceptable to your use cases then you're in a great place! If
| not... well, at scale you often need to find a way for it to
| be acceptable anyway
| pton_xd wrote:
| > Where's the struggle?
|
| If there are no real consequences for reading stale data, and
| your writes are infrequent enough, then indeed you're lucky
| and have a relatively simple problem.
| motorest wrote:
| > I never understood this meme.
|
| If you don't understand how and why and when eventual
| consistency is a problem, you will never understand why cache
| invalidation is hard.
|
| By the sound of your example, you only handle scenarios where
| naive approaches to cache invalidation serve your needs, and
| you don't even have to deal with problems caused by spikes to
| origin servers. That's perfectly fine.
|
| Others do. They understand the meme. You can too if you
| invest a fee minutes reading up on the topic.
| graealex wrote:
| That's because relying on a TTL simplifies the concept of
| caching, and makes invalidation trivial, and also inflexible.
|
| It's used in DNS, which already was an example here. There is
| no way to be sure clients see an updated value before end of
| TTL. As a result, you have to use very conservative TTLs.
| It's very inefficient.
| ahoka wrote:
| You can't be sure even after the TTL to be fair.
| Cthulhu_ wrote:
| > Where's the struggle?
|
| > anything that gets cached can only be written by one
| service each
|
| How do you guarantee it's only written by one service each?
| Sounds like locking across network boundaries, which is not
| easy.
|
| > The writing services emit cache invalidation messages via
| SNS that cache users must listen to via SQS
|
| SNS and SQS are both nontrivial services (at least you don't
| have to build / maintain them I suppose) that require
| training to use effectively and avoid any possible footguns
|
| I think you're underestimating the complexity in your own
| solution, and you're probably lucky that some of the harder
| problems have already been solved for you.
| tengbretson wrote:
| I've never really understood it either. In my experience, in
| order for a cache to be a possible solution to a given
| problem at all, you must either:
|
| 1. Be content with/resilient to the possibility of stale
| data.
|
| 2. Gatekeep all reads and writes (for some subset of the key
| space) through a single thread.
|
| That's basically it.
| hatthew wrote:
| If you have a system with "slow storage", caching is a way to
| optimize that to "storage that is sometimes fast".
|
| If you have a system with "slow storage" and "fast storage",
| caching is a way to abstract that away to just "storage".
|
| The author is arguing that the latter is the default way we
| should think about the concept of caching, which is a valid
| opinion to have.
| AdieuToLogic wrote:
| > There's that famous quote "There are only two hard things in
| Computer Science: cache invalidation and naming things.", and,
| sure, it's a bit ironical, but there's some truth in there.
|
| The joke form of this quote goes along the lines of:
| There are only two hard things in Computer Science: cache
| invalidation, naming things, and off-by-one errors.
|
| :-D
| dcminter wrote:
| I rather like the snark of:
|
| _there 's two hard problems in computer science: we only
| have one joke and it's not funny._
|
| Apparently0 by Philip Scott Bowden1
|
| 0 https://martinfowler.com/bliki/TwoHardThings.html
|
| 1 https://x.com/pbowden/status/468855097879830528
| aorth wrote:
| Just remembered another one: there are 10 types of people
| in the world: those who understand binary and those who
| don't. :)
| AndrewOMartin wrote:
| Which leads to
|
| > I don't see what's so hard about DNS, it's just cache
| invalidation and naming things.
| manyaoman wrote:
| ... and avoiding off-by-one errors.
| dcminter wrote:
| Oh that's gooood. Got a cite or is it yours?
| SAI_Peregrinus wrote:
| My favorite variation only really works in text:
|
| There are three hard problems in Computer Science:
|
| 1) Cache invalidation
|
| 2) Naming th3) Concurings
|
| rency
|
| 4) Off-by-one errors
| Joker_vD wrote:
| There is also an important (but often overlooked) detail that
| you/your application may not be the only user of the cache. At
| which point caching, indeed, is an optimization via abstraction:
| when you fetch an X, you are in no position to predict that the
| next fifty completely unrelated to you requests would also want
| to fetch the same X, so it should probably be cached to be
| readily served.
|
| Which is why solving the "I want my data in fast storage as often
| as possible" problem may be counter-productive on the whole: you
| ain't the only client of the system; let it breath and server
| requests from others.
| eigenform wrote:
| Even more obvious if you think about the case of hardware-managed
| caches! The ISA typically exposes some simple cache control
| instructions (and I guess non-temporal loads/stores?), but apart
| from that, the actual choice of storage location is abstracted
| away from you (and your compiler).
| necovek wrote:
| On top of the other things mentioned (caching always introduces
| complexity with lifetime tracking, and thus can't make things
| simple), the article's got it the wrong way around.
|
| When code has abstract interfaces for data access, introducing
| caching can be simpler (but not simple) by localizing it in the
| abstraction implementation which has or doesn't have caching.
|
| But it is not an _abstraction_ (you can perfectly well do caching
| without any abstractions, and it 's frequently done exactly that
| way).
| movpasd wrote:
| I think you and the article are referring to abstractions over
| different concerns.
|
| The concern you're talking about is about the actual access to
| the data. My understanding of the article is that it's about
| how caching algorithms can abstract the concern of minimising
| retrieval cost.
|
| So in some ways you're coming at it from opposite directions.
| You're talking about a prior of "disk by default" and saying
| that a good abstraction lets you insert cache layers above
| that, whereas for the author the base case is "manually
| managing the layers of storage".
| foldU wrote:
| This is correct, I appreciate you for putting it so
| coherently :). I think I didn't make it clear enough in the
| piece that I'm coming from a stance of fast access being
| table stakes, and the question being about how that's
| accomplished.
| necovek wrote:
| "Caching" is an _idea_ of storing a result of an expensive
| computation in storage that is faster to get from than
| doing the original computation (in very generic computer
| terms, computation can be simply fetching from the network
| or slower local storage).
|
| What you describe as "caching algorithms" are not really
| caching algorithms, but cached object lifetime management
| algorithms (LRU, LFU...).
|
| "Abstraction" is a higher level, simplified view of a set
| of concepts, yet caching is a single concept. See eg. https
| ://en.wikipedia.org/wiki/Abstraction_(computer_science)
|
| It sounds like you are both trying to redefine what
| "caching" means (tying it to _implementations_ of
| particular algorithms), but also what "abstraction" means.
|
| We should be very deliberate with the language we use, and
| our main goal should be to make it simpler to understand,
| not harder -- I believe you are doing the latter here.
| necovek wrote:
| The language used is seriously confusing here.
|
| Algorithms can't really abstract anything since they are,
| well, just algorithms (formal descriptions of how a
| computation should be done).
|
| Looking at the author's examples again, I think most
| everybody would say that caching is used in both:
| if data_id in fast_storage: return
| fast_storage.get(data_id) else: data =
| slow_storage.get(data_id) fast_storage.set(data_id,
| data) return data
|
| and # Uses fast storage or slow storage just
| like above, but behind the get() method. return
| storage.get(data_id)
|
| The first one does not make an abstraction on storage, the
| second one does, but they are both "caching" data internally.
|
| While there are _generic_ _implementations_ of caching
| algorithms and we can consider those abstractions, "caching"
| is a wider term than those implementations, and is
| specifically not an abstraction (the fact that there is a
| caching implementation that abstracts something does not make
| all caching an abstraction).
|
| Edit: Let me also point out that "abstract the concern of
| minimising retrieval cost" is not caching -- I can say that
| eg. a simple interface with method FastGet(id) does the
| former, and it needs not use any caching if the underlying
| structure is fast enough and eg. directly in memory.
| gmuslera wrote:
| "fast storage" is about performance, your abstraction includes
| performance elements. If you go that down, then you are
| optimizing on your abstraction designs. What doesn't have to be
| wrong, but then don't say that is not optimization.
| LudwigNagasena wrote:
| Caching is an optimisation. Sometimes caching can be abstracted
| away, eg CPU cache or build cache are pretty much abstracted away
| for a usual web developer. But web page caching is very hard to
| abstract without any abstraction leaks and weird bugs. And even
| CPU cache is no longer an abstraction if you deal with very high
| performance code.
| gblargg wrote:
| It sounds like they are arguing that when performance matters,
| you have to know more about caching. Fair enough, you have to
| know a lot more about things when optimizing. For a lot of
| cases you can ignore caching because it can be done
| transparently. You depend on it to some extent because if e.g.
| every instruction had to be fetched off rotating storage like
| the old days, it would play a big role in your design. It's
| just something solved in general for most software to not have
| to know much about it.
| k__ wrote:
| Anything can be an abstraction if designed carefully.
| jbverschoor wrote:
| Everything is caching. Almost nothing operates on the target data
| directly.
| necovek wrote:
| Do you think that's a useful definition of the term?
|
| If everything is caching, why even introduce the term: language
| should help us describe ideas, it should not be superfluous.
| jbverschoor wrote:
| Because you _can_ operate directly on data
| canyp wrote:
| Did you hand-draw that graffiti? Never quite realized that
| graffiti of technical ideas looks really goated. Best part of the
| post, to be honest.
| jxjnskkzxxhx wrote:
| > looks really goated
|
| Oof you're trying so hard you could cut diamond with that line.
| canyp wrote:
| I don't even understand what that means. Care to explain?
| the__alchemist wrote:
| I think it's a drug reference!?
| timewizard wrote:
| > I've always been told that caching is a tool to make software
| faster.
|
| Who told you that?
|
| > you don't have to go all the way back to some backend database
| or API server or SSD [...] Caching is thus a tool to improve
| performance.
|
| That's called "latency." This is not at all the same as
| "performance."
|
| > My feelings now are that that perspective on caching is wrong
|
| I agree.
| taeric wrote:
| This reminds me of the use of materialized views as both a cache
| strategy and as an abstraction helper.
| bravesoul2 wrote:
| And they too can _slow things down_. Like all caches can. Like
| Redis can. Cache is a leaky abstraction.
|
| (Although a materialised view is more like an index than a
| cache. The view won't expire requiring you to rebuild.)
| necovek wrote:
| I believe this same language use is what makes this article
| confusing: Redis is not a cache, it is a key value store.
| Caching is usually implemented using key value stores, but it
| is not an abstraction (leaky or not).
|
| In RDBMS contexts, index really is a caching mechanism (a
| cache) managed by the database system (query planner needs to
| decide when it's best to use one index or another).
|
| But as you note yourself even in these cases where you've got
| cache management bundled with the database, having too many
| can slow down (even deadlock) writes so much as the database
| tries to ensure consistency between these redundant data
| storage elements.
| bravesoul2 wrote:
| I thought Redis grew up as a KV cache and persistent
| storage came later.
|
| In some sense though. If it ain't L1 it's storage :)
| necovek wrote:
| Maybe Redis started up as an in-memory KV store focused
| on caching use cases, but it was still a KV store that
| could be used for caching, or not.
|
| Even if you use "cache" in the name (eg. memcached),
| that's still not a cache, even if it's a KV store
| designed for caching.
| neuroelectron wrote:
| This is basically semantic argument, and I will not be engaging
| in it
| jxjnskkzxxhx wrote:
| You're right, but caching is an optimization.
| pclmulqdq wrote:
| Use of a better abstraction is an optimization, though.
| jongjong wrote:
| I was discussing this with someone recently, caching is one of
| those things that people might do behind the scenes, thinking
| that it doesn't affect the API but in fact it can create all
| sorts of issues/complexity.
| zmj wrote:
| This article is talking about single-writer, single-reader
| storage. I think it's correct in that context. Most of the hairy
| problems with caches don't come up until you're multi-writer,
| multi-reader.
| TristanDaCunha wrote:
| This whole discussion on caching and abstraction was completely
| befuddling to me.
| klabb3 wrote:
| Note: the author means that caching can be used as an
| implementation detail in an (abstracted) storage access system,
| as opposed to a baseline of having multiple storage systems
| (fast, medium, slow) and managing them directly.
|
| This was confusing to me - the most obvious way to judge the
| purpose of a system is to compare with the baseline of _not_
| having that system at all, especially in the case of caching
| where the program is functionally complete and correct without a
| cache. Anyway, there may not be a right or wrong here. Just
| tripped me up.
| yetanotherjosh wrote:
| Yes "good" caching - a consistent storage interface - is an
| abstraction over "bad" caching - multiple different storage
| interfaces with different speeds. But caching overall is not an
| abstraction over not having caching.
| armchairhacker wrote:
| Most optimizations require you to think about how your code is
| structured, so as a side-effect you make the code more
| understandable.
|
| In this article, it's cache levels forcing you to separate
| different types of data because they're accessed at different
| frequencies. Another example is Rust's borrow checker, whose main
| purpose is arguably to facilitate both safe and efficient memory
| management, but which can also be used to enforce invariants that
| aren't clearly memory-related (e.g. builder pattern, temp files
| that auto-delete after they're dropped).
|
| These aren't abstractions though. An abstraction is the opposite,
| hiding structure when it's noisy and making it easier to change.
| For example, if you already have an architecture in mind and
| don't want to manually determine how frequently each type of data
| is accessed, it's better to use a compiler or library that
| automatically determines what to cache with little to no code or
| thought on your end; that's abstraction. Similarly, the abstract
| analogue to Rust's borrow checker is garbage collection, which
| allows programmers to not think about their data-structures'
| lifetimes at all. The cost is usually performance and you
| understand your application less in some ways (although you
| understand it more in other ways; abstraction hides details but
| too many details make it hard to see the big picture. Ideally,
| with abstractions in the right places, you hide only the
| "unimportant" details in ways that insignificantly affect
| performance).
| charleshn wrote:
| As can be seen from other comments, people tend to focus on the
| consistency implications, but something not discussed often in
| the context of distributed systems is that caches tend to
| introduce bimodality and metastability [0] [1]. See e.g. DynamoDB
| for an example of design taking it into account [2].
|
| [0] https://brooker.co.za/blog/2021/08/27/caches.html
|
| [1]
| https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s...
|
| [2] https://brooker.co.za/blog/2022/07/12/dynamodb.html
| dasil003 wrote:
| What? No, caching means a specific thing: keeping a copy of data
| away from the source of truth, closer to where you want to read
| it. Caching always makes systems more complex, it never makes
| things simpler, and it damn sure doesn't serve as any kind of
| abstraction unless you're redefining what words mean to indulge
| your technical philosophizing.
| hansvm wrote:
| What if you have to keep some data closer and away from the
| source of truth though? Given that constraint, TFA argued that
| other architectures could do the job but that caching functions
| as an abstraction.
| 0xbadcafebee wrote:
| Sometimes posts are so difficult to read they're hard to respond
| to. I _think_ I get what they 're saying. I think they're saying
| that they think caching _should_ be simple, or at least, that it
| should be obvious how you should cache in your particular
| situation such that you don 't need things like algorithms. But
| that argument is kind of nonsense, because really everything in
| software is an algorithm.
|
| Caching is storing a copy of data in a place or way that it is
| faster to retrieve than it would be otherwise. Caching is not an
| abstraction; it is a computer science technique to achieve
| improved performance.
|
| Caching does not make software simpler. In fact, it always, by
| necessity, makes software more complex. For example, there are:
| - Routines to look up data in a fast storage medium -
| Routines to retrieve data from a slow storage medium and store
| them in a fast storage medium - Routines to remove the
| cache if an expiration is reached - Routines to remove
| cache entries if we run out of cache storage - Routines to
| remove the oldest unused cache entry - Routines to remove
| the newest cache entry - Routines to store the age of each
| cache entry access - Routines to remove cache entries which
| have been used the least - Routines to remove specific
| cache entries regardless of age - Routines to store data in
| the cache at the same time as slow storage - Routines to
| store data in cache and only write to slow storage occasionally
| - Routines to clear out the data and get it again on-demand/as
| necessary - Routines to inform other systems about the
| state of your cache - ...and many, many more
|
| Each routine involves a calculation that determines whether the
| cache will be beneficial. A hit or miss can lead to operations
| which may add or remove latency, may or may not run into
| consistency problems, may or may not require remediation. The
| cache may need to be warmed up, or it may be fine starting cold.
| Clearing the cache (ex. restarts) may cause such a drastic
| cascading failure that the system cannot be started again. And
| there is often a large amount of statistics and analysis needed
| to optimize a caching strategy.
|
| These are just a few of the considerations of caching. Caching is
| famously one of the hardest problems in computer science. How
| caching is implemented, and what it affects, can be very complex,
| and needs to be considered carefully. If you try to abstract it
| away, it usually leads to problems. Though if you don't try to
| abstract it away, it also leads to problems. Because of all of
| that, abstracting caching away into "general storage engine" is
| simply impossible in many cases.
|
| Caching also isn't just having data in fast storage. Caching is
| cheating. You want to provide your data faster than actually
| works with your normal data storage (or transfer mechanism, etc).
| So you cheat, by copying it somewhere faster. And you cheat
| again, by trying to figure out how to look it up fast. And cheat
| again, by trying to figure out how to deal with its state being
| ultimately separate from the state of the "real" data in storage.
|
| Basically caching is us trying to be really clever and work
| around our inherent limitations. But often we're not as smart as
| we think we are, and our clever cheat can bite us. So my advice
| is to design your system to work well _without_ caching. You will
| thank yourself later, when you finally are dealing with the bug
| bites, and realize you dodged a bullet before.
| scrubs wrote:
| Ousterhout's grad students did work on ramcloud with some
| research at facebook and Amazon on cache use at scale in complex
| organizations.
|
| One bit of interesting trivia say for facebook (from memory): if
| you add all the RAM caches in redis/memcached/disk + db caches to
| make the thing work at scale, then for about 20-30% more memory
| you could've had the whole thing in memory 100% of the time.
| chrisjj wrote:
| The problem there is /the whole thing/ grows - often faster
| than your memory.
| scrubs wrote:
| I think you need to read the papers.
|
| Obviously things grow - look who i mentioned afterall.
|
| The focus is on how you handle growth.
| kazinator wrote:
| Optimization isn't separable from abstraction. Abstraction is
| something that can be implemented in more than one way, while
| meeting the terms of its contract. That flexibility allows for
| optimization.
| chrisjj wrote:
| Why not both? :)
| suspended_state wrote:
| Let's first get the obvious out of the way: caching is not an
| abstraction, the "Storage" abstraction is what enables caching to
| be implemented. If I had to put Caching in a category, I would
| say that it's an optimization strategy.
|
| But that's not really what the blogpost is about. The issue that
| it tries to discuss is the fact that this abstraction is often
| imposed to us, without any way to control its behaviour. That's
| the examples of the LOAD_NAME in python he points at. Without
| having a clear understanding of the access patterns the
| application mostly uses, a caching strategy cannot be well
| defined, and you'll end up with an inadequate solution.
| kiitos wrote:
| Cmd+F "invalidation" -- not found.
|
| Author is talking about the least interesting, and easiest, piece
| of the overall caching problem.
___________________________________________________________________
(page generated 2025-07-04 23:02 UTC)