[HN Gopher] A major Postgres upgrade with zero downtime
___________________________________________________________________
A major Postgres upgrade with zero downtime
Author : stopachka
Score : 65 points
Date : 2025-01-29 16:57 UTC (6 hours ago)
(HTM) web link (www.instantdb.com)
(TXT) w3m dump (www.instantdb.com)
| xyzzy_plugh wrote:
| The title is pretty misleading. They're not even running
| Postgres, but AWS Aurora, which is Postgres compatible, but is
| not Postgres.
|
| Also, pausing queries does count as downtime. The system was
| unavailable for that period of time.
| stopachka wrote:
| > The title is pretty misleading. They're not even running
| Postgres, but AWS Aurora, which is Postgres compatible, but is
| not Postgres.
|
| For what it's worth, every command ran works on normal
| Postgres. Hence we didn't think it mattered to mention Aurora
| specifically in the title.
|
| > Also, pausing queries does count as downtime.
|
| If a query takes a bit longer to respond, I don't think that
| counts as downtime. From the perspective of the user, they
| couldn't distinguish this migration event from some blip of
| slightly slower queries.
| lionkor wrote:
| It depends if it feels like an outage
| awesome_dude wrote:
| > If a query takes a bit longer to respond, I don't think
| that counts as downtime
|
| "We're sorry that your query took 7 hours to be responded to,
| but it wasn't an outage - honest"
| stopachka wrote:
| We would count 7 hours as downtime too. Our pause was less
| than 5 seconds.
| ElijahLynn wrote:
| Less than 5 seconds seems pretty reasonable to me to call
| it zero down time.
| tossandthrow wrote:
| 5 seconds pause on queries would make our app server drop
| connections and throw errors under cyclical high load -
| which would result in a incident.
| libraryofbabel wrote:
| Nice job, then! Technical downtime that's virtually
| undetectable to users is a big win. In fact, "less than 5
| seconds of downtime" in the title would actually make me
| want to read the article more as I tend to be suspicious
| of "zero downtime" claims for database upgrades, whereas
| <5s is clearly almost as good as zero and actually
| quantified :)
| awesome_dude wrote:
| Yeah - a quantifiable amount in the headline would change
| the likelihood of the article being taken seriously - it
| goes from "No downtime? I call BS" to "Less than 5
| seconds, that seems reasonable, and worth investigating"
| RadiozRadioz wrote:
| Completely depends on what the "user" is. Are they a human,
| or a machine that explicitly requires timings within a
| particular threshold?
| scottlamb wrote:
| > If a query takes a bit longer to respond, I don't think
| that counts as downtime. From the perspective of the user,
| they couldn't distinguish this migration event from some blip
| of slightly slower queries.
|
| It comes down to defining Service Level Objectives (SLOs)
| that are meaningful to your users. For one system I worked
| on, latency was important, and so one SLO was "99.999% of <a
| certain class of> requests with a deadline >=1s should
| succeed with latency <1s", so if this affected more than
| 0.0001% of requests in <time interval defined in our SLO>,
| we'd have called it an outage. But I've also worked on
| systems with looser SLOs where this would have been fine.
| nijave wrote:
| Not only that but I think you also need to take upstream
| systems into account. With a reasonably robust frontend
| that handles transient issues and retries reasonably, I
| think it's ok to say "no downtime"
| paulddraper wrote:
| > They're not even running Postgres, but AWS Aurora
|
| But everything described is also PostgreSQL compatible.
|
| > downtime
|
| Context switching pauses execution too FYI.
| unethical_ban wrote:
| They reduced their potential downtime from 60s to what I assume
| is only a few seconds (they don't state in the article).
|
| If there is not noticeable user impact or unavailability of
| services (this is unique to each service in existence) then
| there is no downtime.
| stopachka wrote:
| > they don't state in the article
|
| Thank you for pointing this out. I updated the essay to
| mention how long the pause took explicitly:
|
| After about a 3.5 second pause [^13], the failover function
| completed smoothly! We had a new Postgres instance serving
| requests, and best of all, nobody noticed.
|
| [^13]: About 2.5 seconds to let active queries complete, and
| about 1 second for the replica to catch up
| SahAssar wrote:
| AWS Aurora Postgres is a forked Postgres with a different
| storage engine. Sure you are technically correct, but there are
| many things called "Postgres compatible" that are very much
| less Postgres that AWS Aurora Postgres (like for example
| CockroachDB).
| nijave wrote:
| Iirc AWS explicitly calls out they still use upstream
| Postgres query engine and some other parts. It very much _is_
| Postgres but not 100% pure upstream Postgres.
| SahAssar wrote:
| Yep, for example that is how they advertise protocol,
| feature and language compatibility.
| darth_avocado wrote:
| Pause all writes > let 16 to catch up > resume writes on 16
|
| Isn't that..... downtime? Unless you mean downtime to be only
| when reads are also not available.
| stopachka wrote:
| We count downtime if a service is unavailable and drops
| requests. In this case, since the pause took about 3.5 seconds,
| we were able to service all requests.
| upghost wrote:
| I can't believe they took the time to make such an amazing write-
| up. With formatting and everything. Normally I just grab whatever
| broken crayon is closest and scribble on the back of an overdue
| bill "don't fsck up next time"
| honestSysAdmin wrote:
| Zero-downtime Postgres upgrades have been kind of normalized, at
| least in the environments I have been exposed to, with pgcat
| https://github.com/postgresml/pgcat
| stopachka wrote:
| Is there some resource that explains how to do a major version
| upgrade with pgcat? Would love to take a look
| honestSysAdmin wrote:
| As far as I know, there is not. I could probably write
| something up.
| jedberg wrote:
| This is impressive! I know others are questioning the "no
| downtime" bit, but that is why service level objectives exist --
| because it really depends on the customer experience.
|
| If you managed to have a cutover with no noticeable dip in
| business metrics (aka the users didn't notice) then I'd call that
| a no-downtime upgrade!
|
| Very clever on the improvement over Lyft's methods. Thanks for
| the writeup. Now maybe someone can get it down from 3 seconds of
| pausing. :)
___________________________________________________________________
(page generated 2025-01-29 23:00 UTC)