[HN Gopher] Self-hosting a high-availability Postgres cluster on...
___________________________________________________________________
Self-hosting a high-availability Postgres cluster on Kubernetes
Author : siamese_puff
Score : 80 points
Date : 2024-01-02 16:40 UTC (6 hours ago)
(HTM) web link (ryan-schachte.com)
(TXT) w3m dump (ryan-schachte.com)
| andrewmunsell wrote:
| My holiday project was doing another pass at my Homelab
| Kubernetes cluster, part of which involved switching to a proper
| operator to manage Postgres. Coincidentally, I setup cloudnative-
| pg (https://github.com/cloudnative-pg/cloudnative-pg) yesterday.
| x86hacker1010 wrote:
| Any reason you landed on that Operator compared to what OP is
| using (Zalando)?
| andrewmunsell wrote:
| Honestly no, it's mostly due to inexperience with operators
| and not really understanding what the "best" way to find
| operators is. I did also look at the Crunch Data one (I was
| having some issues setting that one up), but didn't even find
| Zalando during my search.
|
| OperatorHub is currently the main resource I use, but GitHub
| stars aren't exposed in the search so I have been looking at
| the "Capability Level" chart and checking for Github
| popularity when I find one with the feature support I want.
|
| I'm facing this exact same issue now when trying to find an
| operator for Redis. I am not sure if I am just missing out on
| the "right" option by limiting myself to Googling and
| Operator Hub and looking for the one with the most Github
| stars, so I am open to tips.
| turtles3 wrote:
| A subtle advantage of cnpg is that it doesn't use
| statefulsets, instead the operator handles things like
| mapping storage volumes and stable identities. Regular
| kubernetes statefulsets have some tricky sharp edges for
| failure recovery.
|
| I don't know if all of these alternatives use statefulsets
| but I remember several doing so.
|
| I've personally found cnpg to be pretty robust, and
| supports everything you will eventually need once you're
| locked into a solution (eg. Robust backups, CDC, replica
| clusters).
|
| I'm yet to find anything of a similar standard for mysql.
|
| EDIT: it should also be noted that CrunchyData is a
| proprietary solution and requires a license to use in
| production. This is not particularly obvious from their
| docs.
| activescott wrote:
| What sharp edges are you referring to with statefulsets?
| turtles3 wrote:
| Cnpg's docs articulate this better than I could:
| https://cloudnative-pg.io/documentation/1.16/controller/
|
| Statefulsets have their place but are surprisingly
| inconvenient for database workloads.
| Szpadel wrote:
| I was setting fairly important database with Zalando pg
| operator and after first good impressions it went downhill.
| after like a month of use WAL files used for point in time
| recovery started failing to offload to dedicated nodes and
| kept growing on database pods filling up all the space. I
| firstly assumed that maybe there is not enough space for some
| scheduled work (I do not really know details how this process
| work, I assumed that operator should handle all
| implementation details for me) but even after upscaling
| database 2.5x it just kept failing with full storage and
| requiring manual recovery to bigger storage, where most of it
| was WAL files.
|
| HA didn't handled this case at all whole cluster went in
| crash loop
|
| there was also issue of huge pages caused crashing and not
| easy way to disable those without some dirty injecting of
| config files at runtime
|
| there could be some my fault at misconfiguration on by side,
| but I wasn't able to figure anything better from docs
| ahachete wrote:
| I'm the founder of OnGres [1] the company behind StackGres [2].
| I'd love to hear your feedback if you'd be interested in also
| trying StackGres. It's one of the most feature-full operators
| available, has a complete Web Console and REST API and supports
| close to 200 extensions.
|
| Hope it would be interesting for you.
|
| [1]: https://ongres.com [2]: https://stackgres.io
| bo0tzz wrote:
| I've been using CNPG on my home cluster since it came out, and
| it's been an absolute pleasure to use. I haven't done a full
| comparison, but I get the sense that it's learned from (and
| improved on) the other postgres operators like Zalando and
| Crunchy.
| gchamonlive wrote:
| Just like with cloud providers, it seems to me like with
| kubernetes, it is not a matter of if but when orchestration
| problems will arise. Specially in this case of hosting databases,
| a composition of provisioning complexities (db operational
| complexities on top of k8s operational complexities) is really
| scary.
|
| Is there any way to overcome hidden complexity biting your hand
| other than studying k8s extensively?
| renegade-otter wrote:
| This kind of resilience is a form of art, and it's also kind of
| a full-time job.
|
| I would not advise trying this for a "side project at work".
|
| Generally we all agree that we move to the cloud and it's
| "fully managed". If it goes down - that's the price we pay.
|
| If you have to ask the question "what if the RDS goes down",
| then you are really in a different universe.
|
| That last guarantee of uptime requires a ton of work, testing,
| and money, because you are all the way up and to the right on
| the curve of diminishing returns.
| williamdclt wrote:
| > If you have to ask the question "what if the RDS goes
| down", then you are really in a different universe.
|
| It does go down though, don't neglect the possibility because
| it likely will happen. With very average workloads, I've seen
| RDS databases restart unexpectedly, read replicas being
| completely out of service, and even databases being
| completely frozen (can't even connect as root).
|
| I'd still go with managed, but it certainly doesn't give full
| reliability :) you still have to consider "what if it goes
| down" - it will!
| debarshri wrote:
| Problem is that RDS comes at a price. It is purely about
| operation cost.
|
| When you have 1500+ databases these cost add up. At that
| point, this kind of techniques are required to self host
| the databases. Price per DB with HPA na VPA is way lower
| than what you would pay for managed databases as well as
| you can hire a full time devops+dbadmin and still be
| cheaper.
| SlightlyLeftPad wrote:
| This is a perfect example of a cost savings opportunity
| that offers no benefit and is really just a foot gun when
| it comes to operational complexity. The cloud cost
| savings almost never justify the operational costs here.
| There's typically a very long list of other cost
| optimizations to be made before this would ever be on the
| table for me. If you're talking about 1500+ databases and
| think a single full time devops+dbadmin in some kind of
| unicorn person is going to be a better option, I feel bad
| for both of you.
| candiddevmike wrote:
| > That last guarantee of uptime requires a ton of work,
| testing, and money, because you are all the way up and to the
| right on the curve of diminishing returns.
|
| Not when it's a core part of your business...?
| figassis wrote:
| My company also runs on a k8s cluster, but I agree with this
| fully and I keep my DB outside the cluster. For me, the order
| of preference is: RDS, Cloud Instance, Bare metal DB server,
| something else that is not kubernetes.
|
| RDS will fail, but it likely will come back up without much
| action on my part. K8s will fail, you will spend untold human
| hours figuring out the k8s failure modes, before figuring out
| the database failure modes (which likely are quite straight
| forward). It's just a cost that is not worth it.
| szszrk wrote:
| Not really. Those are two areas you need to really understand
| when things go south.
|
| In Kubernetes you "solve" a lot of application complexity by
| abstracting it into things like Helm Charts or Operators. It's
| fun and easy to use operators to deploy databases, monitoring,
| mesh, minio... This does not make complexity disappear, it just
| abstracts it by creating another layer with an interface (k8s
| API) you may be more familiar with.
|
| Whole Kubernetes in practice is adding more and more layers on
| top of it. All those layers/tools are usually amazing but soon
| you need some real k8s skills AND some app skills to handle
| even unavoidable scenarios like ... upgrades.
|
| That's why it's so confusing sometimes and has such diverse
| opinions on. There aren't many environments that run vanilla
| cluster...
| levkk wrote:
| In the case of running Postgres on K8s, the problem arises
| immediately when you try to resize a data volume and you can't
| because the API doesn't support it. K8s is not really for
| stateful systems, yet, and systems like Postgres that prefer to
| manage their own resources, you don't want another layer which
| doesn't cooperate to get in your way.
| mikepurvis wrote:
| "K8s is not really for stateful systems"
|
| As a relative novice in the space, I'm grateful to hear
| someone say this out loud. K8s seems perfect to me for
| quickly scaling transient stuff like pipeline workers, web
| servers, but I've always been pretty leery of giving up the
| trivial snapshotting and rollbacks and other creature
| comforts of old-school virtualization when it comes to
| deploying long running applications, databases, and so on.
| And I've always felt kind of kind of guilty for not being on
| board to just mindlessly k8s-all-the-things.
| snapetom wrote:
| Our company is finally looking at containerization and
| orchestration, and one product had the gall to say, Docker
| isn't for us. The non-technical people gasped because all
| the other products are moving towards orchestration!
|
| Why? it's an ancient Windows Client/Server app. Each "node"
| manages its own state and communicates with each other in
| this proprietary, janky-ass way. It takes _10 minutes_ to
| start up a node.
|
| K8/Swarm isn't going to do squat for this team except maybe
| launch dev/test environments a little easier.
| DasIch wrote:
| Here is an announcement from last year stating that volume
| expansion is stable since 1.24:
| https://kubernetes.io/blog/2022/05/05/volume-expansion-ga/
|
| The zalando postgres-operator also mentions as a feature:
|
| > Live volume resize without pod restarts (AWS EBS, PVC)
| levkk wrote:
| What about changing iops or storage class?
| cwiggs wrote:
| > ...the problem arises immediately when you try to resize a
| data volume and you can't because the API doesn't support it
|
| What API doesn't support it? k8s has support for resizing PVs
| and has for a while now. AFAIK all 3 cloud providers (and
| more) support increasing the PV using their storage class.
|
| > K8s is not really for stateful systems, yet
|
| Is this written somewhere or is it just your opinion?
| jauntywundrkind wrote:
| Is the complexity better or worse than alternatives? What are
| the alternatives?
|
| People use complexity as a boogieman to justify throwing
| together their own really wild chaotic & only so-so tested
| "simple" alternatives _all the time_.
|
| To me, this feels like a modern wonder. We have layers of
| responsibility. Many people operate Kubernetes clusters already
| for all kinds of reasons. It provides a powerful broad base.
| Now on top of that, we can operate Postgres, with a very smart
| failover system that's super well tested & broadly used, that
| leverages this competent starting place.
|
| What are the Fears Uncertainties and Doubts you have that make
| you scared about composition? What would help address specific
| concerns? To me, this division of responsibilities & use of
| consistent platform for a variety of needs feels like a huge
| win.
|
| People love "simple" options but they're not. Run naked through
| the woods like savages option has appeal, but just getting
| started keeps adding up:
|
| Sure, just add some bash scripts for some wal backups that go
| off-site. Easy! Install pgbouncer like one does, just a quick
| install, point it at the right systems. Setup some replication.
| Install and configure more kind of hairy software to make it
| HA. Configure some TLS cert yourself to not send naked traffic
| over wire. Add monitoring!
|
| Then operationally, how quickly do you think you'll be able to
| fail over (with ogbouncer staying ok), failback, do an upgrade,
| add more replicas, lose a replica? Can you rotate cert reliably
| in a timely fashion? Can your replacement? How well did you
| document everything? Will you have all the monitoring you need
| when incidents start coming in, or did you just spitball a
| couple metrics into place?
|
| The alternatives, in my view, are _obviously bad_. You can do
| them. Either cheaply with risk or industriously with effort &
| applied-talent. But having cohesive _autonomic_ systems at our
| back that try to help, that can faultlessly do many common
| tasks with perfect accuracy (across unimaginable numbers of
| systems, with perfect consistency, in record time): that feels
| like a massively better place in the universe, one that I don
| 't get why so many people kick scream & drag against. Rarely
| are their arguments well elaborated ("scary"), and their
| counter-suggestions feel like they massively underrated how
| multi-faceted & carefully connected production systems are, for
| good reason, and how hard it can be to remember to not forget
| to change X when you do Y.
| marcosdumay wrote:
| > What are the alternatives?
|
| Installing the Postgres on the computer, or on a VM.
|
| Really, just the fact that some people keep asking this
| question is enough to question everything else they say. The
| alternatives are _obvious_.
| seanhunter wrote:
| Exactly. The way people successfully did HA and scalability
| on databases for more than 20 years before Kubernetes
| existed.
| znpy wrote:
| > Is there any way to overcome hidden complexity biting your
| hand other than studying k8s extensively?
|
| No. Technology is hard. Operating services is complex, and gets
| more complex as the scale gets bigger.
|
| Any attempt at making stuff simpler is usually either moving
| complexity elsewhere or making it more expensive (and in some
| cases, both).
| Nican wrote:
| I am happy using CockroachDB. The performance is not as good,
| since all your database writes require a 2 out of 3 quorum. But
| managing the database with the CockroachDB is pretty simple,
| since it can perform a rolling upgrade with no downtime.
|
| Upgrades is handled with an operator, and happens by waiting
| all queries to finish, draining all connections, and restarting
| the pod with the newer version. The application can connect to
| any pod without any difference.
|
| I perform upgrades twice a year, never really worried about it,
| and never had any availability problems with the database, even
| when GCP decides to restart the nodes to update the underlying
| k8s version.
| snapetom wrote:
| I've never been a fan of deploying DBs, either traditional
| RDBMS or distributed, under containerization/orchestration
| other than quickly spinning up dev and test environments.
| Certainly not for production. Databases have been built for
| high availability for many decades. Sure, maybe RDBMS doesn't
| scale quite as easily as a node API application, but it's still
| not really rocket science and you usually have some leeway if
| you get a spike.
|
| Adding Swarm or K8 just seems like redundant and unnecessary
| complexity.
| steveBK123 wrote:
| Wouldn't most data driven, disk-IO intensive workloads be a
| bad fit for k8s? Especially when its not purely read-only..
|
| That seems the crux of it.
| snapetom wrote:
| Absolutely. I haven't gotten in the weeds of K8 volumes,
| but certainly have done a lot with Docker volumes including
| adding some PRs to fix some issues I've ran into. There is
| a lot under the hood with Docker volumes. With how DBs like
| to optimize their IO operations, I'm frankly a little
| amazed how it somehow mostly works between the DB ->
| orchestrator volumes -> file system.
| cwiggs wrote:
| This seems to be the common advise given, but I don't fully
| agree. There have been many times in my career where a DB
| was on a VM with the storage attached via a cloud providers
| block storage. When asked if we should move it to k8s,
| people are quick to mention k8s doesn't do well with
| persistent storage. However all of the big cloud providers
| offer the ability to easily create a persistent volume in
| k8s that then just creates a block device, attaches it to
| the k8s host and makes it available to the pod.
|
| So in both situations you have the same IO limits of block
| storage. The question is does k8s persistent volume api add
| enough of an IO bottleneck to cause issues, IME that isn't
| the case.
|
| Now if you want direct attached NVMe drives for higher IO
| than a network attached block storage will give, then it
| might be easier with a VM vs k8s, but I can't speak to that
| much.
| ysofunny wrote:
| once upon a time I set up an elastic search cluster in kubernetes
|
| after a lot of tweaking I made it so that the pods would be as
| big as the underlying hardware nodes. one pod one node. once that
| was working I realized that I was using the wrong tool for the
| job.
|
| the kubernetes tooling added nothing but complexity. needless to
| say I let it run like that having had wasted about a week getting
| it to work
| jhgg wrote:
| On the other hand, at a certain scale (running hundreds of ES
| nodes across 80 or so ES clusters), Kubernetes actually does
| make a lot of sense.
|
| At work, we moved from hosting elastic search on bare VMs to
| kubernetes. By leveraging scheduler policies we are able to
| pack / over-provision ES node pods of different clusters onto
| the same Kubernetes nodes, allowing for far greater resource
| efficiency, while being able to handle node failure while
| maintaining availability across all clusters. Additionally,
| this simplified operations significantly as we can now leverage
| the operator to do cluster wide operations (e.g rolling
| restarts, node OS upgrades, ES version upgrades, etc...) fairly
| easily.
|
| We did, however, go 6 years (and several hundred million users
| and trillions of documents indexed) without needing to use
| Kubernetes!
|
| We will blog about this at some point this year.
| jen20 wrote:
| A big part of the problem with Kubernetes is it doesn't make
| a ton of sense at small scale, and it just plain doesn't work
| at large scale.
|
| Nomad is generally speaking a much more appropriate
| technology when you hit the point of needing such a system.
| marcosdumay wrote:
| Did you keep adding and removing replicas into your cluster
| based on a scheduler policy? How often did you adjust the
| number of nodes (and how long did it take to make a node
| available)?
|
| At the high-level you are describing your setup, it doesn't
| make sense. You'd spend way more resources managing any
| cluster than what you would gain from a normal-looking
| policy. I seem to be missing some important detail.
| ssijak wrote:
| Is Kubernetes still hard in 2024?
| _joel wrote:
| I don't know about hard but it's fairly straightforward to spin
| up clusters and maintenance seems to have become less of a
| headache (at least imhe). It depends on what you will be doing
| with the cluster and how you use it.
| azlev wrote:
| Yes. Orchestration is not easy.
| jamesu wrote:
| From recent experience I'd say it's the sort of tech that
| starts off simple enough with the right distribution, but then
| gets more complicated the deeper down you dive.
|
| Probably the most hard thing I found was wrapping my head
| around the way the storage works.
| danielvaughn wrote:
| Granted I'm new to devops, only been on a platform team for
| about 9 months now, but I still feel incredibly dumb every time
| I try to work with it.
|
| That being said, we're also layering a bunch of stuff on top -
| helm, nginx, GKE, terraform, as well as a mountain of other
| things, and then to top it off we have a bunch of shell scripts
| doing random things to help tie it all together.
|
| Normally I can pick things up pretty quickly. I just built a
| parser with tree-sitter, despite knowing virtually nothing
| about language design. Didn't take very long.
|
| But the modern devops stack is a learning curve like I've never
| seen before. It's taking me more energy to learn it than it did
| for me to learn programming itself. Then again, maybe I'm just
| getting old.
| cies wrote:
| I heard people say its not for small teams. On top of that I'm
| not convinced it's a good tool for anything "storage" (i.o.w.
| use it for compute/network loads).
| __MatrixMan__ wrote:
| I can only speak from the perspective of somebody trying to
| manipulate k8s environments for use in test, but I'd say yes.
|
| Four times across three companies I have run across frameworks
| which set everything up out of band and then ask the tests to
| test it. Maybe these are shell scripts written by the k8siest
| person on the team. Maybe they're something like tilt. Whatever
| the case they're always a black box to the majority of the
| people who are writing application code.
|
| They get you 90% there, but eventually somebody wants
| assurances that some environment variable has the desired
| effect, and suddenly you need to penetrate that black box and
| change it so that there are multiple kinds of "up" and the
| right tests run against each state.
|
| K8s tooling is commonly installed via curl, so once you unravel
| the black box and integrate it with your tests you end up with
| a lot of fragile interfaces to things like kustomize, kubectl,
| kind... Fragile because maybe the other dev has a different
| version installed. Nix dev shells solve this, but you can't
| usually get the whole team on board with Nix so version
| mismatches come up often and are often difficult to debug. You
| end up in a state where whoever wrote the initial setup scripts
| is authoritative about the dependencies, and you have to ask
| them what they have installed if you want things to work (it
| was easy for them, they just used whatever was lying around at
| the time).
|
| These aren't directly deficiencies of k8s, once you see the
| light (which takes a long time) it's pretty easy to work with,
| but like so many other technologies, the devil is in the
| peripheral tooling and the culture. K8s doesn't (yet?) have a
| very nice boundary with other language ecosystems, it reminds
| me of Java in that way. The die hard k8s people often want to
| solve problems by bringing them more fully into the k8s way of
| seeing the world and I just don't think that is consistent
| enough with reality to be our everything.
|
| I've cultivated a begrudging respect for it, but I still don't
| like it. If I break free and start my own company, I'll publish
| an operator so that my stuff can be installed into k8s, but I
| don't intend to make it primary in any way.
| 8organicbits wrote:
| I've seen projects run on Cloud Run because they fit that
| model. But if you don't quite fit the model, then you fight the
| abstraction in odd ways. K8s gives you control over more
| things, so the easy stuff feels hard but the harder things
| aren't as hard. With k8s operators you can stand up pretty
| complex things very quickly and robustly.
| cwiggs wrote:
| IME yes, yes it is. I usually tell people running a k8s cluster
| is similar to running your own "cloud". If you want to just
| deploy an EC2 instance you just tell AWS you want an EC2
| instance and you are done (mostly). You don't have to worry
| about if the hardware under the EC2 VM has enough resources,
| you do have to worry about that with k8s though. If you want to
| lock down the EC2 VM to have certain permissions you use AWS
| IAM, with k8s you have to use cluster roles and cluster role
| bindings. You can apply these 2 examples to many other things
| in k8s vs "cloud" provider, Ingress, persistent volumes, etc.
| jonfw wrote:
| It really depends on the design of the workload. Kubernetes
| makes orchestration more simple, but it doesn't remove all of
| the complexity of orchestrating something.
|
| If you don't need much orchestration (which is true for a lot
| of postgres users), the complexity from kubernetes is
| compounded on top of the complexity from postgres without
| generating much value.
| justanotheratom wrote:
| dumb question - where is the storage kept?
| iamgopal wrote:
| Me who have never used kubernetes, what if node crash ? Will I
| lost everything ?
| stanac wrote:
| No, attached storage is not part of the node (not directly).
| It's something like attaching external (host) directory to a
| docker container. Your can kill the node/pod and storage is
| not affected, later you can attach new pod to the same
| storage.
| rad_gruchalski wrote:
| The answer is: it depends. It depends on if you use
| persistent volumes, and how well is your pv isolated from the
| failed node. If done right, no data loss.
| marcosdumay wrote:
| A big emphasis to the "if done right" part.
|
| You should test your setup, because it's very often not
| done right, and it's easy to overlook a problem.
| siamese_puff wrote:
| I think people over think how hard this actually is. Data
| replication isn't a new concept. You can use RAID with a NAS
| or setup async replication with operators for things like
| Postgres/SQL.
|
| Obviously it's worth doing simulated disaster recovery to
| ensure you would recover if there is hardware failure. The
| larger the scale and throughput with parallel writes against
| the same keys, etc then the more complicated the setup will
| be. I hope to write more on this topic, but setting up a
| persistent volume with a NAS is a great way to ensure high
| durability.
| bo0tzz wrote:
| Wherever you want, Kubernetes supports basically every storage
| backend you can imagine.
| doublerabbit wrote:
| > Kubernetes supports basically every storage backend you can
| imagine.
|
| Text files?
| znpy wrote:
| The author mentions Longhorn, a storage solution from the
| people behind the k3s distribution, so I'm assuming the data is
| stored in pvc provisioned through longhorn.
| siamese_puff wrote:
| OP here. In general, yes! That is correct. I am using
| Longhorn with K3s, it's very slick and easy to get started
| with. In this case, I'm using multi-replica cluster with
| Zalando _only_.
| siamese_puff wrote:
| OP here. Currently I am using Longhorn on this cluster which
| does data replication on SSDs attached directly to the nodes.
| My backlog item is to run an external NAS with RAID. In this
| post specifically, the replication is handled by Zalando and
| not Longhorn, but the storage itself is on each node (specified
| by the node selector).
| cwiggs wrote:
| All the cloud providers offer a storage class for k8s. The
| storage class allows you to tell k8s that you want a persistent
| volume (PV) and it will make API calls to the cloud provider to
| get you a block storage device. You can tell k8s you want to
| use that PV in your pod and k8s will automatically mount the
| block storage to the worker node that your pod lives on and
| makes it available to the pod.
|
| OP uses Longhorn which is a whole other thing that I've only
| read about.
|
| For at home you can use other storage classes like ceph, NFS,
| etc.
| xenic wrote:
| "Zalando is a Postgres operator that facilitates the deployment
| of a highly available (HA) Postgres cluster."
|
| Zalando is the company. "Postgres Operator" is the software.
|
| Happy user here, not much complaints about the operator come to
| mind.
| siliconc0w wrote:
| I wonder why more don't take advantage of native k8s and just
| rely on it to move over the persistent volume and start the new
| pod. This may have some small amount of downtime but it's a lot
| less complicated.
| oxfordmale wrote:
| No, just no. K8s shouldn't be used to host database systems. Its
| main function is micro services.
|
| Cloud provider provide managed versions of Postgres that are
| highly available. Even if you self host, Kubernetes isn't the
| answer.
___________________________________________________________________
(page generated 2024-01-02 23:02 UTC)