In the case of running Postgres on K8s, the problem arises immediately when you try to resize a data volume and you can't because the API doesn't support it. K8s is not really for stateful systems, yet, and systems like Postgres that prefer to manage their own resources, you don't want another layer which doesn't cooperate to get in your way.
back
6 comments
I don’t agree here. There are operators like the one I’m a maintainer of (CloudnativePG) which works directly with Kubernetes, teaching it how to handle Postgres clusters as a coordinated set of instances. Enormous improvements have been done in the last couple of years, and we are particularly focused in working together with storage groups in Kubernetes to handle database workloads, such as for example declarative support for tablespaces and volume snapshots.
I suggest you to read CloudnativePG documentation as well as this article: https://www.cncf.io/blog/2023/09/29/recommended-architecture...
Also watch the video of my talk at last Kubecon in Chicago about handling very large databases.
I hope this helps.
Thanks for working on CNPG!
I have been a very happy user of CNPG even with occasional issues (database backup to GCS tripped me few times, but it works - mostly a bit of UX that I never was sure wasn't some fail of mine).
Now I only really need to add some automation for handling "recover the database and switch over clients to it" that is more automated (I understand why CNPG doesn't do recovery to existing database, but it is a bit annoying)
"K8s is not really for stateful systems"
As a relative novice in the space, I'm grateful to hear someone say this out loud. K8s seems perfect to me for quickly scaling transient stuff like pipeline workers, web servers, but I've always been pretty leery of giving up the trivial snapshotting and rollbacks and other creature comforts of old-school virtualization when it comes to deploying long running applications, databases, and so on. And I've always felt kind of kind of guilty for not being on board to just mindlessly k8s-all-the-things.
Our company is finally looking at containerization and orchestration, and one product had the gall to say, Docker isn't for us. The non-technical people gasped because all the other products are moving towards orchestration!
Why? it's an ancient Windows Client/Server app. Each "node" manages its own state and communicates with each other in this proprietary, janky-ass way. It takes 10 minutes to start up a node.
K8/Swarm isn't going to do squat for this team except maybe launch dev/test environments a little easier.
>but I've always been pretty leery of giving up the trivial snapshotting and rollbacks and other creature comforts of old-school virtualization when it comes to deploying long running applications,
You can do that too with k8s with APIs which support more than just one backend.
I am the author of k8s resizing feature and its been GAed for awhile and feedback we have got so far has been good. If anything running inside k8s makes it relatively easy to support resizing. You just need to specify new size for PVC and it will both perform resizing on the cloudprovider and of the file system (if needed).
Modifying IOPS and other volume attributes is something less frequently needed but we just released alpha support for that too, if you must need it.
We have also added support for reporting volume usage in CSI specs, which I know some operators use to automatically resize volumes when certain threshold is reached (I however do not recommend using ephemeral metrics for automating something like this). But point is - you can actually define CRDs that persist volume usage and have it used by an higher level operator.
Another thing is - k8s makes it relatively easy to take snapshots which can be automated too and that should give someone additional peace of mind if something goes haywire.
Obviously I am biased and I know there are some lingering issues that require manual intervention when using stateful workloads (such as when a node crashes), but k8s should be just as good for running stateful workloads IMO.
Another thing is - k8s volumes are nothing but bind mounts from host namespace into container's namespace and hence there should be no performance penalty of using them.
Here is an announcement from last year stating that volume expansion is stable since 1.24: https://kubernetes.io/blog/2022/05/05/volume-expansion-ga/
The zalando postgres-operator also mentions as a feature:
> Live volume resize without pod restarts (AWS EBS, PVC)
What about changing iops or storage class?
> ...the problem arises immediately when you try to resize a data volume and you can't because the API doesn't support it
What API doesn't support it? k8s has support for resizing PVs and has for a while now. AFAIK all 3 cloud providers (and more) support increasing the PV using their storage class.
> K8s is not really for stateful systems, yet
Is this written somewhere or is it just your opinion?
Volume resizing support has been added since ~1.11 which is like 5 years old at this point