back
224 comments
I made this decision at a startup (albeit when the eng team was ~30 people, and we had a monolith with ~10 supporting services). I wouldn’t do it again, even for the reasons stated in the article.

The uniformity is nice, we were moving from apps running directly ec2 instances provisioned with ansible. Each time we spun up a new service it was a process to get the ec2 instances provisioned just so.

But k8s is such a pain in the ass. One thing that I think people new to it don’t realize is that it’s not at all batteries included - to get a basic managed cluster setup, you’re still going to be installing a bunch of additional controllers (ingress, cert-manager, external dns to start). And then you’re on the hook for making sure all those processes stay up (hope the admission webhook controller for a critical resource doesn’t go down!). Then you’ve got to do a major upgrade on not only your cluster, but all of those controllers every ~3 months. And no one is shy about introducing breaking changes.

Also you’re introducing a huge amount of complexity with the k8s networking and dns layer that most startups have zero need for (if you’re on EKS, make sure to read about scaling and monitoring CoreDNS).

I think there is a real hole in the market for a simple solution that lets you deploy some containers to some instances in a declarative fashion without all of that complexity and does decent LTS versions. I imagine there’s something out there that does this, but k8s has really sucked up all the oxygen.

Pretty sure if there was a simple alternative, people would hate it.

Everyone initially wants thing A. But then they want to customize it to do all permutations and combinations n of A, B, C. They want it to be extensible. They want redundancy. They want orchestration. They want integration.

It’s why practically every config file format eventually becomes its own scripting language. Even HTML started off simple — now ridiculously complex — all the more ironic since practically nobody writes it by hand. Instead of CSS simplifying it, it became more complex.

There is another thing that is extremely customizable and extensible. It’s called a programming language. People write programs to solve specific problems.

There seems to be a perverse trend of cobbling together a Byzantine mesh of libraries, plugins, and services with complex configuration files to make it do practically everything possible. We just used to write software for such purposes…

And for anyone who thinks HTML is simple… the A (anchor) tag has an “ping” attribute that results in POST requests to a list of URLs when a link is clicked ! The list of attributes and resulting variations in behavior is quite mind boggling. It was supposed to be a damn link! https://html.spec.whatwg.org/multipage/links.html

I really wish there was an 80% kubernetes. I think you could get there with some changes:

1. No overlay networks. 1 IP per machine. pods use dynamically allocated ports, and the kubelet enforces pods listen only on their assigned ports using seccomp.

2. No kube-proxy or equivalent Layer-4 "load-balancer". It's not good, but it's often used. You should use some kind of Layer-7 load balancing instead. Also you need to look up the port number from (1). This also greatly lessens the need for DNS.

3. A better config language. YAML and helm templates are terrible. kustomize is built into kubectl, but it's frustratingly limiting and also still very complicated. Something like nix would have been great. This can make it easier to upgrade third party configs since you can have more logic to validate and merge your settings with upstream defaults or templates.

4. Maybe an EBF-like for the api server? If the built-in k8s objects don't have a setting for something, then you need to write an operator or control loop yourself and then run that too, which is a big lift. Over time, k8s just keeps adding more and more built-in things and then revising them, which creates a ton of churn. If you could easily script simple operations, then they wouldn't have to build in every permutation ahead of time. E.g. the HorizontalPodAutoscaler has 24 config object types with several fields each, but all it does is set replicas based on data read from the api-server, so it could be replaced by some kind of flexible script that runs in the control plane.

I don't know... running a startup sized kubernetes is relatively easy and pain free these days (k3s). Especially when it comes to scaling up.

CNPG is an absolute monster (in a good way). cert-manager is easier than the docker alternative, calico has never failed me (except in bgp mode which has some footguns like not being able to come back from a dead state since it has a chicken and an egg problem unless you point it to the external load balancer which I would have known if I read the documentation). trafeik is all you need. talosos largely mitigates the bare metal problems and comes pre-hardened and pre-optimized.

I solo most of my development projects and have used k3s for all of them. The only complaint is that cert-manager by default will fail silently and your certificates will expire. I largely mitigated this by having proper visibility setup via grafana and automated alerts (warns if certificates are about to expire) which should have been done by me anyway.

Two years ago I'd agree, today with LLMs everything I have runs talos with fully automated updates and I haven't had to be on-call for almost a year.

> I think there is a real hole in the market for a simple solution that lets you deploy some containers to some instances in a declarative fashion without all of that complexity and does decent LTS versions

Hashicorp's Nomad basically is just that, supports various way of running stuff too which is neat. Shame about the license change which basically killed all my interest in it, so seems the hole is indeed still unfilled.

> One thing that I think people new to it don’t realize is that it’s not at all batteries included - to get a basic managed cluster setup, you’re still going to be installing a bunch of additional controllers (ingress, cert-manager, external dns to start).

And if you can do this again, what's your solution to reverse proxy, certificate management, DNS...etc? I guess you can docker-compose some custom stack on a single machine, maybe add one more machine then you can say it's HA enough for small scale. But you can also spend the same amount of time to install those kubernetes controllers with zero customization. In my experience, if you go with the default configuration, most of the well-maintained k8s components are boring as hell these days.

> (if you’re on EKS, make sure to read about scaling and monitoring CoreDNS)

If load to your service increases, you need to scale up/out your service. This is universally true. Do you have a proprietary solution that's easier and more reliable than bumping up the replicas count in kubernetes?

There are lots of design decisions in Kubernetes that I hate. But if you want me to choose between Kubernetes and any proprietary stack, in 2026, I would definitely choose Kubernetes.

Docker swarm is that simple solution you're looking for. But people don't need simple solutions. They want scalable solutions and Kubernetes fits this niche perfectly. You can deploy it on single server today and scale to 100 servers managed cluster tomorrow.

Just to provide a similar example. Linux system is insanely complicated. Kernel alone has thousands of options. Distos have tens of thousands of packages. Wherever you look at, everything is hard and complicated. Firewall, containers, init system, filesystem hierarchy, storage layers. One would think that some people desire simpler operating system. But everybody uses Linux despite all these complexities. Try to find OpenBSD in production, for example. It's not easy.

Dear friend, you have built a Kubernetes

https://www.macchaffee.com/blog/2024/you-have-built-a-kubern...

k8s is not a pain, I would never return to something like Pupet / Ansible / etc ... to deploy bare ec2 instances, it's just re-inventing the wheel badly.

Just use ECS / Fargate with an ALB in front if you need a simpler use case.

>I think there is a real hole in the market for a simple solution that lets you deploy some containers to some instances in a declarative fashion without all of that complexity

That's how I see it as well but it's really tough to go against the grain. I have a small enthusiastic community of users around Uncloud (https://github.com/psviderski/uncloud) who went full circle - fed up with k8s and came back to simple, boring declarative Compose deployments across a handful of interconnected hosts.

Uncloud is essentially a cluster version of Docker Compose without a control plane and cluster management overhead.

One year ago I might agree that Kubernetes is an overkill but today?

Ask your favorite GPT to generate manifests, get primary app into cluster with telepresence or execute straight from container and switch contexts and clusters like it's 90s again.

One reason I dislike Docker Compose and Docker is lack of isolation. Yes sure if you put your arm deep enough you can get it, but on local k8s I can spin cluster per workspace and not worry about conflicting ports between PostgreSQL instances.

Before LLMs writing consistent YAMLs was PITA but today on low/development scale it's pretty much free lunch.

So I’m personally a huge fan of k8s and while I agree it may be „complicated”, it’s because deploying applications is complicated. (I want to point out that there is no requirement no set up cert manager, ArgoCD, external secrets, etc. - and many people who’d consider a VPS would happily slap a .env with an unencrypted secret then ssh to update, but when they choose Kubernetes they take the long route of doing proper GitOps and complain that there are so many things to configure :)

But I found funny that the OP summarized to use Kubernetes when CTO is no longer the only dev.

There is a core 20% to Kubernetes which is very nice, mostly being the Deployment and Service management stuff. That along with a very basic GitOps for cluster management (an infra repo for operators using Flux, applying service level yaml from app repos in CI) above a cloud managed Kubernetes cluster, where you still keep your DB and build servers off the cluster, can be quite nice for a small team.

Beyond that, there are massive holes of despair to fall down if a novice team starts to engage with extensive operators (starving the control plane), DB operators (distributed persistence) and build operators (spikey, expensive loads). At least, I know that I've had to dig out of those holes.

I just hope people don't use k8s in the same way many use microservices: as a way to introduce complexity for complexity's sake.

None of those arguments make sense.

Uniformity ? Try deploying openbao inside kube, if kube decides to restart your pods, you're in for unsealing them at 3am, waking up everybody who owns a Shamir key. So bao stays out of the cluster, or pinned to certain nodes, defeating the purpose entirely. Also, with the ultra wide variety of tools at every layer of the stack, uniformity is a joke ; there are no 2 kube cluster deployment that are the same really.

Standardized knowledge ? The operating system is standardized knowledge. Any competent SRE should be able to login into a Linux box and figure out what's running there. And if you let your previous ops shadow it all you're just a pretty bad CTO.

Tracing who does what ? First of all anybody with admin access can run one time jobs just like anybody with sudo can run one time commands. That's like chapter 01 of the kube doc. Also again at the kube layer itself, below the helm chart, the ops who set that up or updates it can and will change stuff that breaks stuff.

Kube isn't necessarily bad and has it's purpose but it's not a product. It's like Linux, a complex piece of tech that requires a lot more knowledge than "just push this helm chart" to work.

Adopting k8s when you hire your _second_ engineer (first after the CTO)? That’s a red flag that the CTO’s priorities are wrong and he’s just enjoying tinkering with his infra instead of solving the users’ problems.
Well, I totally get the benefits that made those people choose Kubernetes. It's just that those benefits could be had w/o running a massively complicated piece of machinery that is mostly engineered to solve problems I don't have.
It's not uniformity, it's cargo culting and offloading thinking to group norms. Doesn't help engineers are some of the most arrogant people alive and refuse to admit anything is complicated, as they consider it some kind affront to their intelligence.

I would not advise asking the majority of CTOs these questions either. Many got to that position by saying what people want to hear, which is the "average" safe answer. They will parrot whatever is "hot" at that time because it's the least risky response. They are not your friend nor a reliable source.

> The CTOs I talked to aren't making a dumb choice. They're solving real problems.

Unrelated to the content of the article, this sentence structure is a dead giveaway of LLM writing.

This seems to be less about K8’s and more about the infra as code movement. It doesn’t matter if you use K8, CDK, or terraform - you get the same benefits the OP stated across the board.

It is nice to be able to have a consistent deployment pattern, with traceability, rollback support, and production approval checks. It’s nice to not have some archaic something stuck in someone’s head. It’s also nice to be able to see how something works by reading the code, which is usually up to date and deployable.

Kubernetes is so ubiquitous that yeah, as long as you're not trying to run it yourself, a small Kubernetes cluster just isn't all that much to manage. I think it's been so long, people forget how annoying it is to manage servers. All said and done Kubernetes is becoming more the "Boring Technology"
> I still don't totally get why the shift happened when it did. Five years ago all three camps were doing fine. Now the VM+systemd crowd has basically disappeared from job postings, serverless stayed niche, and K8s just won. > > My best guesses: managed K8s (EKS, GKE, AKS) got mature and the talent pool flipped: enough people learned it that hiring for anything else became the harder choice. And Helm made "just use someone else's chart" a real option. But I'm not certain. If you were there for the shift and have a better theory, I'd genuinely like to know.

Pretty much, almost. Have spent a bunch of time in my career working on the "VM + systemd" setups, stuff running on a rack, or in an ec2 on cloud - managed kubernetes is a lot better for me than those cobbled together messes. There's "easier" setups but usually end up costing me a lot more in time and $.

To answer simply, it became good + convenient. I could complain about plenty, and people here like to, but honestly you couldn't pay me to go back to the old way. The one legitimate gripe is the upgrade schedule is exhausting, on AWS it's about every 6 months before you go into extended support. I also hate being at the mercy of arbitrary decisions like "ok we know a huge chunk of the web going back a decade has architected off our Ingress API, but recently we decided we dont really like that way anymore and we want you to use Gateway API instead, so, um, like ya we know it just killed off one of the most used open source ingress configs (ingress-nginx) but yea trust us bro this is going to be so much better" kind of thing.

Even as a solo dev there's generally been a yawning gap between k8s and manual infra that nothing has ever filled that well and it's part of why things like Heroku were so popular for a while.
The reason this is accelerating recently is agents are really good at spinning up k8s clusters. They've made devops work super super simple. Basically all the annoying stuff you know you should do but it's way too much hassle - using let's encrypt to create unique certs for every app in your cluster to enable zero trust, configuring permissions and security profiles for everything etc etc (never mind just standing them up in the first place) - it's all simple now.
> First one was uniformity. Every service deploys the same way.

My current company makes this claim, but it's not true. They also have serverless apps, and also have some services running directly on EC2.

They just think of the Kubernetes deployments as the "standard" way.

> Second was shared, hireable knowledge. K8s is basically a lingua franca now.

People were demanding experience with Kubernetes, long before it was reasonable to expect it. Everyone added it to their resume, because they had to.

In addition to all of OP’s points, another reason k8s is getting popular is that LLMs have made them easier to use! It's reasonably well represented in the dataset and there are pretty strong monitoring and observability tools and verification gates to make sure that you've specified your cluster specifications correctly.
The knowledge is in the YAML

Exactly why I hate CloudFormation, K8S, GitHub Actions, etc. yaml is a terrible format for the knowledge encoded in these artifacts.

This is exactly why I call it 'resume++'. You have to use it to attract talent. People want to use it to expand their employment pool. This is not justification to using it.

To use it is a whole different question, and not in any way related to job interviews. I have worked in places that are crazy for not using it and others where using it was even crazier.

It's a bit odd that the author presents no data other than their interviewing and declares that the shift happened recently. It's not true, there's been steady growth of adoption of kubernetes for years. Just reading CNCF surveys from last years before posting would tell them that.

Their identified reasons are OK though.

I should add a disclaimer that I've never used kubernetes or docker, the latter mostly because I prefer the mental model and data encapsulation of a VM per service.

That said, the app I'm developing in my startup is designed with scalability from the outset. I have a single setup script for each type of node, and can take a fresh ubuntu install and just "wget -O- $URL | sh" as root and it sets up the node from scratch and/or reconfigures it to the latest configuration. That does all the ancillary stuff, setting up sane firewall defaults, blocking SSH from non-whitelisted IPs, setting up NTP, borg backup and zabbix (both require manual work on the respective backing servers currently), setting sane system configs (e.g. systemd logs limited to 100MB), wireguard (for the backends that distribute sqlite databases using litefs), etc. and installing the relevant packages with my software.

The actual backend application is built into a debian package automatically, so it's just a case of adding my private repository to apt sources and installing it. Updating a machine is just "ssh root@$MACHINE 'apt-get update ; apt-get install $APP'". I probably could automate that with ansible, but I prefer to upgrade them piecemeal while I'm testing out an upgrade, so I have a couple of bash scripts that do the ssh in a for loop instead with different targets in each.

This has the advantage for me of being able to buy any old VPS from a cheap provider and add it to my pool in minutes.

I'm sure I could end up with something that's just as easy to update with kubernetes, but it seems like another big learning curve with dependencies that probably change every few months and require me to keep learning new things just to keep it running. I understand my bash scripts, and know they won't just stop working going forwards (modulo exceptional events like having to migrate to systemd scripts, but that kind of change is usually only required on a very few major OS distribution upgrades).

I already have enough pain from some of my tech depending on other people's projects (I have a frontend app written in Flutter, and forced SDK upgrades about every 6 months and then resulting issues with toolchains I haven't even chosen to use, like gradle and kotlin, that seem to break everything every release), that I have no great desire to rebuild everything on someone else's deployment framework. When I get to the point of hiring others to help, I'd hope they'd be clued in enough to understand a simple bash script that sets up everything, and logically follow it through.

I’m personally a huge fan of k8s and while I agree it may be „complicated”, it’s because deploying applications is complicated. (I want to point out that there is no requirement no set up cert manager, ArgoCD, external secrets, etc. - and many people who’d consider a VPS would happily slap a .env with an unencrypted secret then ssh to update, but when they choose Kubernetes they take the long route of doing proper GitOps and complain that there are so many things to configure :)
I pushed k8s in the medium sized company I work for much the same reasons. We use flux for gitops which works really well. The problem is we now have as many clusters as we did bare metal hosts before. There's production clusters, dev clusters, ones in other regions etc. The idea was to have "one place, one way to deploy" but it's actually many places. Am I doing it wrong? Should it all be one cluster and just have different nodes for different reasons and RBAC etc?
Taught me that companies follow hype.

I worked once at a bankm fully kubernetes, the amount of problems were out of reality from this world.

Complexities are being added for no reason at all.

I’m not sure it will cause a reversal, but I bet the k8s takeover wouldn’t have happened with costs of today. You inherently end up using far more resources than you need - between HA, built in services, overhead per node, etc.

I personally will be using more resource efficient approaches in everything I do. Question is just what provides the closest set of benefits without the full k8s weight.

I had a similar experience after a recent job search and started working on a 'kube-lite' that just uses object storage for coordination and normal cloud primitives like auto-scale groups (skiff.pages.dev).

I ended up in a different non-SRE role but if you're interested in working on it, please let me know and I'd love to walk you through it.

I'm not asking about kubernetes in interviews too, may be trying to understand do the interviewer understand how containerization, CNI, kube-proxy working under the hood. For me the best indicator of strong engineer is that he is interested enough to go under the hood, and understand the foundations rather the tools.
AI, managed control plane, minikube

That makes it a no brainer for me for basically any sized project.

Small project? -> minikube single node deploy it.

Tiny project? -> minimum a docker container

I cringe watching anyone build and run code on a raw machine even locally without atleast a container. The endless hours of headaches you avoid is obvious k8s is just the natural extension from this.

I assumed the reason there are so many job offerings for kubernetes engineers is either because these jobs have high churn, or because kubernetes takes more work to maintain. I think linux+nginx+postgres+favourite-lang is still the most sensible way to go. Good engineers don't overcomplicate things.
The grass is not always greener. I would change the Service Fabric crap I have to deal with any day to K8s.
I’ve worked for a few startups now and we’ve stayed away from k8s, instead being perfectly happy with ECS services managed via terraform.

It gives most of the benefits the author mentions (traceability of changes, clearly written down infrastructure), without the complexity of k8s.

A pretty nothing burger of a post with a bunch of ai-isms. Is this written by a real human?

K8s is a complicated beast. CTOs hiring for their 10 person company because of its "used everywhere" is a bad reason to adopt a major piece of technology. You can always graduate to it later if need be.

I'm going against the grain but I read: we have a cultural/policy issue and we 'fix' it with tools.

I think what you hear is never the whole story, there is much more going on.

I call BS on that. I serve SMB clients and many are happy like a clams with monoliths deployed using those proverbial bash scripts that also does lots of other things. Understanding scripts in the age of AI is trivial for newcomers. I for example fed my own uber script to AI as an experiment and it has produced all encompassing nice documentation with examples and tests.
I am still unsure if the complexity is an inherent trait of any big organization or a byproduct of using kubernetes
Multi-cluster federation is still hard.
OP gets it.

Right now, I’m one dinosaur managing a startup’s tech portfolio. Everything lives in my head first, then in my break-glass vault for addressing the bus problem. Our public cloud footprint is a single KMS for backups. We have no VMs, everything is a cloud service.

The literal fucking second we have real infrastructure requirements for compute, it’s right to GCE. No ifs, ands, or buts. Here’s our Git Repo, here’s the managed K8s control plane, make it work.

If (or when) we need on-prem compute, we add them to the K8s control plane as worker nodes and taint accordingly.

It’s just so much more interchangeable, even if the learning curve for non-SDEs can be a little steeper than VMs.

Here's my conspiracy theory --

There's a certain type of engineer (maybe 25% of them) who does "hype-driven-development." No matter the technology, they are huge advocates for the technology. The hype may be absolutely real, complete nonsense (e.g. mongodb), or somewhere in between (ai). The vast majority of the time it's hype for a new technology that feels 90% the same from the end-user perspective (react vs vue, docker vs colima, go vs other, whatever vs whatever).

These engineers though, only care about something when it's new and trendy enough to be a differentiator. This is because they don't give any hoots about the actual usefulness of anything, they are just trying to differentiate themselves in a market by leveraging vibes rather than raw competence. I think these types of engineer drove kubernetes for companies that don't need it, but tipped the scales enough that it has critical mass.

The irony being kubernetes is way too heavy/clumsy an abstraction for most companies. The savings of packing pods onto the same node is usually a tiny fraction of the engineers' salaries who are managing it.

The other irony is now that kubernetes isn't the new sexy thing, but a standard tool that AI or a normie can do all the hard work for, the hype driven engineers are off looking for the next thing.

> shared, hireable knowledge

So it's the same motivation as enterprise java.

for smaller b2b startups - serverless is still a win.