▲ 281 points
back
251 comments
Previous discussion: https://news.ycombinator.com/item?id=19467067 (315 comments).
All these "maybe you don't need this or that X" posts die in an instant when the user already knows how to do X (when the learning curve argument is gone).
Let's get it right:
Kubernetes is really really cheap. I can run 20 low volume apps in a kubes cluster with a single VM. This is cheaper than any other hosting solution in the cloud if you want the same level of stability and isolation. It's even cheaper when you need something like a Redis cache. If my cache goes down and the container needs to be spun up again then it's not a big issue, so for cheap projects I can even save more cost by running some infra like a Redis instance as a container too. Nothing beats that. It gets even better, I can run my services in different namespaces, and have different environments (dev/staging/etc.) isolated from each other and still running on the same amount of VMs. When you caculate the total cost saving here to traditional deployments it's just ridiculously cheap.
Kubernetes makes deployments really easy. docker build + kubectl apply. That's literally it. Deployments are two commands and it's live, running in the cloud. It's elastic, it can scale, etc.
Kubernetes requires very little maintenance. Kubernetes takes care of itself. A container crashes? Kubes will bring it up. Do I want to roll out a new version? Kubes will do a rolling update on its own. I am running apps in kubes and for almost 2 years I haven't looked at my cluster or vms. They just run. Once every 6 months I log into my console and see that I can upgrade a few nodes. I just click ok and everything happens automatically with zero downtime.
I mean yes, theoretically nothing needs Kubernetes, because the internet was the same before we had Kubernetes, so it's certainly not needed, but it makes life a lot easier. Especially as a cheap lazy developer who doesn't want to spend time on any ops Kubernetes is really the best option out there next to serverless.
If learning Kubernetes is the reason why it's "not needed" then nothing is needed. Why use a new programming language? Why use a new db technology? Why use anything except HTML 4 + PHP, right?
BTW, learning Kubernetes can be done in a few days.
All of this glosses over the biggest issue with Kubernetes: it's still ridiculously complex, and troubleshooting issues that arise (and they will arise), can leave you struggling for days pouring over docs, code, GitHub issues, stackoverflow... All of the positives you listed rely on super complex abstractions that can easily blow up without a clear answer as to "why".
Compared to something like scp and restarting services, I would personally not pay the Kubernetes tax unless I absolutely had to.
If you're running on a single host anyways, why not just use init scripts or unit files? All Kubernetes is giving you is another 5-6 layers of indirection and abstraction.
EDIT: Quick clarification: still use containers. However, running containers doesn't require running Kubernetes.
> learning Kubernetes can be done in a few days
The basic commands, perhaps. But with Kubernetes' development velocity, the learning will never stop - you really do need (someone) dedicated part time to it to ensure that a version upgrade doesn't break automation/compliance (something that's happened to my company a few times now).
"BTW, learning Kubernetes can be done in a few days. " - Learn something in a few days and the ability to run it in production are completely 2 different things. Security, upgrades and troubleshooting cannot be learned in couple of days.
So let’s take step by step:
1. Learn shell scripts.
2. Learn Docker conf.
4. Learn yaml. (May be helm and plethora of acronyms specially designed for k8s).
5. Combine spaghetti of 1,2,3 to build container image scripts.
6. Tie it into CI/CD process for adventurous.
7. Learn to install and manage k8s cluster or learn proprietary non open source api of google or amazon or azure.
8. Constantly patch and manage plethora of infrastructure software besides application code.
Now with all this use a tool designed for million user application on an application which will be used by 100’s to 1000’s of users. I think k8s is designed for google kind of problem and is an overkill for over 80-90% of deployments and applications.
May be just use simple deployment with Ansible, puppet,chef, nix, gnu guix etc. to deploy and manage software based on necessity on a single vm and extend it to a large cluster if necessary of bare metal or vm or container in a cloud agnostic manner.
Not sure when the technology fashion overtook the infrastructure area like complex web app tooling in JavaScript world. K8s has its own place at scale required by handful of Organization with google level traffic and load for most traditional companies Simple cluster management and configuration management tool will work wonders with less moving parts and cognitive load.
> Kubernetes is really really cheap
In terms of infra costs, I can believe it. But what about engineer resources to setup and maintain your infra in k8s clusters?
Where I work, we have a full time DevOps team that's almost the same size as the main product teams. That's really not cheap.
Agree / disagree. I’ve been using k8s in large deployments (1000s of nodes) for about 3 years.
It’s easy to get started with using GKE, EKS, etc. it’s difficult to maintain if you’re bootstrapping your own cluster. 3 years in, and despite working with k8s at a pretty low level, I still learn more about functionality and function every single day.
I do agree it’s great tooling wise. I personally deploy on docker for desktop k8s day one when starting a new project. I understand all the tooling, it’s easier than writing a script and figuring out where to store my secrets every damn time.
The big caveat is - kubernetes should be _my_ burden as someone in the Ops/SRE team, but I feel like you frequently see it bleed out into application developer land.
I think that the CloudRuns and Fartgates* of the world are better suited to the average developer and I think it’s Ops responsibility to make k8s as transparent as possible within the organization.
Application developers want a Heroku.
Edit: * LOL
> Kubernetes requires very little maintenance. Kubernetes takes care of itself. A container crashes? Kubes will bring it up. Do I want to roll out a new version? Kubes will do a rolling update on its own. I am running apps in kubes and for almost 2 years I haven't looked at my cluster or vms. They just run. Once every 6 months I log into my console and see that I can upgrade a few nodes. I just click ok and everything happens automatically with zero downtime.
> BTW, learning Kubernetes can be done in a few days.
This is true if you are using managed k8s from a provider or have an in-house team taking care of this. Far, far from the truth if you also need to set up and maintain your own clusters.
I'm very comfortable with k8s as a user and developer, would not be comfortable setting up a scalable and highly available cluster for production use for anything more serious than my homelab.
If you are running K8s on a single VM, you by definition do not need K8s. Run your apps with Docker and save yourself 50 layers of complexity.
> Kubernetes makes deployments really easy. docker build + kubectl apply.
The parts between "docker build" and "kubectl apply" is literally CI versus CD; they're more complicated than 2 steps. And when there's a problem with either, K8s is not going to fix it up for you. You'll have to be notified via monitoring and begin picking through the 50 layers of complexity to find the error and fix it. Which is why we have deployment systems to do things like validation of all the steps and dependencies in the pipeline, so you don't end up with a broken prod deploy.
> Kubernetes requires very little maintenance
Whatever you're smoking, pass it down here... Have you ever had to refresh all the certs on a K8s cluster? Have you ever had to move between breaking changes in the K8s backend during version upgrade? Crafted security policies, RBACs, etc to make sure when you run the thing you're not leaving gaping backdoors into your system? There's like 50 million different custom solutions out there just for K8s maintenance. Entire businesses are built around it.
I'm not very familiar with Kubernetes, but from what I've seen, to me it looks quite complicated. I'm wondering, when people say they use Kubernetes and consider it easy/simple, does that typically include operating the underlying Kubernetes container infrastructure as well? Or does "using Kubernetes" usually mean deploying containers to someone else's Kubernetes hosting (like Google)?
I couldn't agree more. I've built a lot of sideprojects on Kubernetes and couldn't be happier with it. It's incredibly cheap (basically free in terms of resource usage), low-touch, and abstracts away so many pesky problems that I had to deal with before. My cluster has been running for 12+ months without a single application-level outage - containers simply restart and get rescheduled when something goes wrong, and external logging and analytics solutions are a breeze to integrate.
> BTW, learning Kubernetes can be done in a few days.
As someone who has deployed K8S at scale several times, this is nonsense. Learning K8S deeply enough to deploy AND MAINTAIN IT is a huge undertaking that requires an entire team to do right. Sure, you can deploy onto AWS via KOPS in a day, and you can deploy a hello world app in another day. Easy.
But that only gets you to deployment, it doesn't mean you can maintain it. There are TONS of potential failure modes, and at this point you don't understand ANY of them. When one of them crops up, who do you page at 3AM, how do you know it's even down (monitoring/alerting isn't "batteries included"), how do you diagnose and fix it once you _do_ know what's broken?
Not to mention the fact that you _have_ to continuously upgrade K8S as old releases go out of maintenance in under a year. If you're not continuously testing upgrades against production equivalent deploys, you're going to footgun spectacularly at some point, or be stuck with an old release that new/updated helm charts won't work against.
TL;DR: If you can afford a team to deploy and maintain K8S, and have a complex enough application stack to need it, it's awesome; but it's not free in either time or staff.
> Kubernetes is really really cheap. I can run 20 low volume apps in a kubes cluster with a single VM. This is cheaper than any other hosting solution in the cloud if you want the same level of stability and isolation. It's even cheaper when you need something like a Redis cache. If my cache goes down and the container needs to be spun up again then it's not a big issue, so for cheap projects I can even save more cost by running some infra like a Redis instance as a container too. Nothing beats that. It gets even better, I can run my services in different namespaces, and have different environments (dev/staging/etc.) isolated from each other and still running on the same amount of VMs. When you caculate the total cost saving here to traditional deployments it's just ridiculously cheap.
If you're running 20 apps on a kubes cluster with a single VM you are running twenty apps on a single VM. There's no backup, scalability or anything else. There's no orchestration.
your deployment is a hipster version of rsync -avH myapp mybox:/my/location/myapp followed by a restart done via http to tell monit/systemd to restart your apps. It is a perfectly fine way of handling apps.
k8s shines when you have a fleet of VMs and a fleet of applications that depend on each other and have dynamic constrains but that's not what most of k8s installations work
> I mean yes, theoretically nothing needs Kubernetes, because the internet was the same before we had Kubernetes, so it's certainly not needed, but it makes life a lot easier. Especially as a cheap lazy developer who doesn't want to spend time on any ops Kubernetes is really the best option out there next to serverless.
Only in a throw production code over the fence sense.
I am a one-person team running Kubernetes on Google Cloud. It costs around $62 per month before tax. (One n1-standard-1 node for $35.34 per month, HTTP Load Balancing $18.60 per month and SSD persistent disks for $8.50 per month). Kubernetes gives me the ability to scale up quickly when the need comes (which is soon, hopefully).
I evaluated AWS Fargate & Kubernetes, Azure and GKE before settling on GKE. Amazon charges $148 per month just for cluster management alone. Google and Azure charge $0 for cluster management. This ruled out AWS for me. AWS appears to be a reluctant adopter of Kubernetes. They seem to want you to use Fargate instead. I tried it and found it to be crap--very hard to get things running.
I was able to get Kubernetes running on Azure and GKE fairly easily. There were minor hiccups on both those clouds. With Azure initial creation of a AKS cluster failed because some of the resource providers weren't "registered" on my subscription. On GKE it was hard to get ingress working. Static IPs take a long time to take effect, and in the mean time you are fiddling with your yaml files trying to figure out what you may have done wrong, not realizing it is a GKE issue.
The awesome part of Kubernetes is that I didn't need to learn almost anything about Google cloud to get everything working. I only had to learn Kubernetes. If Google raises prices I can easily switch to Azure without learning any Azure technologies. My knowledge as well as my application is completely portable. I can't imagine doing any of this as a one-person team without Kubernetes.
Kubernetes is here to stay.
What will change or what will be enhanced are: - Minimum requirements to actually run it (see k3s) - More managed services (gke, azure and aws exists but also digitalocean) - More/better handling of stateful services - Simple solution for write once read many (relevant for caching and for ci/cd)
At that pace we are already with such a jung project, yeah this is great. This is huge.
And no one needs to migrate already to kubernetes! But it already does a few things out of the box which reduces the complexity: - easy cert management - internal loadbalancing - autoscaling - green/blue deployment - deployment
But you do see how the industry is struggling with certain problems: We are now with kubernetes moving into a cloud native area.
Everyone know has kubernetes available. There was no mesos managed service from google, azure and aws. There was no docker swarm from google, azure and aws.
I'm usually fairly buzzword-adverse, and I didn't follow the NoSQL/Spark/Big-Data phase when it was hip. I was also hostile to kubernetes until I had to work with it against will for a project.
Since then, I completely changed my mind about kubernetes. This is a very good technology for only one reason -and it's NOT about container orchestration-. Portability. K8s is the missing piece that allows you to create a network of cooperating computers independently of hardware, OS, and even architecture.
If I have an home-made k8s cluster on my raspberry pies at home, it's not because it's lightweight and easy to manage -k8s adds a significant overhead-, it's because it gives me the ability to unplug one of them, take the sdcard, format it and plug another board without any interruption or configuration. I could plug my intel laptop to that network and have some pods running on it without having to change a single line in my configuration. Finally, I can zip a folder and email a bunch of yaml files to my friends (or have them git clone the repo), and they will be able to replicate an exact copy of my home cluster with all -or some- of the services. This is truly amazing.
This post should probably have (2019) in the title.
I'm the Nomad Team Lead and can try to answer any questions. Since this post was made the team has expanded, and the task/job restarting issue they link has been (mostly) addressed. Also new since this post is our Consul Connect integration which can accomplish similar goals to k8s network policies, albeit opt in and with the actual discovery/networking code living in Consul/Envoy respectively.
I feel like a lot of the problems K8s solves are fake problems that we have because all our software links/imports so many libraries/packages it's impossible to tell what's going on. It feels like have built an entire infrastructure around hopefully capturing that one time we got it working and being able to reboot to that state when things go wrong.
I am very happy working with Kubernetes in a very small team, I really like the abstraction of the different entities and the declarative ways to configure them.
I think that the learning curve is not that steep if you already had to do the same than with Kubernetes with other alternatives, in my on experience I have been discovering a lot of features that are very helpful not just during production but also during the development environments that were a real pain before.
I am have a lab/cluster/blog running on Kubernetes, I am in charge of this one alone, it is opensource [1], I version everything that goes to the cluster so you can see the evolution of the kubernetes entities, the config, the containers and the code. I started this from scratch and improving it feature to feature, I think that this might be a big factor with my positive experience with Kubernetes.
I wonder if an issue with adopting kubernetes is to try to migrate a big system into kubernetes in a very limited time lapse and trying to push/force features as they were working/handled in the previous approach?
A good friend and old coworker has been hitting me up for how to do stuff with docker since his job now uses containers. He has about 15 containers (on 2 hosts) that do not change to much and was asking about setting up k8s (it was a buzzword his manager heard). i talked him into just setting up swarm.
It took all of about an hour (over text messaging) to get it set up and all stacks/services running. They could not be happier.
It comes down to the right tool for the job. If you don't need all the bells and whistles, then keep it stupid simple. I realize swarm is not a 100% "enterprise" solution, however before they they were just issuing docker command after each reboot.
Yep, still doing plain old VMs.
Just like NoSQL, BigData,..., in a couple of years k8s will be slowly forgotten as everyone updates their CVs to the Next Big Thing™.
Agreed. k8s is too bloated at this point, yet alternatives like nomad and swarm are missing some fundamental features, so we have had to adopt to k8s unfortunately.
For example, swarm still have no fault tolerance and nomad relies on Vault, another product from Hashicorp and is also in the same limited state wrt. documentation
I think this post misses the point of where kubernetes brings value the most: multi-team environments. If you are a single-team, then nomad, non-distributed docker, systemd units, etc is probably a better choice today due to the operational complexity of running Kubernetes (GKE,AKS,EKS do lower the complexity, but you still need kubernetes experts on call)
However if you are in an organization with multiple teams (as are the vast majority of developers) then Kubernetes provides a common language for deploying, operating and securing your applications which enables you to go from a process which could takes days, weeks or even months to provision and configure a VM to minutes or hours to provision a kubernetes namespace.
It's funny we use GKE and coudn't be more happy, once your CI/CD is setup you don't have to think about it and makes your developper life much much easier.
I don't even think for one second to go back to uploading tar.gz or DEB / RPM using scripts / puppets ect ...
I wish this article was titled 'Nomad is pretty great' since that was really the point being made here. Nomad is not a true alternative to Kubernetes. It does not support autoscaling out of the box, it does not have most of the bells/whistles that Kubernetes has.
However, it is just a single executable and it just works. Combined with Consul and Fabio, you can run a container orchestration cluster with very little fuss that has service discovery, internal load balancing and cluster-wide logging.
It is a feature-rich task scheduler with a pretty good CLI. I highly recommend it, but if you need stuff like autoscaling, you need to use something else.
It's really odd that around 30% of the people who commented defending kubernetes are merely echoing stuff that the article itself mentioned.
Also, it's weird that the argument of another 30% of people defending kubernetes boils down to: "Using kubernetes is really easy, just hire Google/Azure/etc to do it for you."
Can't begin to wrap my head around that one.
But what do I know, I prefer to KISS and I like nomad. In fact, I'd be using swarm if its future wasn't spotty.
I agree with the sentiment, but I think the hardest part of kubernetes is dealing the complexity of the resulting cloud. You end up having to manage that complexity no matter what you're doing; the upfront cost of figuring out kubernetes is that maybe it takes a couple days longer to learn than something else, but now you have a huge ecosystem of technology and people that are solving the same problems you have.
docker compose and a jwilder proxy is all I need for most of my projects
I see Kubernetes also being used for ML workflows, yes...while you haven't figured out how to run a ML pipeline yet, add K8s (KubeFlow recommendation) As a ML researcher solutions like KubeFlow has been very painful as I don't want to learn k8s, I need to continue focusing on ML work and figure out my pipeline before I deploy K8s
There are lots of options out there. I'm currently exploring Cloud Foundry for our platform, since our core API is written in Spring Boot. This page from AquaSec has lots of links to articles comparing Cloud Foundry to Kubernetes. https://www.aquasec.com/wiki/display/containers/Kubernetes+v...
I also found this book to be helpful https://www.amazon.com/gp/product/B07T1Y2JRJ/ref=ppx_yo_dt_b...
I think most Nomad users are well aware that Kubernetes can do all it can - and vastly more.
If anything is wrong with Kubernetes, it would be the complexity of it and that it has a steep learning curve.
It seems it's best to have a small team of people to manage it and to solve solutions to problems that arise.
We started using Nomad last year and one thing I can say is that it's relatively easy to use and works well for its intended purpose, especially for us hybrid-cloud folks.
The part about all of this containerization technology I don't understand is that it was originally sold as a way (among other benefits) to unburden devs of having to think about ops stuff and just code.
Well, my experience with Docker, k8s, and related technologies, is that I now need to be an expert with these things just to get through my day. It's exhausting.
Before I started with k8s, I've been checking Nomad many times but eventually went with k8s. Nomad's learning curve didn't seem to be better, docs felt sparse and the much bigger ecosystem with k8s shouldn't be undervalued.
If you don't need container orchestration, you don't need kubernetes. If you need reliability and instead of using k8s, orchestrate your container manually you have issues.
Given all the talk here about Kubernetes’ learning curve, does anyone have any resources for learning it for those that haven’t yet? Whether blogs, books, video courses, etc.
What are you talking about, my blog that has an astounding 2 visitors a month (one or both of which are accidental clicks) definitely needs to be on kubernetes!
I built a big globally scalable system without Kubernetes or containers.
It’s very easy to understand and is highly reliable.
Personally, I've found Chef Habitat to be a sane alternative. I don't have any experience with k8s, but if you are not using containers especially, Habitat gets you:
- packaging
- deploys
- configuration management
- supervision
- service discovery
- isolation
in a more classic "VM" environment, using unix jails. For anyone who isn't interested in learning k8s, but also wants a relatively modern, all-in-one solution to these problems, I recommend checking it out.
Nomad is awesome. Just like the original author, our tiny team went with nomad for on-prem container orchestration instead of k8s. Since then we haven't spent much time on any nomad/consul maintenance, and all that time we got back - we spent on developing the product.
Basically, I agree with others that k8s is both complex and not fully mature yet.
Anecdotally, about the worst place I've seen Kubernetes used is for CI/CD pipelines where every pipeline gets a fresh cluster. It's truly a special kind of hell--it takes at least 10 minutes to get a cluster up and running (depending on cloud provider), and it causes all kinds of indeterminism and frustration. You wait for a cluster to be spun up, wait for all of the extra services to be added, have one of them fail (completely unrelated to your PR), and have to do the whole thing over again. It's truly the new version of the "compiling" XKCD.
If you ever want to see how brittle and bizarre Kubernetes can be, use it in a 100% automated fashion and hope for the best.
I heard the word Kubernetes one too many times from listening to podcasts put out by The Changelog. The overhype bit got flipped in my head and now I'm predisposed against it. Marketing kills!
I know it's an unpopular opinion but I don't see the point of kubernetes. It's a complexity monster which is not providing any real advantage over way simpler solutions.
Even pod to pod communication, which would be trivial to do using any sane solution is a huge pain in kubernetes.
I would actually be frightened by running kubernetes in production. The simplest things are so hard to do that I don't even want to think about how to fix a weird issue when something goes wrong...
Of course most software companies (ie, where most of HN work) don't need Kubernetes. At this point it's become a meme for wasted engineering.
Are you creating a cloud provider?
- If you are creating a cloud provider, then yes you might want Kubernetes. If you are making your own cloud provider, you should question why you are doing that.
- If you're not creating a cloud provider (for example, you're a software company), then use whatever VM / container / MicroVM / etc your cloud provider gives you rather than layering your own unnecessary complexity that adds questionable value on top of what your pay for from your cloud provider.
I get really sick of these types of posts because it helps to fuel a weird contrarian and/or fear based avoidance of these technologies. Then the company goes down a rabbit hole of inventing their own thing that basically is an in house version of kubernetes but worse. I think kubernetes is pretty easy, I'm not sure why companies are so against it other than wanting to be "even cooler than the cool kids".