Item — HN Reader

> As a disclaimer, I will add that this is a really stupid mistake and shows my lack of experience managing auto-scaling deployments.

There is a reason why these DevOps certifications exist in the first place and why it is a huge risk for a company to spend lots of time and money on training to learn such a complex tool like Kubernetes (Unless they are preparing for a certification). Perhaps it would be better to hire a consultant skilled in the field rather than using it blind and creating these mistakes later.

When mistakes like this occur and go unnoticed for a long time, it racks up and creates unnecessary costs which amount as much as $10k/month which depending on the company budget can be very expensive and can make or break a company.

Unless you know what you are doing, don't touch tools you don't understand.

7 comments

> Unless you know what you are doing, don't touch tools you don't understand.

I'm sorry but this is an absurd comment, and a fairly miserable reply to make to an engineer who is sharing a mistake in hopes of helping others to learn. We all touch tools we don't understand, and hopefully we learn something and come to understand them better. FWIW I have been running prod. workloads on GKE since 2016, and I think any engineer who is familiar with managing and deploying to cloud infrastructure can readily learn to run a GKE cluster.

I've been in this industry for over 15 years. I don't have a single certification. I learnt everything by reading and doing. With your method, I would still know nothing.

> When mistakes like this occur and go unnoticed for a long time, it racks up and creates unnecessary costs which amount as much as $10k/month which depending on the company budget can be very expensive and can make or break a company.

> Unless you know what you are doing, don't touch tools you don't understand.

I think it might be better to say "don't take risks you can't afford". You should experiment with new tools to see if they're better than what you're using now. Just don't deploy systems to prod before you really understand them.

> There is a reason why these DevOps certifications exist in the first place and why it is a huge risk for a company to spend lots of time and money on training to learn such a complex tool like Kubernetes (Unless they are preparing for a certification). Perhaps it would be better to either hired a consultant skilled in the field rather than using it blind and creating these mistakes later.

That's laughable but I will play:

I will pay anyone with a devops cert $0.01 for a right to 10% of my savings over a year period. If I end up paying more for the service after hiring such person, that person will pay me 110% of the excess that I paid for the service as a result of hiring them. If a devops cert is actually any good then this would be a license to print money for anyone with a devops cert.

OP's problem is that his organization did not engage in any sort of risk management which is why they had

a) K8s as something magical that makes things work

b) Someone who did not know how K8s works being allowed to re-engineers K8s

c) No alert on a change of the usage data exported by Google

P.S. If you are on a cloud, drop everyting and implement the (c). It will save your shirt donzens if not hundreds of times a year.

Nah, you've probably just got a fragile configuration that won't scale and will cost you money in downtime, lost sales, or failure to live up to contract.

An engineer who does it right isn't going to save you much money over your best case scenario - but they're going to keep you from losing millions in the worst case scenarios.

> Nah, you've probably just got a fragile configuration that won't scale and will cost you money in downtime, lost sales, or failure to live up to contract.

Those are the tales consultants and engineers that like to play with toys tell: it is a typical case for a premature optimization. The odds of you having enough traffic that needs to scale are slim to none.

If you do need to scale, the odds are your apps are over-engineered on corner cases and under-engineered in the main path: if your ORM takes 300 ms to initialize on every request without fetching any data from the database "scaling" is the last thing you should be worried about.

> An engineer who does it right isn't going to save you much money over your best case scenario - but they're going to keep you from losing millions in the worst case scenarios.

You will go out of business before those savings are going to matter.

Love that explanation! Part of why Facebook grow so big is, that it was basically never down!

I'll save you tons of money but we'll need to move to my data hosting center in my basement. I have many c64s networked together most with a 1581 drive.

I'll offer it for free so you can pay me immediately. I got my DevOp certificate in 1999 from BrainBench as a web master.

> Unless you know what you are doing, don't touch tools you don't understand.

That approach makes sense for businesses that need to be risk-averse, e.g. self-driving cars.

But in a "move fast/break things" operation, wasting some money might be preferable to taking the time to do things right the first time around.

>> There is a reason why these DevOps certifications exist in the first place...

Yes there is a reason, but you're not going to like the answer if I tell you what it is.

> Unless you know what you are doing, don't touch tools you don't understand.

But how do I know I’m an idiot?

Even if you are just a noob, how will you ever learn if you don't get in there and have a try? Very un hacker sentiment for hacker news!

Remember that not everyone here is a hacker. Some are specialized consultants who have vested interests in speaking out against DIY. (And that’s not to say that they are wrong)