Item — HN Reader

>>> If you're here, chances are you're already compromised.

WTF does that mean? Our salt implementation is on an entirely private network, so why would I be more likely than not to be compromised already?

----

Edit: Re-downvotes -- This is a sincere question. Is there some evidence that the majority of salt implementations are compromised, or some mechanism by which this hits private networks? Or is that line just for dramatic effect?

4 comments

I assumed that the problem was that if a master is accessible on your intranet, it could be hit with some sort of XSS attack from browsers inside the firewall.

But apparently there are 6000 people just straight up exposing their masters to the internet:

https://gbhackers.com/saltstack-salt/

> This whole "don't have your salt master exposed to the internet" thing has me annoyed. The whole point of salt is to manage boxes all over the place. I manage around 500 machines. Most of them are behind the firewalls of incompetent admins who have spent hours in the past trying to set up port forwards when salt-minion crashed so I could access the box again. I'm about to test binding salt-master to localhost and salt-minion to localhost and then setting up spiped to wrap the traffic...

Some companies need better DevOps apparently

Nitpick: this is purely an operations/sysadmin problem.

The DevOps grouping doesn't apply too much, Operations-minded staff should be focusing on keeping things locked down.

Unless, you know, you're not hiring those people and instead are hoping that developers take the Ops burden. ;)

People downvoting: I guess I hit a nerve, but could you explain why?

Development and Operations are different disciplines and the idea was to remove silos, not make one person responsible for both.

But that's like half the point of SaltStack. It's supposed be secure enough to run on the public internet to manage road-warrior endpoints.

"I found this site called Alexa that lists a bunch of companies that just straight up expose their web servers to the internet. Crazy."

And I have some bad news about how many companies are exposing their VPN servers to the internet too.

It's really not.

You also wouldn't expose your database just because it's password-protected?

I bet all these public servers have a firewall running to protect other services.

Shouldn't be hard to include the saltstack port(s) and whitelist the relevant IP addresses.

Also,the main difference with your example is that the clients that connect to webservers and vpns are either not known in advance or don't have static IP addresses.

You can't really compare Salt to a password protected database. SSH is probably a better comparison as both use public key crypto and plenty of servers are exposed on the internet via it. Its just that salt hasn't been audited as heavily as OpenSSH or other implementations of SSH so its risky exposing Salt to the internet, but I can see why some admins might do that.

Anyone who's putting their Salt master on the internet is having to deal with clients without fixed addresses. And it's not uncommon to have to deal with clients that aren't known advance which is when autosign scripts are used.

Sans the current CVE this is otherwise a service that is safe to expose to adversarial networks.

From Algolia's outage retrospective:

  What we did so far:
  We’ve secured the impacted SaltStack service by updating it and adding additional IP filtering, allowing only our servers to connect to it.

So clearly unrestricted access wasn't a necessity.

I understand it's a pain, I've been running a 1000+ server stack with puppet on a public network and relied on iptables to secure it. But I'd rather cope with the daily iptable rules update than having to fight a 0-day exploit...

I would (and do) expose my database to the internet, because it's properly secured (client certificates rather than just passwords).

I don't run firewalls. If a service doesn't need to be exposed, the port isn't open. Means no worrying about who has access to the wifi (because that's outside the security boundary) and no mucking about with VPNs when accessing remotely.

>because it's properly secured

That's what these Saltstack users thought too until this week.

People are saying they used a hand-rolled crypto stack rather than something standard like TLS? That should have been a massive red flag.

> road-warrior endpoints

would or could often have a vpn tunnel back to some static infrastructure, this and not the public network can be used for mgmt

Right, except that registering with the salt master is how your VPN is provisioned.

Including DigiCert? https://news.ycombinator.com/item?id=23062904

>>>> If you're here, chances are you're already compromised.

> WTF does that mean?

The website assumes that if you dropped by or googled the vulnerability, it's because you had saltstack exposed to the public internet.

You don't have to react so harshly to that turn of phrase.

My main impression was that this site was blurring the line between professional "Bulletin" and scare-mongering.

> Even if you didn't notice any unexpected symptoms, please: nuke and restart.

If I were writing this, I'd probably write something like "On May x, a remote-code-execution in (all versions?) public salt-masters (not minions?), was unveiled. Shortly thereafter, actual exploitation in the wild is being used. If your salt installation uses a salt-master, and it's internet-reachable, it may already be compromised, along with much your infrastructure. Section 2 is how to see if you are infected, and Section 3 is how to remove the infection."

> Section 2 is how to see if you are infected and Section 3 is how to remove the infection

That's much worse. That implies there is a reliable way to detect and remove the infection. That's not the case. This website included some known attacks and such. There's a high chance that there were attacks of this vulnerability with additional payloads.

There's no way to know if you were hit with a rootkit that persists itself in the bootloader or other parts of your system. There's no way to know if you were infected or not.

The only way to be sure is to nuke the machine, as they said.

Security advice should always err on the right side for a naive reader.

With you on this one... a coworker put it well "[this tone is] at a 9, it needs to be at a 3".

99% of companies I've encountered running SaltStack are masterless anyway.

Salt is a bit of a rarity among ops people but the funny thing is my experience is the opposite of yours - 66%+ of companies I see with Salt run in master/minion mode. I prefer it over Ansible when my ssh settings are super awkward (MFA, multiple bastions, etc) to integrate into an Ansible inventory file. One place with 900+ hosts and 300+ random IP ranges with tons of (badly written) compliance I spent about 3 weeks trying to get an inventory file hacked up and I gave up and deployed Salt within a day for the basic stuff I needed to do for basic running of shell commands from a single point of control.

Most of the places I've seen it are banks. Oddly.

Your use case makes absolute perfect sense though -- nice!

Depends on their use case.

IMO, Salts best feature is probably running with the Master / Minion setup because it connects out to the master. Masterless is handy for using in conjunction with Vagrant, Packer, or another provisioning tool not so much for managing a the lifecycle of a server / OS.

I'm not clueless. I know this. This doesn't detract at all from what I said.

In production, an outsized number of companies are using SaltStack a certain way.

The reason is because of the way we all evaluate business risk. I could have said that out of all of the companies I've worked with using Salt that have compliance requirements, 100% of them are using it Masterless, but then some smart-ass would have piped up with a "not me" comment.

How would you compare it to ansible in that usage model? I don't do much config mgmt these days, so curious for a master(less) mindset, whats the comparison for salt vs. ansible?

Not me

not me :D