▲ 93 points
back
76 comments
> Censys ARC identified 4,148 Internet-exposed hosts that respond to EtherNet/IP and self-identify as Rockwell Automation/Allen-Bradley. The United States remains dominant at 71.0% (2,945 hosts), with Canada a clear second at 11.5% (476 hosts).
Describe the network security of the industrial automation industry and their customers in a single statement. Lol.
The numbers are a bit lower if you exclude honeypots (~2k hosts): https://www.shodan.io/search/report?query=rockwell+port%3A44...
And the problem has actually gotten better over the years:
https://trends.shodan.io/search?query=tag%3Aics+rockwell
The situation used to be worse with things like the Lantronix password recovery service (i.e. a UDP port that would just send you the device password without any auth). It's still not ideal and takings things offline isn't easy (https://blog.shodan.io/taking-things-offline-is-hard/) but it's getting better (slowly).
And how many more are on the same internal networks as dozens or hundreds of ordinary Windows desktops on which municipal workers are checking their email? Hardly better.
It’s far worse, just last week I was assessing some architecture and there’s still dial up and 3G connected devices in some of the most critical infrastructure around..
I don't see the issue with either of those things? At least as long as they're properly secured. (Which they probably aren't but that's neither here nor there.)
Well I think it speaks to the age of the equipment they are speaking of in the industrial sector.
How do you lockdown something that may have not been taken offline for decades because it will cost downtime or harm. Or something that can’t be locked down without tossing new tech around it that may not be compatible with the protocols etc.
Dial up into an air gapped network defeats the purpose of being air gapped. I would think the bare minimum standard is that there is no way to change anything in the control systems without being physically present at the facility, past it's physical security boundary.
Is there reason to assume someone even tried to air gap it?
It is an issue, dial up lacks tunnel encryption and if you managed to make it, it will be useless in real scenarios, and 3g is being phased out and obsolete
Secure the contents of the tunnel and it doesn't matter. Provide a secure backbone and it doesn't matter. Realize that dialup is so slow that you can transparently tunnel it across ~any modern network (for which you can encrypt the tunnel) and it doesn't matter.
Tunnel your dialup within your obsolete 3g network, and then tunnel that obsolete 3g network within something modern. The obsolete technologies are not the issue here.
Also the thing about dialup is that it's point to point so the attack surface isn't even remotely comparable to exposing a port on the open internet. I should generally be able to trust the link that my phone company provides. Faxes are still used in many secure settings in preference to email.
If.
Used to drive me mad that the clipper terminals in the SF Caltrain station used dial up, in 2017!
https://www.linkedin.com/pulse/end-complacency-i-can-hope-an...
I much prefer the non-vendor perspective on this. Andy Krapf, co-chair of the Water ISAC, has a great breakdown about the status quo systemic problems that water faces today.
The playbook, per federal officials, was crude but effective: attackers remotely accessed internet-facing operational technology, changed device IP addresses and passwords, and locked utility operators out of their own monitoring and control systems, NBC News reported. The PSA is now pleading with utilities to do the bare minimum - pull programmable logic controllers off the open internet and put them behind gateways and firewalls, use actual passwords, and restrict which devices are allowed to talk to one another.
There can be surprisingly few critical components and processes in a water and sewerage network due to gravity-fed designs (far more cost efficient), large buffers (water reservoirs, sewage sumps, etc) allowing intermittent operation of otherwise critical components, retained ability for humans to manually operate equipment, and also the fields of availability and safety engineering which typically prefer elimination of software failure modes by designing equipment to not rely upon software.
The aim of a water network is to:
1. Take water from a water source (elevated dam -- strongly preferred, river, ocean) and as much as possible, gravity feed it to a treatment plant.
2. Treat the water using processes that are simplified/fail-safe as much as possible and could be operated manually by humans if necessary. This is where availability and safety engineers would design equipment to not be dependent on software and instead use mechanical or analogue electronics control.
3. Pre-position treated water as much as possible at ~50-90m hydraulic head (~500-900kPA) above the water faucets where people want to use the treated water and provide a buffer for X days of usage. Any pumps between the treatment plant and elevated storage therefore only need to operate intermittently to refill the buffer.
Sewerage networks have similar aims:
1. Let sewage flow as much as possible downhill to the treatment plant via gravity. Where a rising main (elevation gain) is required, place a large enough sump for X hours/days of usage and pump up to higher elevation from the sump.
2. Treat the sewage using processes that are simplified/fail-safe as much as possible and could be operated manually by humans if necessary. For example, a compressor used for aeration can be manually switched on/off with a mechanical switch and plugged into a diesel generator, and not require someone logging in with multi-factor authentication to a laptop to issue a command to a PLC to turn on the compressor.
3. Design overflows into the system for emergency release of partially or untreated sewage, and practice this process as part of disaster recovery exercises. This is generally an aim arising due to risk assessment process that says building a $1bn sump with 8 independent pumps is cost prohibitive versus the 1-in-200 year chance of untreated sewage messing up a downstream river for a few weeks.
Ultimately a lot of the cybersecurity risk comes down to government appetite to accept 1-in-1000 (or whatever) year failure modes. Is it worth investing now in triple modular redundant automated control systems (mostly used in safety-critical sectors such as aviation and space), or installing a just-in-case diesel generator at every one of 500 pumping stations across a region, or building $10bn of sewage sumps to hold sewage for up to a month, or building 2 treatment plants instead of 1 and using different technology for each, or hiring and training more humans to regularly exercise manual control and operation of a network, etc? Or just accept that once every 1000 years, some water rationing may be required, or a downstream river will be polluted for a few weeks?
Sadly this instantly became a political football, with the states pointing fingers at Iran, but Trump was not wrong in this case. This is gross incompetence at all levels — IT malpractice if you will.
CISA and its predecessors have been warning utility operators about critical infrastructure vulnerabilities for what, 15 years at this point? That goes back to the first Obama administration.
Yet here we are in 2026 and these utilities are still connecting these things to the raw Internet with default passwords. You cross a threshold where you're being deliberately careless.
When you are putting more effort into securing your Plex server on your home network then public utilities are taking on machinery that dumps chemicals into the local water, something is not right and finger-pointing isn't going to fix it.
I think it's less carelessness and more the inability to attract (pay) people who have the technical knowhow to properly secure infrastructure. Even a lot of developers are poor network engineers and treat IT like magic at their own companies.
You’re on the right track, I think. But, I wouldn’t say it’s about the pay to attract competent workers. I think it has more to do with the incentive structures once you’re in. Incentives and performance management are fundamental problem in civil service. The incentives to set high standards and hold individuals accountable simply do not exist.
The one and only exception is the military, because lives are literally on the line in a way that is not at all abstract.
Usually when people say this they are dog whistling privatization. Which is the exact opposite thing people need in infrastructure, ask anyone who has to deal with PG&E.
Paying skilled people highly does actually incentivize people to do better work, especially if they are actually embedded into the community they are essentially working for. If being a civil servant was as "glorious" as being a techie is SF there would be a very different attitude around the work.
In an environment without accountability, higher pay is just more incentive to lay low and not take any personal risk. Why do anything other than the bare minimum when there is no upside? In a bureaucracy where responsibility is diffused and the culture is purely political and not merit/performance-based, few are willing to step out of line to do the right thing.
The people who climb to the top aren’t the ones who took a risk and got the reward. Quite the opposite, they’re the ones who learned to play the game and didn’t upset the power structure by rocking the boat. No one is going to tell the emperor he has no clothes when the path to power is political and has no grounding in reality.
Highly conscientious, intrinsically motivated people will do the right thing in any environment. And those people don’t last long in political/bureaucratic environments where the incentives are misaligned.
Why do you assume that PG&E has a good incentive structure, or that the public sector must necessarily have a bad one? My only objection to the preceding comment would be that struggling with perverse incentives isn't limited to civil service. Incentive structures are a core struggle of approximately all large groups of people.
I think the big problem is administrative capacity.
There are better run governments than we have in the US.
The contempt for the state is a self-fulfilling prophecy. The state is incompetent because many of us believe it is inevitable that it will be. Compensation is just a part of it; coherent administration with continuity is even more important.
We just started replacing our PLCs. They absolutely were setup with default passwords, but weren't put on the public internet.
What policies do y’all have in place for this? Is there a program in place or the beginnings of one at least?
I've met info-sec / vulnerability researcher types that were egregiously reckless, like plugging Raspberry Pi's into the production network kind of thing.
Public sector has always paid low. But the problem is widespread, almost universal, and they've had a 15 year head start of the federal government telling them to get their shit together.
At some point it just became standard industry practice is my guess.
Upgrades to waste water are project based. Lowest bidder will not provide security for free. Security may be mentioned in spec but in hand waved language that can be hand waved away. That company doing the improvement project will have next to no documentation from the previous engineering effort. Just do bare minimum and move on to next job, because no one is getting paid enough to do put in more effort.
In all things when it fails in a drastic way the system will be corrected. People will die, it will suck, changes will be made.
Government is supposed to respond to these things and create incentives to correct. Telling a small municipality to do something without a carrot ir stick does nothing.
In this instance someone else will be providing the stick.
Funny enough I used to work for a water system in the “Information Systems” department - there was far too much of the classic “not my problem” from colleagues.
Our original bill pay (ran from 2006 to 2019) stored passwords in plaintext in flat files. Concerned citizens noticed because our password reset would just email you your own password. Instead of fixing it, they just removed the ability to recover an account without coming into an office. Our CTO knew, he wrote the whole thing!
We had a fun one, Outlook was sending employee passwords to our bill pay system and ending up plaintext in our logs due to some quirky fallback default behaviors around DNS and VPNs. Reported and ignored, of course.
Regarding TFA, we had an insurance requirement to properly air gap our PLCs - which we didn’t do. We (the CTO) just put them on a separate subnet and lied to insurance.
There’s not much you can do to an organization that has no real oversight here, especially once the “coast to retirement” types infest the place. We need something like HIPPA or PCI-DSS with real auditors and real teeth for utilities.
> Yet here we are in 2026 and these utilities are still connecting these things to the raw Internet with default passwords.
I work with PLCs. Default passwords of not, the idea that such weakly secure devices are being made accessible from the public internet boggles my mind.
Yeah I meant the default passwords are simply the cherry on top of already egregiously poor security.
What industry? Very relevant.
Not utilities. And none of the PLCs I work with are internet-connected or unsecured.
Absolutely 100% spot on. It’s not a political issue, it’s a technical issue. Disconnect them from the internet. Run your security patches. Check your logs. Water supplies are pretty important, do your job.
Kind of feels like national security is the job of the federal government. Seems fair to say the federal government should do their job. They started a war for no reason and failed to anticipate not only these infrastructure breach but also the closure of the Hormuz strait.
So the federal government should be responsible for every rinky-dink water well in Bumblefuck, Minnesota?
> failed to anticipate not only these infrastructure breach
They've been warning them for close to two decades.
Minnesota chose the path of no locks on their front doors and are now crying that someone walked in without knocking first.
I will repeat a comment from below. There are over 150k water utilities alone in the US.
Passing the buck to the Federal Government is not understanding the problem.
“Run your security patches” is easier said than done in the case of OT and it’s actually an issue that is further upstream than this. Policies, procedures, culture, and resources to execute. None of which are technical.
Not really. Just patch and reboot. Pretty simple.
Not IT malpractice and this where the industry diverges. IT folks usually don’t work on or understand these systems.
Which is one of MANY problems OT faces. IT best practices don’t suffice in OT and even when they do, most of these orgs are too resource hamstrung to do anything about all of the fires they have to put out.
Not to mention all of the OT vendors who flooded the market with tools instead of people being taught the boring process driven work.
Yes, utilities shouldn’t be negligent, but national security is 100% the federal government’s responsibility. If the vulnerabilities were so trivial, then it’s even more damning that the federal government was caught with its pants down, particularly since they were the only ones who knew they would be starting a war.
> finger-pointing isn't going to fix it.
Your entire comment was finger pointing…
There are over 150k water utilities alone in the US.
Passing the buck to the Federal Government is not understanding the problem.
There are 1000x as many tax payers and the government still makes sure every single mom who gets a venmo payment for $60 pays taxes. National security is a federal responsibility, they should absolutely do their jobs.
Until the people in charge face jailtime for hurting innocent people, why would they care? The government shouldn’t be warning, it should be ordering and imprisoning. And funding and educating where there are genuine gaps.
CISA was formed in 2018, so not quite 15 years but closer to half that. The security industry as a whole has been yelling for longer than 15 years about the vulnerability of utilities. They've been marked as soft targets before the Bush administration restructured the government.
a broken clock, yada yada.
"I blame it on Minnesota because they are grossly incompetent."
"I think Minnesota is behind it."
The first quote makes it the state's responsibility to secure local water systems, which I'm not sure that it is. The second makes it at least sound like the state of Minnesota is the entity running the attack on local water systems within their state, which is off in paranoid conspiracy territory.
Trump was absolutely wrong.
Some people allegedly say that the US should treat heads of companies like they do in China, when found guilty of certain crimes. Allegedly.