▲ 259 points
back
168 comments
Part of my career encapsulates a period where I was a PLC programmer, installer, commissioner, and troubleshooter for massive build outs (sky scrapers, data centers, laboratories, factories). Interestingly, this was after years of teaching myself software engineering, eventually participating in large open source projects. The clash of entering the PLC world was _extremely_ harsh.
Let me give an example: I once worked with an integrator who was working on an AHU feeding an extremely critical portion of a datacenter (I was a lead by this point and mostly played babysitter). During certain points of the day you couldn't open the door to this room due to negative pressure because the logic was over-ramping the exhaust fans. As I watched this contractor work, I saw him open his laptop, with Windows on it (because Microsoft has had a death grip on this industry for decades now), and proceed to backup the PLC program into a massive folder with God knows how many other "customer projects" he was carrying around in this thing. He then proceeded to go do some physical checks in the field, came back, and prepared to upload the fixed program. As I watched, I noticed he _grabbed a backup from ANOTHER customer_ and I immediately had to intervene. Who knows what untold damage I saved from that single move.
I tell this story to demonstrate just how far into the dark ages this industry is. I vividly recall coming into the data center for a fortune 50 company, one everyone here would know, and being astonished that they had never heard of Network Attached Storage or RAID and why they might want to consider a disaster recovery plan for their multi-million dollar mechanical plant.
This industry is in _desperate_ need of strong technical help, but unfortunately the "higher ups" tend to be the same people who are "comfortable" with the way thing are and refuse to move. I literally tried for a decade before giving up and moving into software engineering proper.
So, anyways, just imagine the most archaic and barbaric set of IT software, controls, and procedures, dumb that down even further, and you've landed on the infrastructure/teams that operate probably half of critical infrastructure.
Another odd observation about the industry. For better or for worse, I inherited a little system controlled by a Fanuc PLC. It had a bug and would occasionally get stuck in a bad state. By some minor miracle, I managed to track down the person who had originally programmed it (now retired and moved out of state), and he emailed me a file and introduced me to his apprentice (who lives in a third state). I found a trial version of the design software, opened the file and found the bug (despite never having seen this style of programming before). Since I had no way to upload a new file (and since it wasn’t clear that uploading it without a matching outdated and not easily available copy of the programming environment would be wise), I convinced the apprentice to come visit while we was nearby for another client. He did not see the bug even when I pointed it out, but I convinced him to make the change and upload the modified file. Now the system works. (He had a Windows laptop with a giant folder of customer files, of course.)
While he was on site, I asked him why there was nothing on the PLC or its enclosure to identify his old boss or give any contact information, and he seemed surprised and told me that no one ever does that. I asked how a new owner is supposed to get support, and he shrugged.
If this thing ever fails, I’ll probably replace it with an Arduino or an ESP32 or something along those lines. Or I’ll just find something off the shelf to replace the entire system.
Everything you're describing mirrors my experience with remote scientific deployments and lab equipment (wet labs in particular). I support a team which manages dozens of these, and while it's shocking how inefficient and unsafe their practices are, it's similarly shocking how poorly the industry has kept up with or even tried to establish better practices. We get brand new equipment that costs in the realm of $10k for a tiny logger, and the thing is barely capable of remotely managed deployment of version controlled firmware. There are no tools in place to allow proper testing of the firmware unless you have the cash to buy extras and build test harnesses from them. The firmware itself is often in languages that don't have proper testing frameworks, type systems, IDEs, etc.
It's HARD to do the right thing. Dragging and dropping files into proprietary hardware management programs is the de facto standard.
Then you layer on top the unfortunate reality that sometimes electricity does weird stuff, people design weird circuits or wire the wrong components in, and firmware tends to have to deal with non-deterministic inputs a lot more often than, say, an API on the web. It's rough.
The pay is also so much worse in my experience.
Im supposed to be doing endpoint work with people working in this field amd basically have to convince compliance they'll need to be local administrators to do their job. They get onsite and dont know what app they'll need until they look at a plc model. They'll download something from a .tw ftp server and its an unsigned executable that absolutely must run as admin to program these things.
It is amazing how much of the sysadmin community just doesnt believe this is a thing you need to work with, everyone insisting its just security people being lazy and so on.
Oh I can tell stories too! Meanwhile these guys often have huge amounts of practical experience as electricians or control technicians/engineers to go with their PLC skills, so you can't just say we should get rid of them either.
Breaking into the industrial market is tricky if you don't have connections too. And if you're hired as the PLC programmer, it's sometimes an afterthought AFTER the plant is already built. "What do you mean it'll take another month? The plant is finished, isn't it?".
Oh, and some projects ban "PC"s to begin with. Which sort of excludes any kind of PC programmer. And it sort of even makes sense. A lot of default PC behaviors (especially commercial software), are no longer user-unfriendly but potentially very expensive or even user-lethal when attached to a physical plant.
Sounds like I could learn some things from you (and maybe vice versa). Poke me on the email in my HN profile!
A colleague of mine transitioned from a senior IC role to manager of a PLC group in the same company. Their methods were just as unsophisticated as you describe, and now he is attempting to evangelize/impose the finer parts of software engineering discipline, such as source control and integration tests (which can be trickier when it requires a hardware test bed.) Bringing some new tools to bear can be a fun job, as long as the team trusts you.
> So, anyways, just imagine the most archaic and barbaric set of IT software, controls, and procedures, dumb that down even further, and you've landed on the infrastructure/teams that operate probably half of critical infrastructure.
Same with SCADA: just as bad as what you describe.
Sounds like we have a similar development path. People fail to understand how complex some of these systems can be, especially since you have to have the project to understand what registers/variables are what. Some when Joe the contractor does a job and walks away, you better make sure they give you the project and you store it in a nice safe place.
A good system integrator is worth their weight in gold. Sure they cost more, but getting a whole package turned over to you is worth a million more than 5 years into the lifecycle of a factory when someone wants to make mods/fix a bug/etc and has to reinvent the whole car, not just the wheel.
This industry is always 10-20 years in the past. My company’s preferred vendor only just started supporting virtualization (this is for a DCS) in the past 10 years. I still have to tell my sales people to provide A/V and minimalistic backup and recovery on every project (they essentially cost nothing compared to the rest of any project).
The worst part -as I mentioned in another comment few days earlier- is most of the network is ancient, dialup, poor architecture designed by people who barely touched a computer, very old OSes, one utility had windows 3.1 a couple years ago! For them as long as it’s working, don’t touch it, because a downtime is far more costly and it will bring many eyes and attention to them than just sit and hope nothing will happen. Physical security is a major gap too, I have been to many locations where the field server panels are just behind the door, the RTUs are just closed with panel keys, even the whole facility are using some old HID cards that you can unlock in few seconds, this is even in R&D that beyond automation but also in robotic ones.
I was on the cybersecurity side of that, "advising" huge, multinational companies that hold the world together.
Oh boy.
I was told for instance once that my concerns about an open, unencrypted Wi-Fi for SCADA systems were not a big deal because "we are in the desert anyway, who would come here".
I saw terrible things and people who were made "cybersecurity expert" when they did not want to. They were sent to courses which were totally useless and the architecture was completely fucked up anyway.
I often wondered over the years why we have so little significant incidents in the utilities sector.
Yep, everything is lacking. No version control, ladder diagrams instead of written code, goto statements as primary control flow in code. A $500 PLC with less processing power than a 1985 086 and 1 whole MB of memory. Upgradable with a $200 license to 1.5MB, only programmable with a 20GB monstrosity of a piece of software in which download means copying a program.e to the PLC, and upload means downloading a backup to your laptop. That world needs some disruption really really bad
I agree. The problem is this: most companies using PLCs work in isolated networks with internet disconnected, based on confidentiality. There are many difficulties even with using git, because the structure forces you to implement features in a way that makes it hard to separate them. Branching, in particular, doesn't work well with PLC ladder logic—PLC uses proprietary binary formats that make diff tracking impossible, so you're forced into a unified development approach
> This industry is in _desperate_ need of strong technical help, but unfortunately the "higher ups" tend to be the same people who are "comfortable" with the way thing are and refuse to move. I literally tried for a decade before giving up and moving into software engineering proper.
As soon as AI can automate the development and deployment of physical infrastructure and the firmware that drives it, watch how willing to embrace the "cutting edge" those same higher-ups become.
May I ask: In which year was this?
> I vividly recall coming into the data center for a fortune 50 company, one everyone here would know, and being astonished that they had never heard of Network Attached Storage or RAID <
The thing is: Big companies, even non critical are usually bonded to some frameworks/regulatory standards/etc.
So, mentioning that they didnt know NAS or even RAID (which is very old, end of 80s), makes me wondering in which decade this happened?
Storytime: Once upon a time, I was a young grasshopper at a small consulting firm at which every employee was issued a Windows machine, our source control system was "\Customers" on C: drive, and our deployment strategy was "putty + copy-paste + hot-reload".
One morning, I heard something terrible while my boss's boss's machine was booting. Something akin to grinding. The thing was angry.
I gave boss^2 a heads-up that his hard drive might be failing, and that he might want to run a S.M.A.R.T. check on the poor thing. I also asked where the backup hard drives were, because I was brand-new and assumed that I just hadn't been issued one yet.
I checked the supply closet, found no hard drives, popped over to the office admin person, and recommended a deal I'd seen on some WD black drives before I realized that the place had gone quiet.
Everyone looked at me like I was from Mars.
Exactly 7 days later, boss^2's hard drive failed. We lost a week of work and had to zero out a 5 days x 3 employees worth of billable hours. We also ended up delivering late.
The client was pissed.
Based on the nasty looks that I got afterwards, it appeared that the standard assumption was that I had tampered with the drive to prove a point. (Um, nope).
I have since learned to ask prospective employers about their backup strategy.
I mean it worked fine for 50 years, surely it will work fine for another 50
There are many wireless pump-and-reservoir systems that while not internet connected, use insecure RF links. These local RF (and casting a wider net, Bluetooth) interfaces are also ripe for abuse.
If that means you need to at least be physically present then I'd say that's a lot of protection already. Means someone in a foreign country can't simply get lucky fuzzing.
Wouldn't the physical facilities themselves have security?
With coding agents now being used for hacking, there's a decent chance we'll see a 9/11-scale hacking incident as a result of NSA/DHS negligence in securing American internet-connected services. Similar to how the CIA's negligence allowed 9/11 itself.
The USG should be deploying thousands of security engineers armed with the latest coding models and agents, in attempt to secure systems before they're hacked. A few billion dollars spent here could save us trillions.
How is it NSA/DHS negligence? Neither is responsible for securing the infrastructure of state and local governments nor private companies.
They should provide guidance, but I’m not sure we really want the NSA inside of networks more than they already are.
If voters and CEOs don’t want to spend the money required to secure their infrastructure, that’s on them.
Unlike the CIA and/or FBI being in charge of preventing attacks like 9/11, NSA and DHS aren't in charge of each state's or city's infrastructure. They could run some opt-in program that local governments could then engage with, but "negligence" is a bit strong for systems they're not in charge of and have no access to.
https://www.nsa.gov/Cybersecurity/Cybersecurity-Advisories-G...
While that’s not the job of the NSA, they do produce a lot of good cybersecurity guides.
At modern population densities, basic infrastructure breaking on down on a large scale can kill millions in a matter of weeks.
The largest threat isn't bombs falling on our heads, it's incompetent fools leaving the door open to their enemies.
These aren't mistakes that can be excused. Failing in one's duty to steward important infrastructure must mean immediate replacement of leadership.
He is wrong and right. They should be connected to the Internet when they aren't 30 year old PLCs ripe for abuse. Until then, cut the data lines and do water monitoring the old way.
Disagree. Why connect them to the internet? They should be super hardened against attacks, and should NOT have a physical connection to the internet. Same with electrical infrastructure. Network access? Possibly, however that network should NOT be accessible from the internet.
The only exception I can think of would be for meter reading, which should be a separate, read only device with no ability to do harm altogether.
If it’s connected, it’s compromised. Or will be.
Folly to think otherwise.
Rather than "on/off" I think we need to distinguish between at least four things:
1. Connected naively to the internet.
2. Behind a hardened VPN endpoint which is on the internet.
3. Has a separate physical private network.
4. Requires physical access.
I think it's obvious that #1 should be prohibited in favor of #2. After that point we need to ask what the impact is of a Denial of Service attack that prevents anyone from remotely accessing the system.
The difference between #2 and #3 may depend on whether things could be Very Bad if the system is disconnected at a time of the attacker's choosing. For example, disabling access to flood-control valves during a hurricane.
A data diode can allow monitoring via the internet without risking ingress of control. Commercial units aren't cheap though.
Obviously we need open source designs.
Perhaps the easiest way would be a Raspberry pi set up with an opto isolated CGA/EGA/VGA/SVGA capture that could be viewed via the internet? (I mean, we're probably talking systems still running MS-DOS or Windows 98 running these systems)
Things that don’t have a compelling need to be connected to the internet should never be connected to the internet. When there is a need, connectivity should be restricted to the bare minimum required for the task.
Don't put your PLCs directly on the internet. In fact most industrial stuff is very not made to be directly connected to the internet. But interpose a firewall+VPN solution and you might be ok, if done competently.
And remote access to hardware definitely makes management and maintenance a lot easier and quicker. (else you need to drive out for every minor issue)
"If done competently" is a bold assumption unfortunately, not just these days but always
If we can beat a security-state airgap by sprinkling USB drives in the parking lot, I think the Iranians can beat an "internal" network by getting someone to click on an email attachment or visit the wrong website on their municipal issued Windows machine.
It's very common for the proprietary software for interfacing with ancient, expensive machines to break after OS upgrades, so they're probably unpatched... you might not even need to burn a 0-day.
Well yes this is true but air gapping isn't perfect security, it's important to understand that too.
And true air gapping isn't possible because it'll need to be monitored somewhere central so there'll have to be some vpn or mpls whatever. Meaning it can be hacked.
Having it exposed to the public internet is not good practice but should be far from the only layer in its security.
And really, when a PLC is found unfirewalled on the public internet, you can bet that's far from the only security screwup in that infrastructure. If they won't even handle the low hanging fruit.
We could say that about a lot of infrastructure that has been recklessly placed on the open Internet because it was cheaper than more secure solutions. I would say the same about home security systems, for instance.
No infrastructure should be on the open internet the potential for abuse is incredible. At a minimum a VPN should be used to tunnel all connections back to what ever command and control server exist, leaving equipment visible on the public IP should be a crime if not a felony.
what does belong on the internet in a post-mythos world?
one argument: only services which need to be available to unauthenticated endpoints should be default reachable.
all other services should be default unreachable (no data plane until authorized ...then use internet and other networks to establish the connections).
yes, that is not always easy. it is much more possible than it used to be.
and arguably we now need to commit to the tradeoffs of default unreachable services.
>Other countries start securing their water
>nsa: what no, stop that
I sometimes wonder how much damage (and potential damage) to US infrastructure exists simply because intelligence-agencies prioritize being able to exploit it globally over fixing it on defense.
Another vulnerability is natural gas pipeline compressor stations are all internet-connected, often via wireless because they are in remote locations. Simple DOS attack against one will force it to shutdown flow, causing things like power stations to drop offline.
I mean, this statement shouldn't be shocking to anyone. Critical infrastructure should be air-gapped if at all practical.
>Water system controllers don't belong on the internet, says ex-NSA chief
sounds like cope for a bunch of felons that management, its director and congress can't get a handle on.
Duh.
NSA chief also says water is wet.