In the early days of my career I had to modify some code for a PLC that operated on a car production line. The modified code took too long to run so a watchdog process assumed the code had frozen and performed an emergency shutdown of the hydraulics of the line's welding robots. Six cars were damaged when the heavy robot arms crashed and buckled car roofs, and the one-car-every-45-seconds production line ground to a halt for 15 minutes.
back
4 comments
A similar PLC story: A friend was an intern at a well known coffee manufacturer. He was writing code for new sensors to use which were to be installed on the assembly line machines later that year. There was a staging machine that his team used in the same room, and they would log into the machine and push their code to run tests.
He sent the machine an update, rebooted it, yet the staging machine was acting unchanged from the previous version. Moments later a supervisor ran into the room, yelled to put his hands up from the keyboard, reached over him, ran some commands, and disappeared into another room. A few minutes later he announced that my friend logged into the main assembly line machines and rebooted them with code that used a sensor that didn't exist yet, which stopped the entire chain for ~20 minutes. The company suffered $250k production losses during that time.
How come an intern has access to the main production line PLCs?
This reminds me poor intern's story posted few days back on Reddit: https://www.reddit.com/r/cscareerquestions/comments/6ez8ag/a...
It must have been an error on the network's access restrictions, because I agree, he shouldn't have had the ability to get to that part of the network. Maybe it was a routing error, since he claims he didn't switch the machine destination before uploading.
Emergency shutdown meaning all the huge robot arms fall down, instead of just freezing in place, doesn't sound like a good idea to start with :)
Hydraulics are scary things there can be a lot of stored energy released very quickly if one of them gives out suddenly. I've heard horror stories about limb amputations etc due to sudden release of hydraulic pressure causing shrapnel to go flying across the room.
If there is a fault the safest thing for them to do is bleed out energy slowly unfortunately in this case sounds like this crushed the 'obstruction' in the process.
My Industrial plant code screwup story was not caused by me but was pretty impressive, what was supposed to be a "simple firewall change" knocked out communication between two interlinked parts of our plant which caused the line to stop and a big delay with a few million dollars lost. I believe the root cause was someone fat fingered the addition of a new firewall rule and we ended up dropping every incoming packet.
Cutting power = you know that the arms will go slack.
Freezing = you hope that whatever the problem is, maintaining power doesn't make it worse.
Ah yes, PLC programming. I expect there will be a lot of stories from PLC programmers in this thread ;)
"It is easy to make a mistake but if you really want to stuff it up you need a PLC". :)
Hey! I'm working at my first software internship and I will very soon be doing some PLC programming on moderately important chemical equipment. Do you have any tips on how to not spectacularly fuck everything up?
I found it helpful to not change "complex" outputs directly, but rather call a function that does that for me and handles the complexity in one place. For instance, if a brake must be removed before activating a motor, rather than pasting the same code every time you need to use the motor, wrap that in a small function block and use that.
In general, make the code easy to interpret, especially when debugging - which means organizing your code and using simple abstractions. State machines can be useful because they're easy to interpret if you comment your states, leading to high-level understandings like "the machine is waiting for an item" rather than "it will do something when I0.5 goes high".
That sounds pretty spectacular, alright. Although, I can't help wondering what it would have sounded like to set the robot arms to playing the William Tell Overture as a restart test. I mean the cars were already messed up...