▲ 58 points
back
86 comments
This reminded me of Operation Greif, a German covert mission during WW2.
https://en.m.wikipedia.org/wiki/Operation_Greif
...German soldiers, wearing captured British and U.S. Army uniforms and using captured Allied vehicles, were to cause confusion in the rear of the Allied line.
Reconnaissance patrols of three or four men were to reconnoiter on both sides of the Meuse river and also pass on bogus orders to any U.S. units they met, reverse road signs, remove minefield warnings, and cordon off roads with warnings of nonexistent mines.
As a result, U.S. troops began asking other soldiers questions that they felt only Americans would know the answers to in order to flush out the German infiltrators, which included naming certain states' capitals, sports and trivia questions related to the U.S., etc. This practice resulted in U.S. brigadier general Bruce Clarke being held at gunpoint for some time after he incorrectly said the Chicago Cubs were in the American League[7][8][9][10] and a captain spending a week in detention after he was caught wearing German boots. General Omar Bradley was repeatedly stopped in his staff car by checkpoint guards who seemed to enjoy asking him such questions.
Is there a programmatic equivalent of asking what league the Chicago Cubs are in?
There is an authentication system named after the concept, if that is what you mean: https://en.wikipedia.org/wiki/Shibboleth_Single_Sign-on_arch...
I am reminded of the questions asked by the original Leisure Suit Larry game in order to keep children from playing; the questions were designed to only be answerable by adults.
I'd you mean "question that sounds basic, but which many professional programmers wouldn't be able to answer at gunpoint": How do prefix and suffix increment operators (++x vs x++) differ in functionality?
"Is CloudWatch in Google Cloud or AWS?"
When used to filter out computers from humans such a procedure is known as a Turing test.
See also CAPTCHA
(a somewhat forced acronym of “Completely Automated Public Turing test to tell Computers and Humans Apart”)
The problem with most of these types of "attacks" is that they're about the same as any other messing with roadsigns or what not. There's no end of things you could do around the roads to cause crashes, but they're all illegal and ultimately low impact.
This is a lot more interesting though from how it highlights the difference between current machine interpretation and human interpretation of road signs: which is to say, explainable AI is still incredibly important - the ability to slightly discolor the signs and have a blackbox AI read something entirely different means the systems and trainers we have are not cueing off a "safe" set of inputs to deliver their interpretations.
From that context, this is very interesting and important work - and I would argue also points the way to at least some basic road safety standards for self-driving systems (they should be resistant to this sort of non-human perceptible visual distortion).
> There's no end of things you could do around the roads to cause crashes, but they're all illegal and ultimately low impact.
But most of them are easy to spot, at least in the aftermath. If you spayed over a sign, people would notice. If you removed one, a driver would probably not be able to directly notice, but he might notice the situation (i.e. you should not go fast in front of a school, even when the sign is missing) or you could find evidence if an accident happens.
The scary thing about these signs is that they could easily go unnoticed, especially if placed well (for example a crossroad with already sun-bleached signs). Neither a driver (who could possibly intervene) or an investigator would be able to spot this, unless they're specifically looking for it exactly.
Things like this are, in my opinion, also exactly why people put the bar of trust so high for autonomous vehicles: They might fail in ways totally opaque to us. You can at least somewhat estimate what a person might do [0]; for autonomous cars, this is hard or even impossible.
[0] I know someone could be extremely drunk or on drugs and possibly also mistake a stop sign like this, but you'd usually see some warning signs. An autonomous car might go from totally normal to batshit insane without any prior indication or apparent reason.
> from totally normal to batshit insane
Self driving cars don’t put that much emphasis on street signs, their not for example going to drive into a brick wall because of an arrow.
Alter speed limit signs in front of a speed trap might hit a lot of people but their unlikely to try and take a sharp turn at 70MPH or anything.
> Things like this are, in my opinion, also exactly why people put the bar of trust so high for autonomous vehicles: They might fail in ways totally opaque to us. You can at least somewhat estimate what a person might do [0]; for autonomous cars, this is hard or even impossible.
This is the subject of the story Car Wars by Cory Doctorow - https://web.archive.org/web/20170105065118/http://this.deaki...
> But most of them are easy to spot
I mean, walking upto a stop sign dangling a projector on a stick is pretty damn easy to notice.
It's also much easier to just hold up a bogus actual sign than to go through this mess, and it would be highly illegal as well.
I feel like there's a categorical difference between actually changing a road sign to say the wrong thing, versus adding some kind of smudge to it that certain computers read in an objectively wrong way.
The latter could easily be part of a dumb prank, "doing nothing wrong", but cause serious damage.
This piece of tape shouldn't kill anyone: https://www.extremetech.com/wp-content/uploads/2020/02/tesla...
I do wonder if there could be some kind of 'adversarial training' of large networks, like those of Tesla, in which drivers in an area collude to normalize dangerous behavior in an area (e.g. drive in oncoming lane, stop at an overpass on a major highway, etc).
Even with humans in the loop in the training process, there's only so much context they can bring to override data that appears authentic.
I'd me more worried about adversarial employees. Tesla has a reputation of working people to the bone for not much comp compared to the competition. I can't imagine they are connecting their stressed workers to psychologists. If one engineer just cracked one day, they could potentially push an update to all cars that does something terrible.
> ...and ultimately low impact.
A suboptimal metaphor in context.
chef's kiss
The problem is not that you can cause crashes, it’s that you can cause crashes at scale.
Realistic impact of such issues is usually quite exaggerated: https://xkcd.com/1958/
I think that any massively deployed true autopilot system will maintain a global semantic map of roads. Most sudden traffic sign changes, especially those which may influence car behavior, will be manually verified and committed to the global map. Any consistent discrepancies between the model and reality will be investigated as well, especially if it causes customer complaints. And it can be followed by releasing map patches telling software to prefer the model over detection for the problematic sign. Considering that at this stage such maps will be probably integrated with government services (i.e. the map will be compared against official road data), I think such attacks on the autopilot systems will be found quite fast.
How would that work where streets signs _do_ change suddenly? A couple of examples I see every day are roadworks (new fixed signage for the duration of the works, or even a person flipping a sign between "stop" and "slow") and "smart" freeways/signage (screen-based speed limit signs that change, and have additional signs for speed-limits if all screens are off). Here in AU, we also have "school zones" with speed limit signs that turn on only during certain times and dates.
I'm not sure that governments would be quick enough to keep this "central roadmap" correct up-to-the-minute (to be honest, I suspect that "up-to-the-month" would be a stretch!).
That comic is fun and all, but the specific examples are things people actually do to reduce traffic speed. Painting lines that turn a two way road into a single lane for a single car length is common for low speed roads in order to force cars to take turn passing it, thus reducing speed.
Cutout of pedestrians is extremely common, especially in the country area where they want to remind drivers to slow down next to homes. Sometimes they wear vests to look a bit like traffic police, through that practice is a bit (more?) illegal.
If I remember right, when that comic came out there was a HN thread discussing exactly those things. Those tricks are not done in order to murder people, but they might confuse a computer.
Right.
Also, if you want to mess with autonomous cars it's much easier to just buy this $15 T-shirt instead
https://www.amazon.com/Novelty-Signs-Halloween-Outfit-T-Shir...
Something I didn't find terribly clear: did the researchers manage to fool the software into seeing a shape other than an octagon, or is the software so bad it thinks an octagonal sign could be anything other than a stop sign?
the latter. The software doesn't think. That is why neural networks are dumb unless allied with expert systems. As a matter of fact that is more or less how human cognition works: the fusion of a pattern recognition system with multimodal sensory inputs filtered through an expert system looking for meaning in what is sensed.
See for example
https://slazebni.cs.illinois.edu/fall18/lec12_adversarial.pd...
> that is more or less how human cognition works
Yes, but pranksters frequently vandalize stop signs with messages like “stop driving” or “stop eating animals” and human drivers are smart enough to figure out what to do. I’ve never seen someone throw their car into park or toss their burger out the window upon seeing a confusing stop sign.
I have the same question. If it's the latter then systems like that cannot ever be allowed on roads.
In case you’re thinking you’ve seen this sort of research before, the difference is that they don’t touch the sign, just illuminate it.
I’m not sure how much a difference that makes in practice?
It makes a massive difference. To the human eye, the altered sign is not really distinguishable from the normal one and the attackers don't need to approach the sign in order to alter it. It makes a remote attack scenario quite achievable (depending on signal strength required for the laser etc). Think people camped out in a hotel room over looking a freeway and causing Telsas all to crash.
What about adding signed QR codes to road signs? It's practically impossible to forge those, it doesnt't require complex vision systems, and you don't have to revamp your whole road infrastucture, just gradually replace existing signs.
The reason for reading the existing traffic signs is that they already exist. If you would design roads for autonomous cars, you'd replace the signs with eg. radio beacons and the like.
They would have to be signed including GPS coordinates, otherwise they would be vulnerable to a replay attack. And since GPS is not that precise, you would still be able to, say, take two nearby signs and switch them.
> It's practically impossible to forge those,...
Forging isn't necessary, just copying one that is inappropriate in a particular setting.
Forgery isn't difficult either if the QR code isn't signed. QR codes are standardized and software to generate any QR code is reasonably straightforward to write (one has to know a bit about Reed-Solomon error correction), but writing software isn't necessary because there are a number of easily available libraries that can be used to generate QR codes.
Cryptographically signed QR codes (which I believe are what you are talking about) would need to include location data that could be verified by the vehicle, such as GPS coordinates and the sign's facing direction. This might work, but of course the logistics could be difficult in practice. Since signs aren't on the internet, it would be impossible to update them whenever a signing key was compromised. Key compromises would probably happen because signs are manufactured all over the world.
clever! and it reuses the existing (low) tech of painted signage.
I've never driven a tesla or any other car with "autopilot" features..
theoretically, if someone was to 'borrow' a 20km/h sign from somewhere, and take it to a 130km/h highway, and attach it to a random spot, would the car slow down, until the driver intervened? Or would it "guess" there's something wrong with such a drastic speed change, and beep/ignore/alert the driver?
Luckily the tesla autopilot ignores speed limits on the high-way, it only follows it (when on auto pilot) on slower roads.
There have however been issues with e-trons and id.X cars reading side street signs here in Norway and braking hard on the highway.
If you have a car with speed sign recognition, and activated cruise control, it would definitely slow down, albeit slowly.
Academically certainly interesting, but everything I've seen on adversarial attacks against ML models relies on knowing the model or having a way to run the model against attacker-determined input.
How would one go and attack a real-world vehicle? Essentially you'd need to gut a vehicle entirely, mock every sensor and actuator to fool the car's control unit to believe it is still driving a car... a lot of effort.
Dynamos for vehicle performance are common, and GPS spoofing is not hard. In a world where AI driving systems are a couple of decades old and on the used car market now, obtaining the source hardware would be very straight forward.
> everything I've seen on adversarial attacks against ML models relies on knowing the model or having a way to run the model against attacker-determined input
I thought that actually turned out not to be the case quite a bit of the time. An attack that's trained against one model will often work against different models as well, because the various models, even if trained on different data, will often end up looking at similar features.
I'm not an ML expert though, so maybe someone with more knowledge can chime in.
Or, you could just pull a page from intelligence agencies. All you would need to do is bribe or radicalize a disgruntled employee, then you'd know the model.
This title doesn't even begin to scratch the surface of what this is capable of. A much more viable (and damaging) attack would be encoding "invisible" data onto a widely-shared photo, and then manipulating it to trip the CSAM filters for the respective platforms. These attacks are designed to be invisible in real-world applications, but I can imagine a similar implementation designed to be invisible to humans (eg. encoding important data as two seemingly identical colors, or manipulating dither patterns).
Stuff like this is why we can't trust AI. That doesn't mean we can't use it, but most of our models are black-boxes that can only be "debugged" as much as a pizza can be "decooked"
Is a stop sign really the same shape and colour as a speed limit sign in the USA?
In the UK they are a different shape and colour, you don't need the text to know a STOP sign is what it is.
Is a stop sign really the same shape and colour as a speed limit sign in the USA?
No, but some of these sign reading algorithms probably don't take color and shape into account. A naive implementation might first checks the sign for something that can be OCRed, and if it finds that that is a 2 digit number that matches known speed limits then it's a speed limit.
Still stupid.
This following article wouldn't have happened if the vehicle was automated, it would have seen something was wrong, and also already have the correct signage from it's database -
3 Are Sentenced to 15 Years in Fatal Stop Sign Prank https://www.washingtonpost.com/archive/politics/1997/06/21/3...
What does need to stop is moving vehicles having uncovered street signs on them, if it's not in place cover it up.
As an aside, those defendants were eventually exonerated: https://www.law.umich.edu/special/exoneration/Pages/casedeta...