▲ 266 points
back
217 comments
I think we’re just getting started, with fake images and videos.
I suspect that people will be killed, because of outrage over fake stuff. Before the Ukraine invasion, some of the folks in Donbas made a fake bomb, complete with corpses from a morgue (with autopsy scars)[0]. That didn’t require any AI at all.
We can expect videos of unpopular minorities, doing horrible things, politicians saying stuff they never said, and evidence submitted to trial, that was completely made from whole cloth.
It’s gonna suck.
[0] https://www.bellingcat.com/news/2022/02/28/exploiting-cadave...
> We can expect videos of unpopular minorities, doing horrible things
Expect? You can post a random image of an unpopular minority, add some caption saying they did horrible things, that is not reflected in the image at all, and tons of people will pile on. Don’t even need a fake video.
So far, I see the most concern about this sort of thing from people who came of age around or after Web 2.0 hit, at a time when even a good photoshop wasn’t too hard to place as fake.
Those I know who lived through this issue when digital editing really became cheap seem to be more sanguine about it, while the younger generation on the opposite side side is some combination “whatever” or frustrated but accept that yet another of countless weird things has invade a reality that was never quite right to begin with.
The folks in between, I’d say about the 20 years from age 20 to 40, are the most annoyed though. The eye of the storm on the way to proving that cyberpunk lacked the required imagination to properly calibrate our sense of when things were going to really get insane.
Oh heck yes. One India focused study that I saw, introduced me to the term Cheap Fakes. Another report studied how genAI made phishing pipelines more efficient, allowing profitable targeting of groups who hitherto were too poor to be targetted.
So on one end you have large scale pollution of the information commons, and on the other end we are now creating predator pipelines to generate content with all the efficiency of our vaunted AI productivity. Its creating a dark forest for normal people to navigate, driving more government efforts to bring control. This in turn puts this in conflict with freedom of speech and expression while dovetailing nicely with authoritarian tendencies.
Yes, Its heartening to hear all the people who find productivity gains from AI, but in totality it feels like we got our wishes granted by the Evil Genie.
People were able to make very realistic fakes of anything 10-20 years ago, using basic tools. Just ask the UFO nuts or the NSFW media enthusiasts. And like what you mentioned, staged scenes have become somewhat common as well, including before the internet.
We can expect more of the same. Random unverified photo and video should not be trusted, not in 2005, not in 2015, and not today.
I believe that this "everything was fine but it's going to get really bad" narrative is just yet another attempt at regulatory capture, to outlaw open-source AI. This entire fake bridge collapse might very well be a false flag to scare senile regulators.
> We can expect videos of unpopular minorities, doing horrible things
While manipulation of photos exist, and real photos misattributed are very common, for the most part a lot of that does happen as well. And some people are too quick to ignore or gloss over it
>We can expect videos of unpopular minorities, doing horrible things, politicians saying stuff they never said, and evidence submitted to trial, that was completely made from whole cloth.
AI videos of unpopular minorities already comprise an entire genre and AI political misinformation is already mainstream. I'm pretty sure every video of Donald Trump released by the WH is AI generated, to make him look less senile and frail than he really is. We're already there.
What this incident really shows is the growing gap between how easy it is to create a convincing warning and how costly it is to verify what's actually happening. Hoaxes aren't new, but generative tools make fabrication almost free and massively increase the volume.
The rail operator didn't do anything wrong. After an earthquake and a realistic-looking image, the only responsible action is to treat it as potentially real and inspect the track.
This wasn't catastrophic, but it's a preview of a world where a single person can cheaply trigger high-cost responses. The systems we build will have to adapt, not by ignoring social media reports, but by developing faster, more resilient ways to distinguish signal from noise.
Would calling and saying, "Hey, the bridge is destroyed!" without an image not have also triggered a delay? I question the safety standards of the railway if they would just ignore such a call after an earthquake. Generative AI doesn't change the situation at all. An image shouldn't be treated as carrying more weight than a statement, but the statement without the image would be the same in this situation. This has really been an issue since the popularization of the telephone, which made it sufficiently easy to communicate a lie from far away that someone might choose to do so for fun.
Given the number of cctv cameras that operate in the UK, and their continued growth, I am surprised that the rail operator did not have access to a direct view of the bridge. I am also a bit surprised that there isn't technology to detect rail damage, especially the power lines that runs over the track.
Where I live it is not uncommon for rail to have detection for people walking on the rail, and bridges to have extra protection against jumpers. I wouldn't be that surprised if the same system can be used to verify damage.
Not a hope.
Most economic value arises from distinguishing signal from noise. All of science is distinguishing signal from noise.
Its valuable, because it is hard. It is also slow - the only way to verify something is often to have reports from someone who IS there.
The conflict arises not from verifying the easy things - searching under the illumination of street lights. Its verifying if you have a weird disease, or if people are alive in a disaster, or what is actually going on in a distant zone.
Verification is laborious. In essence, the universe is not going to open up its secrets to us, unless the effort is put in.
Content generation on the other hand, is story telling. It serves other utility functions to consumers - fulfilling emotional needs for example.
As the ratio of content to information keeps growing, or the ratio of content to verification capacity grows - we will grow increasingly overwhelmed by the situation.
I also think not a hope, check Brandolini's law[1]: The amount of energy needed to refute bullshit is an order of magnitude bigger than that needed to produce it.
Great comment and very true in this AI world. In 2030 it will be even easier to make even more realistic images much quicker...
Reminds me of the attacker vs defender dilemma in cybersecurity - attackers just need one attack to succeed while a defender must spend resources considering and defending against all the different possibilities.
It is cheap to have live monitoring of key infrastructure these days, and in the case of rail infrastructure it would also save time and money in general. Perhaps this hoax will push this higher up the todo list.
You don't need AI for this kind of disruption. People have been making fake bomb threats for years. You just have to say it, either directly to the railway/etc. or publicly enough that somebody else will believe it and forward it to them. The difference might be of intent - if you say you planted a bomb on the bridge, you're probably committing a crime, but if you just post a piece of art without context, it's more plausibly deniable.
It's also pretty common in the UK for trains to be delayed just because some passenger accidentally left their bag on the platform. Not even any malicious intent. I was on a train that stopped in a tunnel for that reason once. They're just very vulnerable to any hint of danger.
an AI sees the image on social media, deploys a drone to quickly go there, looks at the live video feed, and declares all is good
Sir, this is AI prose. Wendy's doesn't allow AI prose.
From 1950 - 2005(ish) there were a small number of sources due to the enormous moat required to become a broadcaster. From 2005 to 2021, you could mostly trust video as the costs of casual fakery were prohibitive. Now that the cost to produce fake videos are near zero, I suspect we will return to a much smaller number of sources (though not as small as in the pre YouTube era).
Some of the “smaller sources” also distorted facts.
We might even have fewer than before - between Internet commentators and loss of confidence from AI, real journalism may not be as highly valued as it was before the Internet…
There will be people who care about trusted and reliably accurate news sources, and at least some of them are willing to pay for it. Think 404 Media.
But there are people who don't want their news to be "reliably accurate", but who watch/read news to have their own opinions and prejudices validated no matter how misinformed they are. Think Fox News.
But there are way way more people who only consume "news" on algorithmically tweaked social media platforms, where driving "engagement" is the only metric that matters, and "truth" or "accuracy" is not just lower priorities but are completely irrelevant to the platform owners and hence their algorithms. Fake ragebait drives engagement which drives advertising profits.
We're also seeing a barrage of commercials featuring AI generated animals talking like people. It's getting old.
> Network Rail said the railway line was fully reopened at around 02:00 GMT and it has urged people to "think about the serious impact it could have" before creating or sharing hoax images.
Perhaps Network Rail should have a system of asserting rail integrity that is independent of social media (?!!?)
for real, pick up the phone and ask someone (??)
To be clear, you don't need AI for this.
You can also just call the railroad and report the bridge as damaged.
Hoaxes and pranks and fake threats have been around forever.
Amazingly, no one seems to have actually checked that this picture was really "circulating on social media". I've been investigating for the past hour or so and can't locate a single public post or reference anywhere other than reposts of the BBC article.
Typically, postings that gain traction have many many reposts and though some may be deleted, there's a long tail of reverberation left behind. I can't find that at all here.
I wonder if the hoaxer just emailed it to Network Rail directly?
Reminds me of https://en.wikipedia.org/wiki/Fall;_or,_Dodge_in_Hell with the Moab plot point.
> A BBC journalist ran the image through an AI chatbot which identified key spots that may have been manipulated.
The image is likely AI generated in this case, but this does not seem like the best strategy for finding out if an image is AI generated.
In a extreme safety-first organisation — and the GB railway is exactly that — it's easy to exploit that weakness.
It actually had very minimal impact. An hour or two wasn't bad for an organisation which stripped staff to a bare minimum, and for the area.
And it's very much the customer's job to work for the railway these days: it's our job to report police matters we are told incessantly with announcements. It's our job to buy the right ticket as there are very few ticket staff and staff with any knowledge these days. It's our job to use third party websites during disruption and to Tweet the railway company for assistance because again there is not enough staff.
So Network Rail is not going to come out and say "it's absolutely our job to be aware of all our infrastructure at all times and our defence to this new threat is to bolster staff and CCTV and reduce our reliance on third party reports"
> Network Rail said the railway line was fully reopened at around 02:00 GMT and it has urged people to "think about the serious impact it could have" before creating or sharing hoax images.
> "The disruption caused by the creation and sharing of hoax images and videos like this creates a completely unnecessary delay to passengers at a cost to the taxpayer," a spokesperson said.
I don't think this will work the way they think it will work. In fact, I think they just proved they're vulnerable to a type of attack that causes disruption and completely unnecessary delay to passengers at a cost to the taxpayer
You don't need AI to make these hoaxes, pranks have been around forever etc etc.... but as with alot of the areas AI touches, the problem isn't the tools or use of them exactly, it's the scale. In this case the low barrier to creating the fake media coupled with the pervasiveness of social media networks and their reach (also networks that aren't new), affording the rapid deployment and significant impact by bad actors.
The problem is the scale. The scale of impact is immense and we're not ready to handle it.
> "It's more the fact that Network Rail will have had to mobilise a team to go and check the bridge which could impact their work for days."
Im way more concerned of this statement than whatever is reported in the title.
How fragile is a society that is unable to make a simple visual confirmation of a statement without having a multiday multi-££ impact?
Freight transport is cost-effective in terms of delays, approx. 2mn per minute in the three Pacific Railroad Surveys for a trans-contiental railroad by the War Department circa. 1850.
Much of the world relies on general well-behavedness. The whole Andon principle doesn’t work if you’ve got asshole employees. With the public you don’t have a choice. You have to stop the trains because otherwise everyone will murder you if it turned out to be true. So better to be defensive.
How hard could it be — genuine question — for, say, Apple (Nikon, Sony,…) to embed a QR code (optionally) into an image.
QR leads you to a page, you upload image to page, hashes are compared, image-from-sensor confirmed.
Surely at this point we need provable ‘photography’ for the mass market.
Cannot image how often this will happen after we are buried under fake contents from AI.
>"It's more the fact that Network Rail will have had to mobilise a team to go and check the bridge which could impact their work for days."
It is no surprise to me that Network rail are so understaffed that any special event disrupts their work schedules for days. That is what they call 'efficiency' these days.
Edit: Aside. During a set of fire service strikes it was a relatively common opinion to say something like, 'of course they have an easy job, they get paid to just sit/lie down at the station'. I used to ask, 'what would you like them to do while waiting in case you need rescuing?'. No answer. I spoke to a fireman and he told me that in response to this kind of nonsense a bunch of pointless busy work was invented for them. When real was privatised in the UK they fired a lot of these 'inefficient' workers. After a string of rain crashes, the government had to renationalise Network Rail (the bit that maintains the infrastructure). Another case where 'efficiency' means harming people for profit.
It's a bit of a non story, even with the fake image.
From the article:
Trains were halted after a suspected AI-generated picture that seemed to show major damage to a bridge appeared on social media following an earthquake.
Railway expert Tony Miles said due to the timing of the incident, very few passengers will have been impacted by the hoax as the services passing through at that time were primarily freight and sleeper trains.
"They generally go slow so as not to disturb the passengers trying to sleep - this means they have a bit of leeway to go faster and make up time if they encounter a delay," he said.
"It's more the fact that Network Rail will have had to mobilise a team to go and check the bridge which could impact their work for days."
A fake image of a stone bridge with fallen parapets prompts the same response as a phone call about a fallen stone from a bridge or (ideally !!) just the earthquake itself - send out a hi-railer for a track inspection.
The larger story here (be it the UK, the US, or AU) is track inspections .. manned or unmanned?
Currently on HN: Railroads will be allowed to reduce inspections and rely more on technology (US) https://news.ycombinator.com/item?id=46177550
https://apnews.com/article/automated-railroad-track-inspecti...
on the decision to veer toward unmanned inspections that rely upon lidar, gauge measures, crack vibration sensing etc.
Personally I veer toward manned patrols with state of the art instrumentation - for the rail I'm familiar with there are things that can happen with ballast that are best picked up by a human, for now.
The issue is provenance. We need cameras and phones to digitally sign photos so we can easily verify an unadulterated image.
You also want to be able chain signing so that for example a news reporter could take a photo, then the news outlet could attest its authenticity by adding their signature on top.
Same principle could be applied to video and text.
I mean what does anyone expect from a future where images like this can be generated by any moron for any place, in thousands of variations, with just a few clicks? And then video?
I am surprised headlines like this are only coming out now. I've been saying it for a long time, but people said i am crazy. The web as we know it will be unusable. And a new one will not solve all issues, as we have already made ourselves too dependent on the current web and tech. So the impact on the real world is gonna turn a lot if things upside down. It's gonna be a lot of fun. But sure, let's keep pretending AI can either be nothing but bullshit OR we should only fear losing jobs to robots... i don't get why no one every thinks about the societal impact... it's so obvious, still... i am baffled...
Don’t trust, only verify.
Remember Moab
Remember Moab
Another way that AI is literally destroying the Internet.
Despite using Claude Code almost daily and finding it a useful tool, on balance I think that AI is a net negative to society.
LLM AI has led to job losses (either indirectly for moving investments into AI instead of people) or directly. Generative imagery has and will lead to more bullshit election outcomes, people getting blackmailed, scamed, things like this train stoppage etc etc. The list is endless. Not even getting into how the AI bubble burst will make most of us poor when the huge stock market crash comes, but hey whatever...
What good has it brought us (not the billionaire owners of AI)? It made us 'more effective' and oh instead of googling something and actually going to a link reading in detail the result we can now not bother with any of that and just believe whatever the LLM outputs (hallucinations be damned).
So I guess that's an upside.
(before the AI god bros come: I am talking purely about LLMs and generative imagery and videos, not ML or AI used for research et al)
Yet another attack vector for the Russians.
https://en.wikipedia.org/wiki/Russian_sabotage_operations_in...
See e.g. https://www.polskieradio.pl/395/7785/artykul/2508878,russian... (2020)
> Almost 700 schools throughout Poland were in May last year targeted by hoax bomb threats during key exams, private Polish radio broadcaster RMF FM reported.
> It cited Polish investigators it did not name as saying that a detailed analysis of internet connections and a thorough examination of the content of emails with false bomb threats turned up ties to servers in the Russian city of St. Petersburg.
"think about the serious impact it could have"
They do ... that's why sociopaths do such things.
The BBC says the hoaxer should consider the effect on other people. Should Sir Keir, who wants to "turbocharge" "AI", perhaps consider the effect on other people?
So far we have almost no positive applications for the IP laundering machines.
Law of Buddha: Older tech will have less side effects and more benefits, while modern tech will mostly have side-effects only. Because the older tech came out of need, while modern tech comes out of greed.
Modern tech annoys older tech, like birds poking at dinosaurs. Trains enabled economic progress, which gave rise to computers and AI.