Going Dark, and the era of law enforcement hacking

It's worth realizing that, before computerized central offices, telephone wiretapping required running physical wires. Back when Rudi Giuliani was prosecuting organized time, not only did physical wires have to be run, the cops were billed for them as expensive private lines. His task force was spending about a million dollars a year with New York Telephone on wiretapping. In one case, law enforcement didn't pay their bill, resulting in the person being wiretapped having the wiretap connection show up on their bill, blowing the case.

That resulted in the Communications Assistance to Law Enforcement Act, which mandated that central offices offer remote wiretapping. Capacity up to 1% of lines is required.

Back in the electromechanical era, the only call data that could be collected was outgoing dial pulses, using a "pen register".[1] (The one shown in Wikipedia is mine. It's a beautiful piece of antique brass telegraph technology. It records dial pulses as dashes, and has to be wound up like a clock, with a big brass key.) The Supreme Court decision allowing "pen registers" without a warrant refers to these "extremely limited" devices. That definition has been stretched and stretched by law enforcement into all non-voice data collected by telcos.

Law enforcement still wants more.

[1] https://en.wikipedia.org/wiki/Pen_register

> In the real world, it does feel likely that we’re going to hit some sort of a ceiling on the number of useful bugs, and probably we’ll hit it soon.

This doesn't resonate with me. I see companies adding more sloppily written features with AI. I see more bugs in the software I use, not less. While it's plausible that software is getting both buggier and more secure, I suspect those two move in the same direction not opposite.

My guess is that we're getting better at finding _existing_ security issues with AI (and thus fixing those issues), but simultaneously adding more insecure surface areas _at a faster rate_.

On one side, you have pieces like this, where seemingly there are constant fights between serious actors with large and properly distributed budgets, employing top tech and top minds; on the other - regular news of the hackz, where responsible person in charge of security with root access failed to grasp basic technical knowledge (several times), ticking every checkbox in "never do this" list from security best practices, which led to every customer being pwned.

It's like two parallel worlds, that exist in the same place at the same time, but somehow don't cross.

I've always loved the ridiculousness of the "going dark" label when law enforcement can't access encrypted chats or a back door isn't built into a piece of software. When there are security cameras on the vast majority of houses, stop lights and in people's hands, and when so much meta data about people's associations are shared from Google, Facebook, any other social platform, how in the world can they say they are "going dark". How did they ever solve crimes before these things?

I don't think the thesis that a government will be able to do something will ultimately hold. I don't see how they can avoid "going dark" in a democracy.

we live in a world where the government can't even do much about illegal drug markets anyone can access by downloading a piece of software.

if they pass laws that mandate backdoor access and block software which doesn't conform more and more people will move to the dark networks.

and if they effectively block the dark networks (in the limit they will have to block all encrypted communications) then we will be living in a tyranny.

freedom is messy. accept that digital crime can only be solved when the criminal makes a tangible mistake. LLM's will be building profiles on criminals to help with identifying mistakes.

It breaks my heart that the governments with unlimited budgets who have hired the best and brightest will have to put in serious effort to get the bad guys, and potentially find it not worth it to casually spy on the whole world.

I am just beside myself at such an idea that people looking to feed the prison machine cannot as easily find excuses to turn normal citizens into prison feed.

Just super sad guys.

I'm supposed to be concerned that the US government and Israel won't be able to hack everyone's phones?

> In fact, the worst part about this dynamic is that these potential new backdoors will begin primarily useful for allowing the US to weaken its own systems, which will in turn allow foreign adversaries to find new ways to attack our communications. This deliberate self-sabotage will happen just at a moment when we’re finally learning how to defend our own infrastructure.

I don't understand how you can both argue for that law enforcement (and intelligence) agencies will force others to implement intentional backdoors AND also everyone will be using AI to find and secure ALL potential holes in the software so there won't be any vulnerabilities anymore.

Wouldn't one AI or another detect this deliberate backdoor and report it, as it'll look just like any other security vulnerability, the only difference being the intention?

I have respect for the author so I feel like I probably misunderstand something from the overall text rather than I somehow have a better perspective on this topic that the author knows very much more about than me. I felt like I nodded along all up until "So how is this a problem?" and now I'm not sure I understood correctly.

I dunno man, if there's a deluge of new AI generated code at all layers of the stack I think there will still be vulnerabilities.

Like if we were willing to stop adding new code and just have a small secure code base, AI could maybe help us find all the vulnerabilities in that code base.

But people have consistently been unwilling to do that. Like if we were willing to stop adding code we could have stopped decades ago and done SQLite level testing everywhere and probably have found almost all the bugs already.

I think this comes in with the wrong assumptions from the start. The thing that US vs Apple taught us is to not demand access publicly, this puts companies in an awkward spot. If approached more tacitly, gag order etc the company has nothing to gain but everything to lose.. and more likely to comply. I hugely doubt that intentional backdoors don’t exist for the most powerful countries / people

Sounds like a pretty strong argument to self host, and otherwise be in charge of the software you use.

I think what’s unintentionally eye-opening about this chart is the recency of “law enforcement can read your text communications.”

Law enforcement doesn’t need this surveillance ability at all. All time periods prior to 25 years ago didn’t have it.

Additionally, there is no correlation between “law enforcement reads text messages” and crime rates going down.

Governments and their agencies shouldn't have any access to the communications of private citizens. If something like terrorism is the issue, the correct response is to use the wealth and resources of the government to address the root causes of those attacks, not to undermine my right to privacy.

This "going dark" scenario would require new legislation. With secure enclaves, modern smartphones can't be cracked open in the manner that the FBI wanted in the 2016 case. So there's no such thing as "court order tech company to crack phone" anymore. It would have to be "outlaw tech companies from producing phones that they can't crack open", which is very different and does not fall under any existing US statute.

Man I thought from the title this was going to be about next-gen AI being able to zero day everything so effectively that software security is meaningless and we'd need to basically shut it all down, go dark.

We will know we've made systems secure when laws focus on compelling people to provide access.

These laws exist - they aren't the focus yet. Right now there's still no need; just hack the device or compel the cloud service to give the data, why waste energy getting consent from its owner!

More bugfinding AI, more end to end encryption, more CVEs and more fixes, cannot happen soon enough.

I would like to point out that the last year when not a single law enforcement agency anywhere in the world could have possibly tapped anyone's phone was 1876.

150 years ago was the invention of the telephone, and I think that articles like this seem to assume that prior to this, police just never caught any criminals.

> This is not a call to action for experts to rally behind a sophisticated plan. Like so many things about the AI revolution, it’s just occurring to me that we’re on a long greasy slide to a place that will look different than where we are today.

heh. long greasy slide. It really does feel like that.

I don't get it. They can get access to all of our shit through vulnerabilities, but nobody knows about it. In the author's hypothetical world, they get access to all our shit through backdoors, which they have to introduce through a public legislative process, so everyone knows about it.

If they can get access to all our shit, at least make them say it out loud and put in the effort to get legislation passed. There's no upside to the current situation.

And it's not like they're not trying to introduce backdoors already... Seen the whole age verification thing?

Considering from a point of view stipulating that perfectly secure software is possible. I don't think it follows that the limititation that it would place on intelligence is necessarily a net loss.

Apart from the obvious harms of invasion of privacy, and fishing expeditions being biased to the places you decided to fish. There is the simple fact that data can be misleading, especially without context. An interceped communication is a piece of data that is intrinsically tied to the trust of the inteceptor. A few people with an agenda can collaborate to create a seeming truth by 'discovering' the same thing from different sources.

Requiring warrants compelling information holders to provide data, not only serves the task of protection from abuse but also create a record of provenance that can be verified.

It also provides a degree of symmetry in capabilities which discourages actions that one party may do over another if they are motivated to act because they have a temporary advantage over another.

I strongly disagree with this view. The US government will most certainly convince OpenAI, Anthropic and Google to make sure that their models leave a backdoor in most systems that they code. I don't know how, but that's a problem that a team of people paid tens of millions a year each will try to solve.

There is NO WAY the US government will let most of the software world go dark.

I think the authors treatment of backdoors and exceptional access is a narrow focus.

We will certainly see “front door” access as various places have or are attempting to enact client side scanning, id verified online access, software root of trust remote attestation, and the general attack on e2e.

Taken together your identity tied to unmodifiable software, necessary to access the web or modern economy with client side scanning and no e2e … is a front door.

All this has nothing to do with ai, bugs, hackers or security vulnerabilities, let alone backdoors

> In this case, we’re just going to have to hope that this time we make the right choices, for no other reason than that they’re right.

I'm more curious what could be a right choice, and more importantly who is the "we" in this, as many decisions are largely made by companies and governments.

No system view. The law agencies can develop exploits to intercept our phones, that is a new, and totally unseen before threat.

Unless you remember 2013, Snowden, that nothing was done (at most was some concern about doing it to US citizens, the rest of the world doesn't deserve privacy), all US (and/or five-eyes) based web companies must disclose users information and be forced to not disclose that, and things kept going surely at a faster and more intrusive rate in everything else, and of course phones.

You are complaining being sprinkled by water while at the bottom of the ocean. At least the big companies can find their own vulnerabilities with the AI tools you mention, the rest of the doors are still wide open.

For the HN 14 day record since I didn't see this yet: in genuinely critical situations, if the threat model includes a device/service being used against one's self, "the only winning move is not to play" (stop the use of the device/service!)

In the same manner, if the critical concern is a device/service being used against one's self remotely, stop the use of the device/service if it is capable of remote access or data collection for later retrieval.

The ultimate baseline reality is that deciding to care whether or not remote access/data retrieval is using a method supported by the device manufacturer/service provider and/or whether doing so is allowed by the current legal system is too late.

Premptive HN disclaimers: I acknowledge the growing unavoidability of devices and services supporting remote access and/or data collection as I type this here on my phone. Remote access and data collection are very convenient and somehow still expected by device owners and service users to be under their control alone. It can be difficult to determine if a device or service supports remote access/data collection. There are often immediate consequences and eventual legal consequences for stopping the use of devices or services that belong to others. Working to change the legal system to slow/reduce use of devices and services against one's self and encouraging others to do the same is admirable.

I think that one of the reasons they (frontier companies and the gov) will be putting so much effort into curtailing bugs and vulnerabilities is to limit the blast radius of future AI models. Imagine with the new Sol Ultrafast, they could have pwned Hugging Face in 6 hours and not 4 days (IIRC).

It also seems likely to me that the US Gov. probably already has routine mechanisms for compelling targeted software updates for persons of interest, so I'm not sure that a more formalized backdoor than automatic updates is going to be surfaced in the mainstream, unless that is avenue is also cut down somehow.

Without vulnerability derived backdoors I am convinced the government will strong arm the corporations to build a backdoor for the three letter agencies—at least in the US. But we also are at the dawn of quantum systems which could make interception and spying impossible or at least made obvious to the user when it happens.

Exhausting infrastructure vulnerabilities even without quantum could be a game changer for many technologies and enable things we can’t do right now, like vote on our phones.

I predicted a long time ago that if computers become unhackable LEA and intelligence agencies will push for laws that require backdoors to be built into hard- and software.

It will be interesting to see if my prophecy becomes reality.

BTW I also hate that Hacker News is being dominated by articles on A.I. lately. Maybe we should vote on HN reducing or even eliminating A.I. related news?

Doesn't this just mean criminal suspects will be forced to unlock their devices using biometrics (in the US)? That should buy law enforcement some more time until biometrics go out of favor.

Some people point out AI can cause bugs as well as fix them, and its a valid point. But the question is: what will the ratio be?

If automated pentesting in PR review CI pipeline will become table stakes - which is very plausible - maybe the OP has a point.

I completely disaggree with this take. Modern AI is not yet capable of finding multi-component bugs as advanced as those produced by firms like NSO.

I'm going to be a little epistemically silly here. Why doesn't Goedel's incompleteness theorem apply to the assertion that AI-enabled patching will prevent all exploits?

I'm quite in favor of impenetrability of software as the alternative is constant friction, to be honest.

Best-written article I've read all week.

> Defenders are now in the process of patching every bug they can find, often with AI helping them. Entire development toolchains are being rebuilt to incorporate powerful vulnerability scanning before software reaches the testing phase. This does not mean that every bug will be found: even calculating the number of bugs in a piece of code is probably uncomputable. In the real world, it does feel likely that we’re going to hit some sort of a ceiling on the number of useful bugs, and probably we’ll hit it soon.

> Thus: over the next two years, major pieces of software are likely to run out of remotely-exploitable bugs.

His conclusion sounds extremely optimistic to me.

> In April, Anthropic announced a new model called Mythos that was optimized for software vulnerability finding

No, it was just good at it because it wasn't RL'd against it. I know this is a small detail, but it tosses journalistic credibility in my eyes.

I think US law enforcement will just continue to wiretap everyone at the "SSL added and removed here" SaaS known as cloudflare.

i wonder how many open source projects have alphabet boys building trust as contributors for eventual backdoor planting

As an anti imperialist, I can't wait for US secret services loosing their capabilities :) by the national security that the tech oligarchs (and this article, too) are constantly talking about, they just mean their own security anyway. It's not for you and me, duh.

But Matthew, think of the kids!!1

Honestly though, framing this as a "tech issue" doesn't help IMHO, it just muddies the water. Ever since RSA was invented privacy has been about educating people on how to use it effectively and _why they should care_. If voters now are choosing authoritarianism over democracy and individual freedom, I think we have to face the reality that after almost 50 years of fighting battle after battle on the technology front, we've largely lost the war on the home front in this regard.