Item — HN Reader

Including your AI agents. And models become problematic. There very likely is and/or will be lots of pressure for US AI labs to make models help law enforcement. It could be by implementing backdoors in generated code, or not report some exploitable bugs, or something else. Similar for agents, they basically become the threat in your infra…

(It’s of course not only the US, just that the largest AI providers are US based and we know from history how US agencies operate)

1 comments

Yep.

I'm using open weights models on a privacy-focused provider right now, and that's adequate for my current usage, but I'm rapidly getting to the point where my agent's access level to my data (and to a lesser extent, my accounts) is becoming something I'm not comfortable sending outside my network at all.

My hope is that models that are roughly on par with Deepseek V4 Flash can be run on hardware that I can own for <~$5k in the near future. We're close, but not there yet as far as I know.

The only long-term solution to this is self-hosting.

There is literally no protection or difference of an opensource model doing this.

An actual objection I had while talking to an aerospace company was they don't want opensource models because the threat of it having a poisoned training example on specific systems.

It's easily the most hidden malware possible, completely undetectable until an exact set of tokens unlocks it. Is it line 100,543 of your security product? You will literally not know until it plants it in there.

You can review generated code manually or with models from a second vendor (ideally from a different country). Attackers would have to poison both and do it in a way that also makes them ignore the planted malware when reading code.

Open models also let you read reasoning traces. That means anomalies would show up when the backdoor activates, like a run of unrelated words or a jump in top token probability. It's only undetectable until the first time it happens.

To the best of my knowledge (which, admittedly, is far from comprehensive), that kind of attack on a model shouldn't actually be possible in a deterministic fashion.

If you can't stop them from sometimes telling customers things like "yes, I will give you a penthouse suite at our hotel for only $3/night", why would you be able to guarantee that, with some specific set of tokens, they would produce a perfect and undetectable backdoor customized to the code at hand?

I don’t think it has to be perfect or deterministic that way. It’s enough to have a bias towards implementing a backdoor in some circumstances. Something like, if the machine seems to be used in a Chinese environment, the model is biased towards missing some security issues, or towards implementing the type of bugs that can be used for an RCE, or similar.

Anthropic has done such checks at the agent level: https://cybersecuritynews.com/anthropic-claude-hidden-code/

Their excuse was defending against distillation attacks but you can see how that can be abused