Gemini 2.5 Computer Use model

I've had good success with the Chrome devtools MCP (https://github.com/ChromeDevTools/chrome-devtools-mcp) for browser automation with Gemini CLI, so I'm guessing this model will work even better.

It successfully got through the captcha at https://www.google.com/recaptcha/api2/demo

> Solve today's Wordle

Stucks with:

> ...the task is just to "solve today's Wordle", and as a web browsing robot, I cannot actually see the colors of the letters after a guess to make subsequent guesses. I can enter a word, but I cannot interpret the feedback (green, yellow, gray letters) to solve the puzzle.

(Just using the browserbase demo)

Knowing it's technically possible is one thing, but giving it a short command and seeing it go log in to a site, scroll around, reply to posts, etc. is eerie.

Also it tied me at wordle today, making the same mistake I did on the second to lass guess. Too bad you can't talk to it while it's working.

I wonder how it would behave in a scenario where it has to download some file from a shady website that has all those advertisement with fake "download"

I believe it will need very capable but small VLMs that understand common User Interfaces very well -- small enough to run locally -- paired with any other higher level models on the cloud, to achieve human-speed interactions and beyond with reliability.

Really feels like computer use models may be vertical agent killers once they get good enough. Many knowledge work domains boil down to: use a web app, send an email. (e.g. recruiting, sales outreach)

Many years ago I was sitting at a red light on a secondary road, where the primary cross road was idle. It seemed like you could solve this using a computer vision camera system that watched the primary road and when it was idle, would expedite the secondary road's green light.

This was long before computer vision was mature enough to do anything like that and I found out that instead, there are magnetic systems that can detect cars passing over - trivial hardware and software - and I concluded that my approach was just far too complicated and expensive.

Similarly, when I look at computers, I typically want the ML/AI system to operate on a structured data that is codified for computer use. But I guess the world is complicated enough and computers got fast enough that having an AI look at a computer screen and move/click a mouse makes sense.

I just have to say that I consider this an absolutely hilarious outcome. For many years, I focused on tech solutions that eliminated the need for a human to be in front of a computer doing tedious manual operations. For a wide range of activities, I proposed we focus on "turning everything in the world into database objects" so that computers could operate on them with minimal human effort. I spent significant effort in machine learning to achieve this.

It didn't really occur to me that you could just train a computer to work directly on the semi-structured human world data (display screen buffer) through a human interface (mouse + keyboard).

However, I fully support it (like all the other crazy ideas on the web that beat out the "theoretically better" approaches). I do not think it is unrealistic to expect that within a decade, we could have computer systems that can open chrome, start a video chat with somebody, go back and forth for a while to achieve a task, then hang up... with the person on the other end ever knowing they were dealing with a computer instead of a human.

Hey - I'm on the team that launched this. Please let me know if you have any questions!

This will never hit a production enterprise system without some form of hooks/callbacks in place to instill governance.

Obviously much harder with UI vs agent events similar to the below.

https://docs.claude.com/en/docs/claude-code/hooks

https://google.github.io/adk-docs/callbacks/

Absolutely hilarious how it gets stuck trying to solve captcha each time. I had to explicitly tell it not to go to google first.

In the end I did manage to get it to play the housepriceguess game:

https://www.youtube.com/watch?v=nqYLhGyBOnM

I think I'll make that my equivalent of Simon Willison's "pelican riding a bicycle" test. It is fairly simple to explain but seems to trip up different LLMs in different ways.

I feel like screenshots should be the last thing you reach for. There's a whole universe of data from accessibility subsystems.

Not great at Google Sheets. Repeatedly overwrites all previous columns while trying to populate new columns.

> I am back in the Google Sheet. I previously typed "Zip Code" in F1, but it looks like I selected cell A1 and typed "A". I need to correct that first. I'll re-type "Zip Code" in F1 and clear A1. It seems I clicked A1 (y=219, x=72) then F1 (y=219, x=469) and typed "Zip Code", but then maybe clicked A1 again.

My general experience has been that Gemini is pretty bad at tool calling. The recent Gemini 2.5 Flash release actually fixed some of those issues but this one is Gemini 2.5 Pro with no indication about tool calling improvements.

How likely is it that the end game becomes that we stop writing apps for actual human users and instead sites become massive walls of minified text against a black screen.

I’m looking forward to a desktop OS optimized version so it can do the QA that I have no time for!

> It is not yet optimized for desktop OS-level control

Alas, AGI is not yet here. But I feel like if this OS-level of control was good enough, and the cost of the LLM in the loop wasn't bad, maybe that would be enough to kick start something akin to AGI.

Computer Use models are going to ruin simple honeypot form fields meant to detect bots :(

Interesting, seems to use 'pure' vision and x/y coords for clicking stuff. Most other browser automation with LLMs I've seen uses the dom/accessibility tree which absolutely churns through context, but is much more 'accurate' at clicking stuff because it can use the exact text/elements in a selector.

Unfortunately it really struggled in the demos for me. It took nearly 18 attempts to click the comment link on the HN demo, each a few pixels off.

I really want this model to try userinyerface.com

The irony is that most of tech companies make their money by forcing users to wade through garbage. For example, if you could browse the internet and avoid ads, why wouldn't you? If you could choose what twitter content to see outside of their useless algorithms, why wouldn't you?

doesn't seem like it makes sense to train AI around human user interfaces which aren't really efficient. It is like building a mechanical horse.

I would love to use this for E2E testing. It would be great to make all my assertions with high level descriptions so tests are resilient to UI changes.

Seems similar to the Amazon Nova Act API which is still in research preview.

One of the slightly buried stories here is BrowserBase themselves. Great stuff.

How big are Gemini 2.5(Pro/Flash/Lite) models in parameter counts, in experts' guesstimation? Is it towards 50B, 500B, or bigger still? Even Flash feels smart enough for vibe coding tasks.

Is this different from ChatGPT agent mode that I can use from the web app? I found that extremely useful for my task which required running some python and javascript code with open source libraries to generate an animated video effect.

I greatly appreciated ChatGPT writing the code and then running it on OpenAI's VMs instead of me pasting that code on my machine.

I wish Google released something like that in AI Studio.

At some point just having APIs for the web would just make sense. Rendering it and then throwing llms at interpreting it seems…suboptimal

Impressive tech nonetheless

I assume its tool calling and structured output are way better, but this model isn't in Studio unless its being silently subbed in.

I think it’s related that I got an email from google, titled “ Simplifying your Gemini Apps experience”. It reads no privacy maximize AI. They are going to automatically collect data from all google apps, and users no longer have options to control access to individual apps.

I've always been interested in running LLM locally to automate browser tasks, but every time I've tried, I've found the browser API to be too complex. In contrast, writing scripts directly with Playwright or Puppeteer tends to be much more stable.

Can somebody give me use cases that are faster than using an UX?

How am I supposed to use this. I really can’t think of one, but I don’t want to be blind-sighted as obviously a lot of money is going into this.

I also appreciate the tech behind it and functionality, but I still wonder for use cases

It is actually quite good at following instructions, but I tried clicking on job application links, and since they open in a new window, it couldn't find the new window. I suppose it might be an issue with BrowserBase, or just the way this demo was set up.

Does it work with ~legacy~ software? Eg, early 2000's Windows WhateverSoft's Widget Designer? Does it interface over COM?

There's a goldmine to be had in automating ancient workflows that keep large corps alive.

Only use in environments where you can roll back everything.

This is great. Now I want it to run faster than I can do it.

Have average Google developers been told/hinted that their bonuses/promotions will be tied to their proactivity in using Gemini for project work?

Is there a claude code for computer use models? I mean something that's actually useful and not just a claude.ai kinda thing.

Does it know what's behind the "menu" of different apps? Or does it have to click on all menus and submenus to find out?