back
182 comments
Google announcement:

https://cloud.google.com/blog/products/ai-machine-learning/r...

Interesting that the hardware is NVidia Blackwell, not Google TPUs. That means Google will likely have an energy efficiency and cost advantage, and keep their proprietary hardware out of other people's reach.
It’s telling that their effort was to get an air gapped solution cleared for US government and the US military.
A bit thin on detail, but will this require confidential VMs with encrypted GPUs? (And I wonder how long before someone cracks SEV-SNP and TDX and pirate copies escape into the wild.)
At the pace models improve, the advantage of going the dark route shouldn't really hold for long, unless I'm missing something.
The number of folks that have the hardware at home to run it is going to be very low and the risk of companies for leaking it is gonna make it unlikely IMHO.
> I wonder how long before someone cracks SEV-SNP

https://bughunters.google.com/blog/5424842357473280/zen-and-...

I'd expect watermarked model weights plus a lot of liability to distinctivise leaking the model.
This might be a great way for them to strengthen their model through federated learning.

https://federated.withgoogle.com/

I did my undergrad internship on federated learning. I was tasked with implementing in a simulator different federated algorithms, so to have a way to compare them in a meaningful way. The last that had to be implemented was FedMA. We didn't manage to do it. That algorithm is absolutely devilish. Every issue that I solved made other two issue arise, and neither my supervisors could help. The sheer idea of matching neurons in different networks might (and does) make sense, but the way the approximate costs are calculated require other 2/3 math papers that I could follow for only the first lines of the abstract. I'm happy for the time I spent in my internship there. I'm also happy it's over

The general understanding of how it works is surprisingly easy though, you can find the paper here https://arxiv.org/abs/2002.06440

The whole point of deploying such things on-prem is air-gapping it from Google and its "learning".
This is obvious government contract baiting. Kudos though they might actually move some Google Distributed Cloud this way
Financial firms with significant on-prem datacenter use will love this as well. My company still stays away from the cloud -- we have 6 DCs in the building, and run everything else out of colocated racks.
I don’t think so. To my knowledge GCP has no approval for classified networks, which is by far the hardest part. Contrast with Azure OpenAI has been approved to run on government networks for over a year now.

This feels like a play for companies in highly regulated industries, GCP has a notable list of biopharma customers.

This is obvious government contract baiting

You don't have to be a government agency to not want your company's data all over the place.

They’ll have to fight Microsoft who’s been promising copilot.
Curious if this was forced on Google Cloud by Sundar, or was it something that Google Cloud as an org wanted to do?

At first glance, it seems Google Cloud might lose some revenue from customers who can now deploy Gemini in-house. On the other hand, it's not a complete loss, since presumably Google Cloud is still involved in providing some underlying tech? Not to mention, some customers would never consider using off-premises setup anyway.

I assume Google Distributed Cloud is part of the larger Cloud org so they get the revenue either way. The on-prem version may even cost more.
Like you can with deep seek? Or will it be more complicated and expensive. I don't know who would actually want that.
Absolutely many would, especially those with deep pockets. The biggest concern I'm hearing from companies adopting AI, for basically any use case, is data leaving their network. Especially (but not only) in the EU.
Folks who would prefer to run deepseek are not in the end customer for this product. Deepseek doesn't provide a service contract.
I don't understand how Google is willing to do this but won't sell TPUs to other days centers. It should be obvious from Nvidia's market cap that they're missing a huge opportunity.
The only reasons I can think of is they see them as their secret sauce, they don't want to support them for customers long-term, or they don't have the foundry capacity.
What is the risk that some hacker could exfliltrate the weights?
Very low if they use confidential VMs (CPU rooted encryption). Just like the Xbox uses and remains unhacked 10 years later.
Seems pretty high, this is an air gapped product so at some point the employees of whatever government they are giving it to would need to SSH into the VM's to load new weights etc. Lots of ways to make it tricky/watermark the weights though.
I work for a bank and we're banned from using AI because of the privacy problem. I'd think there is a big market for a product like this.
Is Gemini tied/benefitting from Google TPU hardware? Because you need hardware in the data center to run this, and I feel it is somewhat specialised.
Gemini models are written in Jax which through the XLA compiler can be compiled either to TPU or GPU hardware.

Performance may differ but Google (and Nvidia) are very interested in having good performance on both platforms.

The raw computation is just a bunch of matrix multiplications in a row, most of the algorithmic complexity/secret stuff would be around scaling & efficiency.

For training the model the HW is much more important as you need to scale up to as many chips as possible without being bottlenecked by the network.

This would just be inference, and it doesn't need to be very efficient as its for on prem usage not selling API access. So you could strip out any efficiency secrets, and it would probably look like a bigger Gemma (their open source model).

I wonder if they would/could try and strip out stuff like whatever tricks they use for long context + video support (both of which they are a bit ahead of everyone else on).

The model itself is likely built upon their own open source system JAX so they should be usable in Nvidia. Of course cost efficiency is going to be a different story.
TPUs are definitely the reason why Gemini models have both massive context and very low prices. There is no nvidia tax to pay.
The Google blogpost notes that it's a partnership with Nvidia, so using cuda rather than TPUs apparently.
Google abandoned Coral in true Google style.
Reminds me of the Google search appliance.
That’s the first thing I thought of as well. I had to integrate one into our custom CMS early in my career. I vaguely remember explaining to management that I was not responsible for the order or quality of search results and tweaking queries (now prompts?) with hints to restrict searches to certain paths. It was such an opaque device, but provided better results than MySQL did at the time.
Not a cat's chance in hell that any eu organisation will rush to this offer right now. Or maybe ever in fact.
Huh, how does that matter? Maybe OVH AI is fine for EU.
What a sudden change of heart. Thank you, DeepSeek!
Google AI rackmount appliance coming soon?
Is The Gavin Belson Signature Edition Box is needed to run these?
The google search appliance might have been one of the worst products I've ever used in my career. If they're going to make a box, I hope they put some effort into it.
FTA: As part of the announcement, Google said Nvidia will bring Gemini models to the company’s Blackwell graphics processing units, or GPUs. Companies can buy the chips through Google or other channels.
More like the GSA probably.

https://en.m.wikipedia.org/wiki/Google_Search_Appliance

It's still an advertising company you're doing business with.

I mean, would you buy cookies from a brand that is known for producing rodenticides?

I believe these are pure word tricks to suggest privacy without actually delivering it.

As context, you need to remember that Google deleted their "Don't Be Evil" motto and became a defense contractor. The customer will most likely receive a black box owned and set up by Google. That means they have no way of knowing if the system inside is phoning home or being remote controlled by an US government agency, or not. You can then say that the model is hosted in your own data center, which might make some people feel good, but using it with personal information is still a violation of the GDPR.

If Google, however, would make these boxes fully offline capable and I was also allowed to wipe all hard disks myself before returning it, that would convince me of their good intentions.