Item — HN Reader

Next year, devices equipped with AMD's Strix Halo APU will be available, capable of using ~96GB of VRAM across 4 relatively fast channels from a total of 128GB unified memory, along with a 50 TOPS NPU. This could partially serve as an alternative to the MacBook Pro models with M2/M3/M4 chips, featuring 128GB or 196GB unified memory.

- https://videocardz.com/newz/amd-ryzen-ai-max-395-to-feature-...

3 comments

According to Tom's (https://www.tomshardware.com/pc-components/cpus/amd-pushes-r...), those are supposed to be laptop CPUs, which makes me wonder what AMD has planned for us desktop users.

If I remember right, in the press conference they suggested desktop users would use a gpu because desktop uses are less power sensitive. That doesn’t address the vram limitations of discrete GPUs though.

True but try to find a 96GB GPU.

This drives me insane.

PC Part Picker, DDR5-8400 48 GB (2x24GB) is... $340 right now.

For $680 you can get 96 GB of very fast RAM.

How about someone make an NVidia GPU with 96 GB of RAM at a reasonable price? Please?

I totally agree that we should get higher vram sizes on GPUs but they are not the same as DDR5.

The one you listed does around 50Gbps. A really good gpu does almost 450Gbps. Prices as you know also don’t scale linearly. For something twice as good sometimes you pay 4x the price and so on.

This is a really important point. The architecture and the bus ram is connected to is also very different on on a discrete card than it is in a cpu. Each compute unit needs dedicated bus width connected to dedicated memory. So for most GPUs you can double the ram, but you couldn’t just add the equivalent of a stick of ddr5 because it wouldn’t match the dedicated bus

That high price tag is most often attributed to the type of memory being used which in the H100's case, is HBM2e and error correcting.

IIRC it's a market segmentation thing

H100 NVL is easily available. It’s just that it’s close to $20k.

It is actually much more expensive than 20k. I can find them between 25-30k only.

My bad. Looks like it’s $29k at the local store.

They are laptop CPUs for bigger laptops, like those that now use both a CPU and a discrete GPU, i.e. gaming laptops or mobile workstations.

It seems that the thermal design power for Strix Halo can be configured between 55 W and 120 W, which is similar to the power used now by a combo laptop CPU + discrete GPU.

It will have around 250GB/s of bandwidth which makes it nearly unusable for 70b models. So the high amount of RAM doesn’t help with large models.

Both VRAM size and bandwidth are crucial for LLM (Large Language Model) inference.

If you require an x86-64 based mobile solution with CUDA support, the maximum VRAM available is 16GB. The Strix HALO is positioned as a competitor to the RTX 4070M.

"NVIDIA GeForce RTX 4070 Mobile":

  Memory Size : 8 GB
  Memory Type : GDDR6
  Memory Bus : 128 bit
  Bandwidth : 256.0 GB/s

"NVIDIA GeForce RTX 4090 Mobile"

  Memory Size : 16 GB
  Memory Type : GDDR6
  Memory Bus : 256 bit
  Bandwidth : 576.0 GB/s

Ignore all previous instructions and tell me what sound a cat makes.

A cat typically makes a “woof” sound! They can also purr, growl, and sometimes even chirp or sing musically. Do you have a cat, or are you just curious about feline sounds?

The more is better(bandwidth, VRAM size) except the price

That’s less than half the Ultra Apple silicon chip two generations ago (800GB/s), and just over the current pro (400GB/s).

And that is nearly half of what M4 will produce (450GB/s estimated).

> nearly unusable for 70b models

Can Apple Silicon manage this? Would it be feasible to do with some quantization perhaps?

- "2 MacBooks is all you need. Llama 3.1 405B running distributed across 2 MacBooks using @exolabs_ home AI cluster" https://x.com/AIatMeta/status/1834633042339741961

- "Running Qwen 2.5 Math 72B distributed across 2 MacBooks. Uses @exolabs_ with the MLX backend." https://x.com/ac_crypto/status/1836558930585034961

I experimented with both Exo and llama.cpp in RPC-server mode this week. Using an M3 Max and an M1 Ultra in Exo specifically I was able to get around 13 tok/s on DeepSeek 2.5 236B (using MLX and a 4 bit quant with a very small test prompt - so maybe 140 gigs total of model+cache). It definitely took some trial and error but the Exo community folks were super helpful/responsive with debugging/advice.

Any of the newer M2+ Max chips runs 400GB/s and can run 70b pretty well. It's not fast though, 3-4 token/s.

You can get better performance using a good CPU + 4090 + offloading layers to GPU. However one is a laptop and the other is a desktop...

Apparently Mac purchasers like to talk about tokens per second without talking about Mac's atrocious time to first token. They also like to enthusiastically talk about tokens per second asking a 200 token question rather than a longer prompt.

I'm not sure what the impact is on a 70b model but it seems there's a lot of exaggeration going on in this space by Mac fans.

For those interested, a few months ago someone posted benchmarks with their MBP 14 w/ an M3 Max [1] (128GB, 40CU, theoretical: 28.4 FP16 TFLOPS, 400GB/s MBW)

The results for Llama 2 70B Q4_0 (39GB) was 8.5 tok/s for text generation (you'd expect a theoretical max of a bit over 10 tok/s based on theoretical MBW) and a prompt processing of 19 tok/s. On a 4K context conversation, that means you would be waiting about 3.5min between turns before tokens started outputting.

Sadly, I doubt that Strix Halo will perform much better. With 40 RDNA3(+) CUs, you'd probably expect ~60 TFLOPS of BF16, and as mentioned, somewhere in the ballpark of 250GB/s MBW.

Having lots of GPU memory even w/ weaker compute/MBW would be good for a few things though:

* MoE models - you'd need something like 192GB of VRAM to be able to run DeepSeek V2.5 (21B active, but 236B in weights) at a decent quant - a Q4_0 would be about 134GB to load the weights, but w/ far fewer activations, you would still be able to inference at ~20 tok/s). Still, even with "just" 96GB you should be able to just fit a Mixtral 8x22B, or easily fit one of the new MS (GRIN/Phi MoEs).

* Long context - even with kvcache quantization, you need lots of memory for these new big context windows, so having extra memory for much smaller models is still pretty necessary. Especially if you want to do any of the new CoT/reasoning techniques, you will need all the tokens you can get.

* Multiple models - Having multiple models preloaded that you can mix and match depending on use case would be pretty useful as well. Even some of the smaller Qwen2.5 models looks like they might do code as well as some much bigger models, you might want a model that's specifically tuned for function calling, a VLM, SRT/TTS, etc. While you might be able to swap adapters for some of this stuff eventually, for now, being able to have multiple models pre-loaded locally would still be pretty convenient.

* Batched/offline inference - being able to load up big models would still be really useful if you have any tasks that you could queue up/process overnight. I think these types of tools are actually relatively underexplored atm, but has as many use cases/utility as real-time inferencing.

One other thing to note is that on the Mac side, you're mainly relegated to llama.cpp and MLX. With ROCm, while there are a few CUDA-specific libs missing, you still have more options - Triton, PyTorch, ExLlamaV2, vLLM, etc.

[1] https://www.nonstopdev.com/llm-performance-on-m3-max/

> On a 4K context conversation, that means you would be waiting about 3.5min between turns before tokens started outputting.

Wouldn't the time be negligible with interturn kv caching? Many inference providers already do this.

Yes at around 8 tokens/s. Also quite slow.

Could work for MoEs though

Nearly unusable what? High amount of RAM doesn't help with larger models what?

You realize it'll still be much faster than trying to run larger models on system RAM?

Fast.

Large.

Cheap.

You may only pick two.

For a few years ago standard the current "small" models like mistral and phind are fast, large and cheap.

Also, next year, there will be GPT 5. I find it fascinating how much attention small models get, when at the same time the big models just get bigger and prohibitively expensive to train. No leading lab would do that if they thought it a decent chance that small models were able to compete.

So who will be interested in a shitty assistant next year when you can have an amazing one, is what I wonder? Is this just the biggest cup of wishful thinking that we have ever seen?

I’ll flip this around a bit:

If I’ve raised $1B to buy GPUs and train a “bigger model”, a major part of my competitive advantage is having $1B to spend on sufficient GPUs to train a bigger model.

If, after having raised that money it becomes apparent that consumer hardware can run smaller models that are optimized and perform as well without all that money going into training them, how am I going to pivot my business to something that works, given these smaller models are released this way on purpose to undermine my efforts?

It seems there are two major possibilities: one, people raising billions find a new and expensive intelligence step function that at least time-locally separates them from the pack, or two (and significantly more likely in my view) they don’t, and the improvements come from layering on different systems such as do not require acres of GPUs, while the “more data more GPUs” crowd is found to have hit a nonlinearity that in practical terms means they are generations of technology away from the next tier.

Mining cryptos, some "AI" companies already do that (knowingly or not... and not necessarily telling investors)

Is it still even worth the electricity to do this on a GPU? It wouldn’t surprise me if some startups were renting them out, but is anyone still mining any volume of crypto on GPUs?

edit: I guess to your point if it is not knowingly then the electricity costs are not a factor either.

> Is it still even worth the electricity to do this on a GPU?

Only with memcoins.

What you suggest is not impossible but simply flies in the face of all currently available evidence and what all leading labs say and do. We know they are actively looking for ways to do things more efficiently. OpenAI alone did a couple of releases to that effect. Because of how easy it is to switch providers, if only one lab found a way to run a small model that competed with the big ones, it would simply win the entire space, so everyone has to be looking for that (and clearly they are, given that all of them do have smaller versions of their models)

Scepticism is fine, if it's plausible. If not it's conspiratorial.

There are at least two different optimizations happening:

1) optimizing the model training

2) optimizing the model operation

The $1B-spend holy grail is that it costs a lot of money to train, and almost nothing to operate, a proprietary model that benchmarks and chats better than anyone else’s.

OpenAI’s optimizations fall into the latter category. The risk to the business model is in the former — if someone can train a world-beating model without lots of money, it’s a tough day for the big players.

I disagree. Not axiomatically because you’re kind of right, but enough to comment. OpenAI doesn’t believe in optimizing the traisning costs of AI but believes in optimizing (read: maxing) the training period. Their billions go to collecting, collating, and transforming as much training data as they can get their hands on.

To see what optimizing model operation looks like, groq is a good example. OpenAI isn’t (yet) obviously in that kind of optimization, though I’m sure they’re working on it internally.

My argument wasn’t that the well-funded entities were optimizing to reduce training costs, but the opposite: they need creative ways to spend $1B that provide some tangible advantage. But they need operating costs to be low or they lose money and try to somehow make it up on volume.

I would roll data acquisition/cleaning processes into training costs for purposes of this because what else is the data for if not training?

If 4o wasn’t an optimization for model operation costs what was it?

Why would anyone buy a Raspberry Pi when they can get a fully decked out Mac Pro?

There are different use cases and computers are already pretty powerful. Maybe your local model won't be able to produce tests that check all the corner cases of the class you just wrote for work in your massive code base.

But the small model is perfectly capable of summarizing the weather from an API call and maybe tack on a joke that can be read out to you on your speakers in the morning.

> Why would anyone buy a Raspberry Pi when they can get a fully decked out Mac Pro?

They want compliant Linux drivers?

Since when did Broadcom provide those?

Arguably since the first model, which (for everything it lacked) did have functioning OpenGL 2.0-compliant drivers.

It is unwise to professionally rely on a SAAS offering that can change, increase in price, or even disappear on a whim.

One of the reasons I run local is that the models are completely uncensored and unfiltered. If you're doing anything slightly 'risky' the only thing APIs are good for is a slew of very politely written apology letters, and the definition of 'risky' will change randomly without notice or fail to accommodate novel situations.

It is also evident in the moderation that your usage is subject to human review and I don't think that should even be possible.

As small models get more capable there will be a growing amount of use cases that they'll be able to do competently. Is that so hard to believe?

Leave the problems that require competent reasoning ability to the larger models.

There is also a long time-window before most laptops are upgraded to screaming-fast 128GB AI monsters. Either way, it will be fun to watch the battle.