▲ 118 points
back
79 comments
> That being said, such a split design may help with improved AI inferencing.
Unified memory is the only reason Macs are so coveted right now for local AI. A single 192 gb ram Mac costs less than the equivalent in standalone GPUs.
The execution speed for LLM inference gets so slow once you reach models that even fill out a 64GB machine. I was tempted, too, but then realized it was unusable past 48GB-ish and stacking used 3090s was the best price / perf / vram ratio.
What are the good use cases for very large memory amounts?
Mixture of Expert models, where all parameters must be in memory but only a subset are accessed per token, are a sweet spot for Macs.
DeepSeek v3 for instance has 671B params, but should have the memory bandwidth of a 37B dense model with a batch size of one.
Any model you'd recommend for a Mac with 48 GB RAM?
It's slower than using just GPU RAM, but it's still faster than using a regular PC that has a much more limited bandwith between the main memory and the GPU. It's some sort of middle ground on how fast and how cheap you can do inference with LLMs that don't fit into a consumer GPU's RAM.
It's definitely usable past 48GB, I have a 96GB M2 Max and regularly run models that use around 70GB that are very usable.
I also have a home server with 2x3090 and 2xA4000 (80GB vRAM) - yes it's a lot faster, but it's a pain in the ass to build, it takes up a lot of space, uses 10x the power, and honestly - cost about the same as my MacBook Pro.
I ran all my research papers (about LLMs) on a Mac Studio.
Unusable in what way? Llama 3.3 70b q8 with 100k+ context runs as well as GPT 3.5 did a couple years back, except local on Mac and smarter.
Large context has different perf than large model. Op was likely thinking of running 400b models and finding the compute wasn’t enough to make the memory useful.
This isn't splitting the unified memory, it's splitting the soc into a sip. The ram is still on the same interposer/substrate. The actual mm distance won't be regressing to say the distance of a dimm socket to the cpu socket.
The software will still see a single memory pool
I know im disagreeing with the article
I'm sorry, I think you mispronounced market segmentation... This is Apple we're talking about.
When LLMs and diffusion models began spreading freely, it was very funny to have support for some beefy high end GPUs and MacBook Air M1.
I hope Apple sticks with the architecture. Even if its not very practical, its great to have it as possible.
I don’t think the principal market actually cares about that.
Yeah, this is the edge Macs have right now in the AI space. It's why people are looking forward to the Strix Halo from AMD as it will also have a unified memory architecture and will probably cost a good bit less than a Mac.
> Another intriguing aspect is the separate CPU and GPU design. If true, this essentially means M5 will not use a unified memory architecture (UMA) shared between the CPU and the GPU.
This does not follow. Intel is shipping unified memory processors with CPU cores and GPU cores on separate chiplets but still sharing the same memory controller (on a third chiplet, for Meteor Lake and Arrow Lake). AMD is about to launch Strix Halo, a high-end mobile processor that is rumored to consist of one or two CPU chiplets and an IO die with a big GPU and 256-bit memory controller.
Agree. Here is an article on SoIC from Anandtech [1].
Edit: [2] The tweet doesn't even mention about UMA. The interpretation is entirely made up by Notebookcheck, I feel like I am reading WCCFtech again making stuff up.
I am just thinking if this allow Apple to do something crazy like 1024bit LPDDR5x or HBM3e memory solution.
[1] https://www.anandtech.com/show/21414/tsmcs-3d-stacked-soic-p...
[2] https://x.com/mingchikuo/status/1871185666362745227?ref_src=...
Plus Apple themselves are already kind of doing it with the M Ultras - those are two chiplets glued together, each of which is only connected to half of the systems memory directly, but it still behaves like unified memory even though half of the memory traffic has to be routed via the other chiplet.
And in the datacenter space, AMD has taken things even further with the MI300A:
> Twenty-four x86-architecture ‘Zen 4’ cores in three chiplets
> Six accelerated compute dies (XCDs) with 38 compute units (CUs), each with 32 KB of L1 cache, 4 MB L2 cache shared across CUs, and 256 MB AMD Infinity Cache™ shared between XCDs and CPUs
> 128 GB of HBM3 memory shared coherently between CPUs and GPUs with 5.3 TB/s on-package peak throughput
Indeed. Very similar to the grace + hopper, grace + dual hopper, and grace + dual blackwell.
Maybe Apple has figured out something better than a unified memory architecture.
It's hard to rule out their ability to create silicon that is a step change.
I wasn't really trying to comment on what Apple could or could not pull off. Just pointing out that Notebookcheck seems to be misunderstanding what they're reporting on from Ming-chi Kuo, and the headline itself seems to be something Notebookcheck made up rather than something from Kuo's rumors. So this whole thread is even more baseless than it appears at first glance, but it would still be interesting to have an informed discussion about the pros and cons of unified memory for consumer SoCs, and alternatives.
They didn't even figure out unified memory - even original xbox (20+ years ago) had that.
And SGI O2 30 years ago.
I think the Apple II frame buffer was in a unified system memory.
UMA is trivial if you have so little RAM that bandwidth doesn't matter.
The original XBox (2001) had 64MB. I think my PC from 1998 had that.
For sure, they aren't always the first, but they do seem to scale through some things in their own way. Got me my first fastest cpu laptop with meaningful battery life.
Notebookcheck is a poor source of parsing technical information.
The actual rumour from Kuo is that they’d move to a chiplet style design where the CPU tile and GPU tile are independent. This is actually in the article as linked.
That does not however mean that unified memory would go away. It’s just a new packaging system.
UMA hurts the GPU too much. Widely parallel processing wants to access memory in bigger chunks than a CPU. If you try to mix access and modification, you lose the benefit of widely parallel processing. Other GPU designers have considered and eschewed unified memory models, to the tune of hundreds of millions in research dollars.
I agree that single cache-line fetches are pretty poor for parallel vector units, but supporting the former in an environment designed for the latter doesn't seem to off-putting (the CM-5 did this).
By the way: Does your user name convolvotron refer to the hardware 3d audio audio processing system originally developed at NASA Ames Research?
Such a cool name! And it says just what it does.
https://spinoff.nasa.gov/node/8965
https://spinoff.nasa.gov/sites/default/files/thumbnail0000_2...
https://pubs.aip.org/asa/jasa/article/92/4_Supplement/2376/7...
Body Electric supported the Convolvotron for visually programming VR simulations with 3D sound:
https://news.ycombinator.com/item?id=24266722
Did you ever meet (or better yet get a tour of Ames from) the late Ron Reisman, and see the virtual reality, flight simulator, and air traffic control systems his research lab developed?
Vertical Motion Simulator:
https://www.youtube.com/watch?v=5-lHcv_olkE
Marvin Minsky flies a simulator and wears VR goggles:
I think all the mobile GPUs use UMA. I think the tradeoff point is some complicated function of power envelopes and the benefit of more, though slower, memory vs raw performance at any power or $ cost. Though there are several dozen important algorithms that run much better on GPUs, there are really only two of them, 3D graphics and ML tensors, that have had a big consumer and broad professional appeal.
Why? Not like there's a single memory channel. Keeping the memory controller busy with tons of pending requests is a great way to make use of a large fraction of the total memory bandwidth. The M2 Ultra has 32 or 64 memory channels, a cache line pending for each would allow good bandwidth utilization.
Could you give some concrete examples, including when (approximate year/decade is ok) they were considering UMA for CPU/GPU?
As a couple of others have mentioned, smartphones/tablets/laptops seem to be the driving force in UMA's spread.
Not sure why that would be true. Slow UMA (like the vast majority of Intel and AMD desktop chips with 128 bit wide memory) hurts GPU performance.
However the M4 Pro has 256 bits wide, M4 max 512 bits wide, and M2 Ultra has 1024 bits wide. GPU workloads are latency tolerant and embarrassingly parallel, don't see how allowing a CPU to make random accesses is going to hurt the GPU much.
The gpu has a cache on it. So does the cpu. Blow the cache and performance is gone anyway. So uniform memory access is really annoying to implement, really convenient for developers, non-issue performance-wise.
> So uniform memory access is really annoying to implement,
Is it really, though? It seems like almost every SoC small enough to be implemented as a single piece of monolithic silicon has gone the route of unified memory shared by the CPU and GPU.
NVIDIA's GH200 and GB200 are NUMA, but they put the CPU and GPU in separate packages and re-use the GPU silicon for GPU-only products. Among solutions that actually put the CPU and GPU chiplets in the same package, I think everyone has gone with a unified memory approach.
Is the purpose here to make Apple computers a better alternative for gaming? The market is larger than that of people trying to run local LLMs.
Not mentioned in the article, but another motivation behind this could be that with split a CPU/GPU Apple could try to up-sale on both when purchasing Macs.
The prices they charge just to go from 16GB to 32GB of RAM is outrageous ($400 for Macbook pro).
I love the fact you can buy two 16GB/256GB Mac Minis and have cash leftover compared to somone that bought a single 32GB/512GB Mac Mini. Apple's upsells are insane
I’m seeing 16/256 for $600 and 32/512 for $1,200 on apple.com
You can split the CPU and GPU and still have UMA. Splitting CPU/GPU is a packaging and interconnect concern and is not mutually exclusive with UMA.
Given they control both hardware and software, can they have both like efficiency core + power core, with some memory is unified (up to 64 GB) and some dGPU has different memory which they sell against NVIDIA ... Apple has to grow further is hard, but having a T$ firm in front of you with a hardware piece you have is much better than building an Apple Car ...
I've always wondered if they'll do something for a true monster Mac Pro.. 128 cores, gobs of memory, etc.
The way they've treated the Mac Studio by simply not updating it, letting the MacBook Pro M4 nearly surpass it at the top end of performance, doesn't bode well for the future.
Seems like they think Ultras aren't worth the investment, let alone building a true "unleashed" SIP.
Apple never says "hey what's the fastest and most powerful thing we can build for X price", they always box themselves in with space or energy constraints, so they never truly compete for the high end. The existing Mac Pro body was their chance to do that, and instead they put something designed for a smaller chassis in there.
I agree 100%, though I guess holding onto the monster Power Supply in the Mac Pro has me hoping they're still working on something "big" that would justify it.
My guess is we won’t ever see anything more than what can also fit in a Studio. It’s just not worth the extra R&D for whatever fancy interposer tech they’d need to make a standout Mac Pro
The strange part is the M MacPro still has the monster 100W power supply from the intel days. Why on earth did they leave that?
Hackintosh on MI300A?