back
242 comments
A good technical project, but honestly useless in like 90% of scenarios.

You want to use an NVidia GPU for LLM ? just buy a basic PC on second hand (the GPU is the primary cost anyway), you want to use Mac for good amount of VRAM ? Buy a Mac.

With this proposed solution you have an half-backed system, the GPU is limited by the Thunderbolt port and you don’t have access to all of NVidia tool and library, and on other hand you have a system who doesn’t have the integration of native solution like MLX and a risk of breakage in future macOS update.

Chicken/egg. NVidia tooling is lacking surely in part because the hardware wasn’t usable on macOS until now. Now that it’s usable that might change.
Wrong.

If a model can run on a 512GB M3 Ultra via MLX or CUDA, but simultaneously benefit from the memory bandwidth of something like an RTX 6000 Pro; that would save my company hundreds of thousands of dollars. That's $20,000 for roughly 600GB of VRAM, and enough token generation speed to fulfill the needs of any enterprise that's not a hyperscaler or neocloud.

I'll let someone else do the math for you on what it costs to put together a 10U server to get that kind of performance without the $10K M3 Ultra Studio.

What we're paying for five old 80GB A100s is criminal, but it's nothing compared to what these GB200 Blackwell setups are going to cost in 2030. Market economics aside, the fact that they require sophisticated liquid cooling infrastructure and draw 3x the power of the A100s, will make these cards unattainable for small to medium organizations.

So yeah, if there's some outside chance that we can pair NVIDIA's speed with a an arm-powered machine that offers 512GB Unified Memory while drawing 50W -- you better believe it's a big deal. We'll see. Sounds too good to be true.

Thank you for opening my mind to a viewpoint I didn’t even know existed.

Yes, for many scenarios this is "not even an academic exercise".

For a very select few applications this is Gold. Finally serious linear algebra crunch for the taking. (Without custom GPU tapeout.)

"Nvidia." Not NVidia or nVidia, or the other ways. I feel that I can frequently figure out if someone is going to express a negative view about this company based only on whether they picked a weird way to write their name.
I misunderstood eGPU for virtual GPU. But I was wrong it means external GPU.
> the GPU is limited by the Thunderbolt port

Not everything is limited by the transfer speed to/from the GPU. LLM inference, for example.

> GPU is limited by the Thunderbolt port

I thought Thunderbolt was like pluggable PCI? The whole point was not to limit peripherals.

Even with running ML experiments you'd mostly want to run them on rented out clusters anyway
the tooling is just the standard linux tooling inside the container, no? and thunderbolt is not a real limitation
I don't know how Apple has evaded regulatory scrutiny for their refusal to sign Nvidia's eGPU drivers since 2018.
Evidence that NVIDIA has even been trying? My understanding is that Apple didn’t allow 3rd parties to write graphics drivers past 10.13, but they could’ve done a non-graphics driver like this.
The government doesn’t care? They’re a minority of the market? The vast majority of their computers didn’t have slots to put Nvidia GPUs in, and now none of them do?
Apple doesn’t have a monopoly in any market they are in.
The same way Google evaded regulatory scrutiny for refusing to allow a YouTube client for Windows Phone?
They aren't a monopoly, hence why.
Isn't all you have to do disable SIP?
From what I understand, only works with Tinygrad. Which is better than nothing but CUDA or Vulkan on pytorch isn’t going to work from this.

[1] https://docs.tinygrad.org/tinygpu/

Such a shame both companies are big on vanity to make great things happen. Imagine where you could run Mac hardware with nvidia on linux. It's all there, and closed walls are what's not allowing it to happen. That's what we as customers lose when we forego control of what we purchase to those that sold us the goods.
Don't purchase? I don't own any Apple devices, everything works fine.
Woah, this is exciting. I'm traveling but I have a 5090 lying around at home. I'm eager to give it a go. Docs are here: https://docs.tinygrad.org/tinygpu/

I hope it'll work on an M4 Mac Mini. Does anyone know what hardware to get? You'll need a full ATX PSU to supply power, right? And then tinygrad can do LLM inference on it?

You can buy a cheap GPU enclosure for about 100$ off ali express.

Takes a standard PSU. However, Mac Minis don't have occulink. So you might be a bit limited by whatever USB C can do.

Now if Intel can get there Arc drivers in order we'll see some real budget fun.

https://www.newegg.com/intel-arc-pro-b70-32gb-graphics-card/...

32 GB of VRAM for 1000$. Plus a 500$ Mac Mini.

Maybe I’m lacking imagination. But how will a GPU with small-ish but fast VRAM and great compute, augment a Mac with large but slow VRAM and weak compute? The interconnect isn’t powerful enough to change layers on the GPU rapidly, I guess?
based on your card, it should be using a decent bit of the 600w or so passed through the new 16-pin connector. goes without saying, a proper PSU (doesn't have to be ATX, but at least 750W to be on the safer side) is a must.

for thunderbolt enclosures, consider going through the list - https://egpu.io/best-egpu-buyers-guide/#tb3-enclosures

zero idea about mac support so YMMV.

“Lying around”. I’ve got an unopened 5090 in a box that I know will suffer the same fate, so I’m sending it back. So privileged to have the money to impulse buy a 5090 and yet no time to actually do anything with it.
I followed the instructions link and read the scripts...although the TinyGPU app is not in source form on GitHub, this looks to me like the GPU is passed into the Linux VM underneath to use the real driver and then somehow passed back out to the Mac (which might be what the TinyGrad team actually got approved).

Or I could have totally misunderstood the role of Docker in this.

https://docs.tinygrad.org/tinygpu/ are their docs, and https://github.com/tinygrad/tinygrad/tree/4d36366717aa9f1735... is the actual (user space) driver.

My read of everything is that they are using Docker for NVIDIA GPUs for the sake of "how do you compile code to target the GPU"; for AMD they're just compiling their own LLVM with the appropriate target on macOS.

I think that metal isn’t double precision; so that limits some serious physics simming; but if you’re doing that I guess you just rent a gpu somewhere.

I would definitely be into this if adding an egpu was first class supported.

Interesting, but cannot run CUDA or more to the point `nvidia-smi`.
Pretty misleading. This driver is only for compute not graphics.
> If you have a Thunderbolt or USB4 eGPU and a Mac, today is the day you've been waiting for!

I got an eGPU back in 2018 and could never get it to work. To the point that it soured me from doing it again.

These days for heavy duty work I just offload to the cloud. This all feels like NVidia trying to be relevant versus ARM.

If you could get Nvidia driver support on Mac’s I bet Apple would have sold more MacPro’s.
I'm writing scientific software that has components (molecular dynamics) that are much faster on GPU. I'm using CUDA only, as it's the eaisiest to code for. I'd assumed this meant no-go on ARM Macs. Does this news make that false?
What are the limitations of USB4/Thunderbolt compared with a regular PCIe slot?
Apple should update this page for ARM macs, now runs tinygrad on eGPUs: https://support.apple.com/en-us/102363
My main thought is would this allow me to speed up prompt process for large MoE models? That is the real bottleneck for m3ultra. The tokens per second is pretty good.
These tinyboxes are so expensive (starting at $12,000), why don't they just put a CPU inside and allow users to ssh into them?
Why does Apple need to make the drivers in a walled garden? Atleast they should support major device categories with official drivers.
Can I do prefill on the eGPU and the decode on the Mac?
Isn't it sad that we've ended up in a situation where we are talking about "Apple approves" rather than "someone creates"? Fuck Apple.
yes good report
Idk why this doesn't link to the original source instead of this proxy source: https://x.com/__tinygrad__/status/2039213719155310736
The opportunity cost of Apple refusing to sign Nvidia's OEM AArch64 drivers is probably reaching the trillion-dollar mark, now that Nvidia and ARM have their own server hardware.