back
148 comments
The graphs ranking GPUs may not be accurate, as they don't represent real-world results and have factual inaccuracies. For example:

> Shown is raw relative performance of GPUs. For example, an RTX 4090 has about 0.33x performance of a H100 SMX for 8-bit inference. In other words, a H100 SMX is three times faster for 8-bit inference compared to a RTX 4090.

RTX 4090 GPUs are not able to use 8-bit inference (fp8 cores) because NVIDIA has not (yet) made the capability available via CUDA.

> 8-bit Inference and training are much more effective on Ada/Hopper GPUs because of Tensor Memory Accelerator (TMA) which saves a lot of registers

Ada does not have TMA, only Hopper does.

If people are interested I can run my own benchmarks on the latest 4090 and compare it to previous generations.

Would be curious to see your benchmarks. Btw, Nvidia will be providing support for fp8 in a future release of CUDA - https://github.com/NVIDIA/TransformerEngine/issues/15

I think TMA may not matter as much for consumer cards given the disproportionate amount of fp32 / int32 compute that they have.

Would be interesting to see how close to theoretical folks are able to get once CUDA support comes through.

Can you please run bench from https://github.com/karpathy/nanoGPT ?
When people say “8 bit inference” they mean INT8, not FP8.
I find this article odd with its fixation on computing speed and 8bit.

For most current models, you need 40+ GB of RAM to train them. Gradient accumulation doesn't work with batch norms so you really need that memory.

That means either dual 3090/4090 or one of the extra expensive A100/H100 options. Their table suggests the 3080 would be a good deal, but it's not. It doesn't have enough RAM for most problems.

If you can do 8bit inference, don't use a GPU. CPU will be much cheaper and potentially also lower latency.

Also: Almost everyone using GPUs for work will join NVIDIA's Inception program and get rebates... So why look at retail prices?

> Gradient accumulation doesn't work with batch norms so you really need that memory.

Last I looked, very few SOTA models are trained with batch normalization. Most of the LLMs use layer norms which can be accumulated? (precisely because of the need to avoid the memory blowup).

Note also that batch normalization can be done in a memory efficient way: It just requires aggregating the batch statistics outside the gradient aggregation.

> It doesn't have enough RAM for most problems.

It might not be as glamorous or make as many headlines, but there is plenty of research that goes on below 40Gb.

While I most commonly use A100s for my research, all my models fit on my personal RTX 2080.

I'm not sure any of this is accurate. 8 bit inference on a 4090 can do 660 Tflops and on an H100 can do 2 Pflops. Not to mention, there is no native support for FP8 (which are significantly better for deep learning) on existing CPUs.

The memory on a 4090 can serve extremely large models. Currently, int4 is started to become proven out. With 24GB of memory, you can serve 40 billion parameter models. That coupled with the fact that GPU memory bandwidth is significantly higher than CPU memory bandwidth means that CPUs should rarely ever be cheaper / lower latency than GPUs.

> Almost everyone using GPUs for work will join NVIDIA's Inception program and get rebates... So why look at retail prices?

They need to advertise it better. First time I hear about it.

What are the prices like there? GPUs/workstations?

> For most current models, you need 40+ GB of RAM to train them. Gradient accumulation doesn't work with batch norms so you really need that memory.

There's a decision tree chart in the article that addresses this - as it points out there are plenty of models that are much smaller than this.

Not everything is a large language model.

> Almost everyone using GPUs for work will join NVIDIA's Inception program and get rebates... So why look at retail prices?

So maybe they were including information for the hobbyists/students which do not need or cannot afford the latest and greatest professional cards?

> If you can do 8bit inference, don't use a GPU. CPU will be much cheaper and potentially also lower latency.

Good advice. Does that mean that I can install like 64 gb ram on a PC and run those models in comparable time?

> Also: Almost everyone using GPUs for work will join NVIDIA's Inception program and get rebates... So why look at retail prices?

Out of curiosity, does that also apply for consumer grade GPUs?

Anyone know if the GPUs are relatively affordable through Inception?
The 4090 Ti is rumored to have 48GB of VRAM, so one can only hope.
3060 12GB are the best things you can buy right now. They are cheap, have a ton of memory--which seems to be the issue w/ image generation--and you can fit four of them into the cheapest motherboards.

3060ti 8GB, 3090 24GB, and 4000 series all have performance benefits, but for now this one is off the charts.

With this card you can also run Open AI's Whisper with the Large model (the multilingual one!), as it requires 10GB.
Another one to consider is the A4000 16gb. I recently bought an ex-miner card for ~$500 usd . They are around a 3070 with a decent amount of memory for training scenarios, and are single slot cards. I believe there are a lot of these workstation ex miner cards which are pretty heavily discounted.

Combine this with a second hand X99 / 2011-v3 platform like the Dell Precision T7910 dual socket Xeons and you can have a pretty decent homelab for ML workloads. The Dell can come with a 1300 watt PSU and can fit 4 of those cards comfortably (5 with reduced PCIe lanes on one) since they are 150W each.

Also A2000 12GB, it's a slightly less powerful 3060, but it only requires 75 w of power, meaning you don't need to plug in a power cable.
Any suggestions on which Motherboard would be ideal for a 4x 3060 12GB setup?
For image inference 12GB is ok for now (but you may not be able to use all future models given that T5 language model is becoming popular), for training I'd consider 24GB the bare minimum.
A few weeks ago I got a 3060 12GB for $250 from a guy on FB Marketplace.
Anyone know if high-RAM Apple silicon such as the 128 GB M1 Ultra is useful for training large models? RAM seems like the limiting factor in DL, so I'm hoping apple can put some pressure on nvidia to offer consumer GPUs with more than 24GB RAM.
Having large amounts of unified memory is useful for training large models, but there are two problems with using Apple Silicon for model training:

1. Apple doesn't provide public interfaces to the ANE or the AMX modules, making it difficult to fully leverage the hardware, even from PyTorch's MPS runtime or Apple's "Tensorflow for macOS" package

2. Even if we could fully leverage the onboard hardware, even the M1 Ultra isn't going to outpace top-of-the-line consumer-grade Nvidia GPUs (3090Ti/4090) because it doesn't have the compute.

The upcoming Mac Pro is rumored to be preserving the configurability of a workstation [1]. If that's the case, then there's a (slim) chance we might see Apple Silicon GPUs in the future, which would then potentially make it possible to build an Apple Silicon Machine that could compete with an Nvidia workstation.

At the end of the day, Nvidia is winning the software war with CUDA. There are far, far more resources which enable software developers to write compute intensive code which runs on Nvidia hardware than any other GPU compute ecosystem out there.

Apple's Metal API, Intel's SYCL, and AMD's ROCm/HIP platform are closing the gap each year, but their success in the ML space is dependent upon how many people they can peel away from the Nvidia Hegemony.

1:https://www.bloomberg.com/news/newsletters/2022-12-18/when-w...

My totally user/curious 2 sats experience:

I've recently tried to play with Stable Diffusion. I have an RTX 3060, a base M1 MBP and a laptop with an AMD RX6800m.

- The RTX 3060 just works. Setup was straightforward, performance is good.

- The M1 has those neural engine thingies but it's not compatible with SD. It can run SD on CPU but if you want to make use of the NE you need coreml specific programs and models. Issue here is that it's just not the same stuff. Prompt structure is also different. It doesn't recognize weights and put a lot more emphasis on prompt order. Most of the time it seems to ignore most of what you wrote. On the positive side running stuff on the NE is very fast and doesn't seem to be taxing to the system.

- Finally the 6800M. It should be the powerhouse of the bunch considering its gaming performance ahead of the 3060. Problem is AMD toolkit kinda sucks. They have this obscure ROCm HIP stuff that acts as some kind of translation layer to the CUDA API. It's complicated, it simply doesn't work without a bunch of obscure environment variables and only works on fp32 mode, which means it uses twice amount of video RAM for the same thing. Support is iffy as they seem to only support workstation cards. Using it often throws lots of obscure compilation errors. Bugs celebrating anniversary.

To sum things up, Nvidia is far ahead of the curve in both usability and performance. Apple is trying to do its thing but it's too early yet while AMD is in a messy situation. Hope it helps.

> What is the carbon footprint of GPUs? How can I use GPUs without polluting the environment?

> I worked on a project that produced carbon offsets about ten years ago. The carbon offsets were generated by burning leaking methane from mines in China. UN officials tracked the process, and they required clean digital data and physical inspections of the project site. In that case, the carbon offsets that were produced were highly reliable. I believe many other projects have similar quality standards.

Crusoe Cloud (https://crusoecloud.com) does the same thing; powering GPUs off otherwise flared methane (and behind the meter renewables), to get carbon-reducing GPUs. A year of A100 usage offsets the equivalent emissions of taking a car off the road.

Disclosure: I run product at Crusoe Cloud

I really hope AMD cleans up and invests some money into their software stack. Granted it's really hard to catch up to Nvidia, but I think it's doable in ~5 years. The barrier to entry into rocm compared to cuda is pretty high, and even accounting for the fact that things got better in the last years. AMD has potent hardware for AI/ML (see instinct), they just don't have it in the consumer space. However one of the key factors of getting adoption in the consumer space is the fore-mentioned software stack, which I recall was a pain to setup. The fact that they're going FOSS for rocm shows promise in this regard.
Is a 4090 practically better than a 3090? I just built a new home DL PC with two 3090s because I knew I could fit them both in the case, whereas with the 4090 it seems more than one could be difficult. Also wondering if I can pool the RAM somehow, nvlink won't work because the 3090s are different sizes, and apparently nvlink doesn't do much more than pcie anyway.
Flow chart was nice but I am not an organization and I like training multi billion param models. My next two cards were going to be the be the rtx 6000 ada. The memory capacity alone almost makes it necessary.
Interesting no mention or discussion of FPGAs for DL Neural networks.

"Our enhanced NPU on Stratix 10 NX delivers 24× and 12× higher core compute performance on average compared to the T4 and V100 GPUs at batch-6, despite the smaller NX die size."

"Results show that the Stratix 10 NX NPU running batch 6 inference achieves 12-16× and 8-12× higher average energy efficiency (i.e. TOPS/Watt) on the studied workloads compared to the T4 and V100 GPUs, respectively."

https://users.ece.cmu.edu/~jhoe/distribution/2020/fpt2020.pd...

You can score 3090 for 800-900 used, with 24GB VRAM it's superb value
I hope I'm not getting downvoted into complete white on white for asking: Is there a good resource to learn myself a little ML for greater good, if I'm a complete math idiot? Something very practical, start with {this} and build an image recognition to tell birds from dogs. Or start with {that} and build a algo trading machine that will make me a trillionaire. I do have a 4090 in a windows machine which I can turn into a linux machine.
What's very interesting to me if i read the "slowdown vs power limit" graph right, is you can get 74% of the performance for half the power consumption?

And since I know nothing about deep learning:

Can you do these neural networks on AMD GPUs at all, regardless of performance? Because the hive mind says AMD GPUs are better if you run Linux.

Can you do these neural networks on Linux at all?

RTX 3090/3080 are great value for ML/DL especially at used prices.
Is the "Will AMD GPUs + ROCm ever catch up with NVIDIA GPUs + CUDA?" section accurate? It looks like at least partly an old version as it mentions upcoming 2020 hardware.
So how does training performance work when you don't have enough memory on the GPU? If you just use RAM instead, is the performance hit as simple as PCIE_Bandwidth divided by GPU_Memory_Bandwidth?

Example with 4090 would be 64GB/s and 1008GB/s respectively, so one could expect about 6% performance even when dealing with a model that is 48GB large (or more)?

Or are there second-order effects that cut this speed down even more?

I would love to see the NVIDIA T4 and A10 in the inference price comparison chart given their prevalence / cloud availability
A100 :)

It's cheaper to rent AWS instances and diagonalize a working set than throwing down coin for the latest/greatest GPU.

A bit shock. I purchased two 1080ti based in his recommendation many years ago. I looked at the chart. Is it really these cards are still there above the middle. Is there something wrong. Even the memory is not that bad.

Seems to move to 24G meant 3090 …

The big difference seems to be FP8 support in 4090 once it lands in CUDA 12 - if it gets a lot of adoption (and it should)
Why are transformers and large language models (which AFAIK use transformers, e.g Generative Pre-trained Transformer) in different boxes in the decision tree? They yield the same conclusion so I guess it's fine.
Thanks to Tim for writing and sharing this awesome analysis. I stumbled upon the previously updated version this summer, and so glad to see it updated with new hardware. Really great stuff and I learned a ton.
is there more context for: "Well, with the addition of the sparse matrix multiplication feature for Tensor Cores, my algorithm, or other sparse training algorithms, now actually provide speedups of up to 2x during training" 2x what? CPU, then it's hardly worth it. 2x per core, so #cores x 2. I'd imagine the level of sparsity is also important, but there is no mention of it.
My choice as hobbyist is 3060 12GB: cheap, 12GB vram, low thermals and noise. Cons: 2x time more training compared to 3090, which I am OK with.
Regarding pcie extenders - don’t think those work on gen 5 due to timing issues
All those FPS wasted... Why do we keep calling these chips GPUs?
tl;dr: used RTX 3060
I can't really judge the accuracy of the rest of the article, but if they can't even get the names of the GPUs they're recommending correct, it's tough to trust the rest of their supposedly thorough analysis. Throughout the whole article, the author refers to the "NVIDIA RTX 40 Ampere series", which is wrong. The RTX 30 series is Ampere [1] and the RTX 40 series is Ada [2].

1. https://en.wikipedia.org/wiki/Ampere_(microarchitecture) 2. https://en.wikipedia.org/wiki/Ada_Lovelace_(microarchitectur...