▲ 221 points
back
172 comments
> Give AMD Engineers more compute and engineering resources to fix and improve the AMD ecosystem, they have very few internal gpu boxes relative to what Nvidia provides to their engineers.
This is real. We’ve found ourselves having to give hardware to engineers at AMD because they’re unable to get allocation of it internally.
This is baffling. I’m sure there are many technical reasons I don’t grok that AMD’s job is challenging, but it’s wild that they are dropping the ball on such obvious stuff as this.
The prize is trillions of dollars, and they can print hundreds of millions if they can convince the market that they are closing the gap.
It’s embarrassing that whoever actually tries to use their product hits these crass bugs (same with geohot who was really invested in making AMD’s cards work; I think he just ran their demo script in a loop and produced crashes).
It seems they really don’t understand/value the developer flywheel.
Sadly common at hardware companies. The most extreme case I've heard of is ASML, who supposedly doesn't keep any machines of their own. They test against "almost-ready" machines right before they go out the door to customers.
Coming up next: "We bought AMD stock on the open market and used it to compensate AMD engineers".
This seems so insane, is anyone actually doing the work to provide an alternative to CUDA? Maybe Google?
I was surprised to hear recently that the same happens at NVIDIA! Hopefully less frequently, but I can understand why it's hard to keep many units on hand given the level of external demand.
> AMD is attempting to vertically integrate next year with their upcoming Pollara 400G NIC, which supports Ultra Ethernet, hopefully making AMD competitive with Nvidia.
Infiniband is an industry standard. It is weird to see the industry invent yet another standard to do effectively the same thing just because Nvidia is using it. This “Nvidia does things this way so let’s do it differently” mentality is hurting AMD:
* Nvidia has a unified architecture so let’s split ours into RDNA and CDNA.
* Nvidia has a unified driver, so let’s make a different driver for every platform.
* Nvidia made a virtual ISA (PTX) for backward compatibility. Let’s avoid that.
* Nvidia is implementing tensor cores. Let’s avoid those on RDNA. Then implement them on CDNA and call them matrix cores.
* Nvidia is using Infiniband like the rest of the HPC community. Let’s use Ethernet.
You're painting this like AMD is off to play in their own sandbox when it's more like the entire industry is trying to develop an alternative to Infiniband.
Ultra Ethernet is a joint project between dozens of companies organized under the Linux Foundation.
https://www.phoronix.com/news/Ultra-Ethernet-Consortium
>> The Linux Foundation has established the Ultra Ethernet Consortium "UED" as an industry-wide effort founded by AMD, Arista, Broadcom, Cisco, Eviden, HPE, Intel, Meta, and Microsoft for designing a new Ethernet-based communication stack architecture for high performance networking.
You probably can't call it "industry standard" yet but the goal is obviously for it to become one.
Infiniband is a monopoly from NVidia (Mellanox). Everyone else would much rather use Ethernet which is the actual industry standard.
> Infiniband is an industry standard
Infiniband is not an industry standard lol.
Maybe it used to be, but it definitely is not anymore. Most Infiniband vendors are dead. The only product from those days that endures is Cornelis' Omnipath, and even that only emulated the Infiniband API back with its first gen, and then evolved to be its own thing.
At this point, Infiniband is as good as a proprietary interconnect only sold by Nvidia/Mellanox.
Nvidia has a unique problem, wants to move fast and has a shit load of money.
No need for Nvidia to go first to an industry standard and neither for AMD.
Personally would be great its getting backported but its so far away from an normal use case.
You hardly beat someone by copying him. They have way more experience in the field you try to catch up.
I made the mistake of clicking on one of the links to commits they mentioned only to end up at a MR changing multiple autogenerated yaml files with 10k line diffs and incomprehensible names. I guess this is where the whole "bad talent" thing comes in - a year later and you are thousands of YAML files deep but still no one can run a simple PyTorch compile ops and get the performance you sold, absolutely unhinged.
That MatMul performance is fairly shocking. To be that much below theoretical maximum on what should be a fairly low overhead operation.
I would at least hope that they know where the speed is going, but the issue of torch.matmul and F.Linear using different libraries with different performance suggests that they don't even know which code they are running, let alone where the slow bits in that code are.
Low overhead in what sense? matmul is kinda complicated and there are varying, complex state-of-the-art algorithms for it, no? And then if you know things about the matrices in advance you can start optimizing for that, which adds another layer of complexity.
> It’s not just that it’s immature software, they need to change how they do development.
I remember geohot saying something similar about a year ago
I expect everyone has been saying it for a while, the calls are just getting more strident and public as it becomes clear that AMD's failures are strategic rather than tactical. And as people try to build business on their half-hearted attempts.
I still think it is a mistake to say that CUDA is a moat. IMO the problem here is that AMD still doesn't seem to think that GPGPU compute is a thing. They don't seem to understand the idea that someone might want to use their graphics cards to multiply matricies independently of a graphics pipeline. All the features CUDA supports are irrelevant compared to the fact that AMD can't handle GEMM performantly out of the box. In my experience it just can't do it, back in the day my attempts to multiply matrices would crash drivers. That isn't a moat, but it certainly is something spectacular.
If they could manage an engineering process that delivered good GEMM performance then the other stuff can probably get handled. But without it there really is a question of what these cards are for.
Anyone who looks at the mess that is ROCm and the design choices they made could easily see that.
GPU support lagged behind for years, no support for APUs and no guaranteed forward compatibility were clear signs that as a whole they have no idea what they are doing when it comes to building and shipping a software ecosystem.
To that you can add the long history of both AMD and ATI before they merged releasing dog shit software and then dropping support for it.
On the other hand you can take any CUDA binary even one that dates back to the original Tesla and run it on any modern NVIDIA GPU.
"The software needs to be better" is (and was) an easy call to make for anyone paying attention. The problem is that "AMD just needs to do better" is not and will never be an implementable strategy. Engineering isn't just about money. It's also about the process of exploring all the edge cases.
"We recommend that AMD to fix their GEMM libraries’ heuristic model such that it picks the correct algorithm out of the box instead of wasting the end user’s time doing tuning on their end." Is such a profoundly unhelpful thing to say unless you imagine AMDs engineers just sitting around wondering what to do all day.
AMD needs to make their drivers better, and they have. Shit just takes time.
In buggy numerical code many bugs go trough the software stack without any problems. No crash, no errors. For example,you might switch two double parameters to a function and if their value range is similar, everything works fine except it's all bullshit.
If there are bugs in AMD code that prevent running tests, I bet there are even more bugs that don't manifest until you look at results.
I once tried installing AMD ROCM to run a small llm on a consumer-grade AMD GPU. It was the most horrible software install experience I ever had. Never did manage to get it working.
What I couldn't find is inference benchmarks for consumer hardware. Just pick a reasonable workload with llama.cpp or ollama and show us some numbers.
I'm particularly interested in building a Home Assistant machine that can run the voice assistant locally (STT/TTS/LLM) while using the least amount of power / generating the least amount of heat and noise.
AMD's software for consumer GPUs demonstrates a lack of seriousness. ROCm only officially supports RDNA2 and RDNA3 GPUs (their last two generations of hardware), and for some reason most of them are supported on only Windows (https://rocm.docs.amd.com/projects/install-on-windows/en/lat...) and not Linux (https://rocm.docs.amd.com/projects/install-on-linux/en/lates...), where most AI training and inference occurs. In particular, Linux users can only start playing with ROCm with a top-of-the-line, power-guzzling unit whereas they can get started with CUDA using basically any Nvidia GPU on desktops or laptops.
Based on a machine we had bought at my university with 4 AMD W6800s (which are just RX 6800s with double the VRAM), it's bad _even if it works at all_.
It would be cool to see these benchmarks on the newly released Jetson Orin Nano Super, like faster-whisper.
You might just check out the Home Assistant Voice:
https://ameridroid.com/products/home-assistant-voice-preview...
My anecdata on AMD hiring: they just aren't moving fast enough. They still wanted to fly people out scheduling 3 weeks in advance for AI compiler work. That's just not going to work. Startups and companies like NVIDIA, OpenAI are hiring much faster with much less onerous interview processes, with higher compensation. This is not a mystery. People work for money and aren't going to hop through more hoops to be paid less.
Latest: Dylan Patel ( SemiAnalysis )
"Met with @LisaSu today for 1.5 hours as we went through everything
She acknowledged the gaps in AMD software stack
She took our specific recommendations seriously
She asked her team and us a lot of questions
Many changes are in flight already!
Excited to see improvements coming"
Lisa Su : https://x.com/LisaSu/status/1871362304194859511
"Thanks @dylan522p for the constructive conversation today. Feedback is a gift even when it’s critical. We have put a ton of work into customer and workload optimizations but there is lots more we can do to enable the broad ecosystem. I appreciate all the feedback and desire to engage with @AMD. We are committed to building a world-class open software stack. Lots planned for 2025. Happy holidays to all!"
AMD could spend their market cap in one year to get this done in three and it would be a coup for the shareholders. They could hire all of the best NVIDIA engineers at double their current comp, crush the next TSMC node on Apple levels, and just do it and if it got them a quarter of NVDA’s cap it would be a bargain.
They don’t fucking want to! Believing this is anything like a market is fucking religion.
You make it sound like that's a sure thing, but I doubt it. A lot of this is about processes, team structures and incentives, all those fuzzy things between the people.
Remember, most acquisitions fail. For the same reason, the likelihood of failure with your scenario seems high.
Do you really think nobody at AMD is aware of all the points made in this thread? That seems too bizarre to be true. There are probably some issues in upper management which could perhaps be fixed with some targeted hiring decisions, but do you really believe some random person on here would have a chance making that call?
Try to make sense...
They can spend their market cap by either:
1: issuing new shares worth their market cap, diluting existing shareholders to 50%.
2: Or borrow their market cap and pay interest by decreasing profits. "AMD operating margin for the quarter ending September 30, 2024 was 5.64%" so profits would be extremely impacted by interest repayments.
Either way your suggestion would be unlikely to be supported by shareholders.
> crush the next TSMC node on Apple levels
I would guess Apple is indirectly paying for the hardware (to avoid repatriating profits) or guaranteeing usage to get to the front of the line at TSMC. Good luck AMD competing with Apple: there's a reason AMD sold GlobalFoundries and there's a reason Intel is now struggling with their foundry costs.
And it comes across as condescending to assume you know better than a successful company.
I think it’s hardware not software.
“Cuda moat” is a misnomer. The PTX spec is relatively short (600 page pdf). Triton directly writes PTX, skipping cuda. Flash attention was created by a non nvidia employee without access to any of the secret sauce within Cuda or its libraries.
The hardware is just not as good, and no software can paper over its flaws.
NVidia also refactored their hardware design to follow C++ memory model, I think this still isn't the case of others.
So basically Nvidia is like Windows and AMD is like Wine. I think trying to emulate CUDA and using forked Nvidia libraries is not the best strategy for AMD. They should have made a clean break and come out with a fresh API, like Apple's Metal.
AMD doesn’t just have to fix these issues it has to build up a record of fixing issues like those discussed here.
Otherwise who will bet their firm / cash / career on new hardware without a successful track record.
B100, B200 ramp-up is only 4-6 months away.
Wow they really botched the title. Why wasn’t it “Still Filled With Water”?
> CUDA Moat Still Alive
Wrong conclusion. AMD is slower than NVidia, but not _that_ much slower. They are actually pretty cost-competitive.
The just need to do some improvements, and they'll be a very viable competitor.
The amount of effort this team took, literally co-opting AMD engineers, and working for 5 months, to get closer but not yet usable, means they are not even close to usable. What team wanting to do ML training/inference can afford so much down time for zero benefit? How many except a few big ones can get AMD to devote so many resources simply for that team?
And, if you’re training a model costing you millions, the last thing you need is a buggy, untested stack, breaking training or perhaps worse giving you noise that makes your models perform worse or increases training time.
By the time AMD gets usable out of the box at this point, NVidia will have moved further ahead.
All the cloud providers list MI300x as more expensive than H100. So if you compare performance/cost it is even worse.
Just like that? So a little work and now they are competitive. You know how much work “just a little bit of work” is doing? They us a cultural issue and it will take months to fix if they are lucky and then you start tackling the tech debt they’ve built up. By that time it will be another generation
Back in the late 1990s I met the ATI guys, and they were slipshod then as well. That the ATI legacy of special-casing things lives on is sadly, not too surprising for me.
Sounds like a buy signal for AMD. If you run the right branch and set the right env cars, the thing flies.
Disappointed that there wasn’t anything on inference performance in the article at all. That’s what the major customers have announced they use it for.
TL;DR: AMD still doesn't take software seriously?