back

by raphlinus·2y ago·view on hn ↗
ROCm makes me sad, as it reminds me of how much better GPUs could be than they are today.

I've lately been exploring the idea of a "Good Parallel Computer," which combines most of the agility of a CPU with the efficient parallel throughput of a GPU. The central concept is that the decision to launch a workgroup is made by a programmable controller, rather than just being a cube of (x, y, z) or downstream of triangles. A particular workload it would likely excel at is sparse matrix multiplication, including multiple quantization levels like SpQR[1]. I'm hopeful that it could be an advance in execution model, but also a simplification, as I believe a lot of the complexity of the current GPU model is because of lots of workarounds for the weak execution model.

I'm not optimistic about this being built any time soon, as it requires rethinking the software stack. But it's fun to think about. I might blog about it at some point, but I'm also interested in connecting with people who have been thinking along similar lines.

[1]: https://arxiv.org/abs/2306.03078

3 comments
A workgroup/kernel can launch other ones without talking to the host. Like cuda's dynamic thing except with no nested lifetime restrictions. This is somewhat documented under the name HSA.

Involves getting a pointer to a HSA queue and writing a dispatch packet to it. Same interface the host has for launching kernels - easier in some ways (you've got the kernel descriptor as a symbol, not as a name to dlsym) and harder in others (dynamic memory allocation is a pain).

Yeah, dynamic memory allocation from GPU space seems to be the real sticking point. I'll look into HSA queues, that looks very interesting, thanks.
That's solved too. But as usual there's elements of DIY. The host runtime can allocate memory that is read/write by the host and by GPUs in atomic operation fashion. If you're on pci-e that means load/store/cas/swap/fetch-add. Mutable shared memory is sufficient for arbitrary exchange of information, e.g. a GPU kernel asking the host to allocate some GPU memory and give it the corresponding pointer.

Implementing robust cross device function calls on that was fairly tough going, but these days you could rip the code with 'rpc' in the file name out of the llvm libc implementation where it underpins the GPU equivalent of syscall.

Non-cuda style programming models on GPUs is a pet interest of mine, feel free to email if you want to talk offline.

I heard Unreal Nanite built a job queue system on compute threads (https://www.youtube.com/watch?v=eviSykqSUUw&t=1611s), would that help with your use case or not?
How does this differ from CUDA’s dynamic parallelism, which lets you launch kernels from within a kernel?
There are a lot of similarities, but the granularity is finer. The idea is that you make a decision to launch one workgroup (typically 1024 threads) when the input is available, which would typically be driven by queues, and potentially with joins as well, which is something the new work graph stuff can't quite do. Otherwise the idea of stages running in parallel, connected by queues, is similar. But I did an analysis of work graphs and came to the conclusion that it wouldn't help with the Vello (2d vector graphics) workload at all.
You can do device graph launch in cuda

https://developer.nvidia.com/blog/enabling-dynamic-control-f...