That's solved too. But as usual there's elements of DIY. The host runtime can allocate memory that is read/write by the host and by GPUs in atomic operation fashion. If you're on pci-e that means load/store/cas/swap/fetch-add. Mutable shared memory is sufficient for arbitrary exchange of information, e.g. a GPU kernel asking the host to allocate some GPU memory and give it the corresponding pointer.

Implementing robust cross device function calls on that was fairly tough going, but these days you could rip the code with 'rpc' in the file name out of the llvm libc implementation where it underpins the GPU equivalent of syscall.

Non-cuda style programming models on GPUs is a pet interest of mine, feel free to email if you want to talk offline.