Yes, CUDA is their moat. And it's a very deep moat, filled with sharks, alligators, and mines. It will take a truly massive effort to unseat CUDA and NVIDIA as the dominant software/hardware tools for deep learning.

I'm curiously watching Intel and their OneAPI platform; but it just feels like they're starting from so far behind. And Intel hasn't exactly had a stellar few years on top of that.