This technique can be added on to any encoder–decoder Transformer model post-training, so the added training difficulties you mention don't apply. It honestly is a very interesting approach to me – the main issue I see (which they discuss in the paper) is in pure latency. If you're using a large enough vector database, it will be on the CPU, and transferring hidden states from GPU to CPU and then the embeddings back from CPU to GPU is going to eat up a ton of time.