The openai/whisper repo and paper referenced by the model card seem to be saying it's transformer based.