Not directly related to this API, but I’ve begun wondering if the next revolution in speech synthesis is to integrate with a natural language model like gpt-3 in order to gain semantic awareness, and use that context to produce emotional expressiveness and inflection that is attuned to the meaning and tone of the text.
back
2 comments
Imho, the next revolution in speech synthesis will come from using guided diffusion models, leveraging the recent breakthrough in image synthesis (Dall-e), to generate spectrograms (spectrograms are images).
Using this slower generative approach it will allow to produce large high-quality enriched audio datasets with parametric text, timings and emotion.
Then you use these datasets to bootstrap in a supervised fashion the existing traditional architectures to make the generation faster.
The usual problem of text-to-speech is that you have to go from a low-information space (aka text) to a high-information space (aka sound). And therefore training is ill-defined because one input text can have several correct sound. But once you have an enriched text with inflections and parameters, speaker embedding, the mapping then become one enriched text to one exact audio and the training become well-defined and easy.
Everyone's working on emotional expressiveness right now. Many researches published left & right!
Cool - I probably shouldn’t be surprised that the smart minds in machine learning and natural language processing are well ahead of me, an interested lay person!
Any good links or HN comment threads that you’d recommend?