That's pretty easily fixable if you train a LoRA or similar so that the model has a specific likeness in mind. (You can look at - and despair at - Civitai if you want proof.)
It's harder to do at inference time without training, but I wouldn't assume it'll be impossible forever, especially with the existence of ControlNet.