This has already been tried, llama pioneered it (as far as I can infer from public knowledge, maybe openai did it years ago I don't know).

They looped through a bunch of wikipedia pages, made questions out of the info given there, posed them to the LLM and then whenever the answer did not match what was in wikipedia, they went ahead and finetuned on "that question: Sorry I don't know ...".

Then, we went one step ahead, and finetuned it to use search in these cases instead of saying I don't know. Finetune it on the answer toolCall("search", "that question", ...) or whatever.

Something close to the above is how all models with search tool capability are fine tuned.

All these hallucinations are despite those efforts, it was much worse before.

This whole method depends on the assumption that there is actually a path in the internal representation that fires when it's gonna hallucinate. The results so far tell us that it is partially true. No way to quantify it of course.