Really what it seems to say is that LLMs are pretty good at identifying underlying causes and recommending medical actions but if you let humans use LLMs to self diagnose the whole thing falls apart, if I read this correctly
back
2 comments
Yeah it sounds like "LLMs are bad at interacting with lay humans compared to being prompted by experts or being given well-formed questions like from licensing exams."
Feels to me like how two years ago "prompt engineering" got a bunch of hype in tech companies, and now is nonexistent because the models began being trained and prompted specifically to mimic "reasoning" for the sorts of questions tech company users had. Seems like that has not translated to reasoning their way through the sort of health conversations a non-medical-professional would initiate.
And there seem to be concrete results that would allow you to improve the LLM prompt to make these interactions more successful. Apparently giving the human 2-3 possible options and letting the human have the final choice was a big contributor to the bad results. Their recommendations go the route of "the model should explain it better" but maybe the best results would be achieved if the model was prompted to narrow it down until there is only one likely diagnosis left. This is more or less how doctors operate after all.