Yeah, currently I do things like "Top 5 important factors" for shapely values, but have debated on even doing that much. There is good evidence these reduced form estimates can often be misleading, https://arxiv.org/abs/1811.10154. So it may just be a hopeless task anyway.
back
2 comments
I’m curious, do end users prefer text or graphs such as SHAP’s force diagram?
Also, I keep hearing about XAI but haven’t seen one in the real world applications. Do you happen to know any examples?
End users even have problems interpreting probability statements in my experience (e.g. this factor increases probability of event by 10%) let alone graphs. When you get more complicated models, those probabilities are often spread out among many factors, so that just adds a layer of complexity even well educated (but not quants) can handle.
XAI in general is maybe even more basic than these reduced form summaries IMO. I know alot of "data scientists" copy-pasting code that couldn't describe their deep learning architecture coherently, or who don't know how their variables are encoded in an XGBoost model, or even what exactly they are predicting.
I think those are probably bigger hurdles in general than being able to peak into black box methods and get reduced form summaries of variable effects or importance or whatever.
In some applications (e.g., bank loans) it might be more suitable to offer counterfactuals. like "your loan application would've been accepted if your income were $5,000 higher". I think users care more about this kind of XAI because that gives them an idea about what to do next, whereas dealing with SHAP values or probabilities doesn't give that much insight, as you also pointed out.
That said, I have never seen any explanation for AI predictions/decisions in day-to-day life. Even clicking on Google Ads shows "this ad was shown to you based on your past searches"! Basically zero XAI.
I think probably any post-hoc explanations will be wrong. Explainability needs to be built into the model from the outset.