End users even have problems interpreting probability statements in my experience (e.g. this factor increases probability of event by 10%) let alone graphs. When you get more complicated models, those probabilities are often spread out among many factors, so that just adds a layer of complexity even well educated (but not quants) can handle.
XAI in general is maybe even more basic than these reduced form summaries IMO. I know alot of "data scientists" copy-pasting code that couldn't describe their deep learning architecture coherently, or who don't know how their variables are encoded in an XGBoost model, or even what exactly they are predicting.
I think those are probably bigger hurdles in general than being able to peak into black box methods and get reduced form summaries of variable effects or importance or whatever.