Related to understanding how ML models are making their decisions is the concept of influence [1] (or broadly, instance based explanations) [2] where the training data that is most influential in a prediction (basically the analogy the model used) is highlighted. See also [3] where I provide a take on using these methods for understanding if you can trust a prediction or not, based on whether the model has an appropriate analogy to work from

[1] Understanding black box predictions through influence functions, https://arxiv.org/abs/1703.04730 [2] Evaluation of Similarity-based Explanations, https://arxiv.org/abs/2006.04528 [3] https://www.willows.ai/blog/getting-more