Increasing Trust in Machine Learning Models: Explainability, User Interpretations, and Trustworthiness

Oregon Convention Center 

Little analysis has been performed to determine if machine learning (ML) explanations accurately represent the target model and should be trusted beyond subjective inspection. Many state-of-the-art ML explainability (MLE) techniques only provide a list of important features based on heuristic measures or make assumptions about the data and the model which are not representative in the real-world. Further, most are designed without considering the usefulness by an end-user in a broader context. To address these issues, we present a notion of explanation fidelity based on Shapley values from cooperative game theory and find many MLE explainability methods produce explanations that are incongruent with the ML model that is being explained. We also find that in deployed scenarios, explanations are rarely used. In the cases when the explanations are used, there is danger that explanations persuade end users to wrongly accept false positives and false negatives.