Explaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders

ABSTRACT

Increasingly, algorithms are used to make important decisions across society. However, these algorithms are usually poorly understood, which can reduce transparency and evoke negative emotions. In this research, we seek to learn design principles for explanation interfaces that communicate how decision-making algorithms work, in order to help organizations explain their decisions to stakeholders, or to support users' "right to explanation". We conducted an online experiment where 199 participants used different explanation interfaces to understand an algorithm for making university admissions decisions. We measured users' objective and self-reported understanding of the algorithm. Our results show that both interactive explanations and "white-box" explanations (i.e. that show the inner workings of an algorithm) can improve users' comprehension. Although the interactive approach is more effective at improving comprehension, it comes with a trade-off of taking more time. Surprisingly, we also find that users' trust in algorithmic decisions is not affected by the explanation interface or their level of comprehension of the algorithm.

References

1. Ashraf Abdul, Jo Vermeulen, Danding Wang, Brian Y Lim, and Mohan Kankanhalli. 2018. Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 582.
2. Saleema Amershi, Max Chickering, Steven M Drucker, Bongshin Lee, Patrice Simard, and Jina Suh. 2015. Modeltracker: Redesigning performance analysis tools for machine learning. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 337--346.
3. Julia Angwin. 2016. Make algorithms accountable. The New York Times 1 (2016), 168.
4. Barry Becker, Ron Kohavi, and Dan Sommerfield. 2002. Visualizing the Simple Bayesian Classifier. In Information visualization in data mining and knowledge discovery. Morgan Kaufmann, 237--249.
5. Victoria Bellotti and Keith Edwards. 2001. Intelligibility and accountability: human considerations in context-aware systems. Human-- Computer Interaction 16, 2--4 (2001), 193--212.
6. Or Biran and Courtenay Cotton. 2017. Explanation and justification in machine learning: A survey. In IJCAI-17 Workshop on Explainable AI (XAI). 8. Explaining Decision-Making Algorithms through UI CHI 2019, May 4--9, 2019, Glasgow, Scotland Uk
7. Jenna Burrell. 2016. How the machine 'thinks': Understanding opacity in machine learning algorithms. Big Data & Society 3, 1 (2016), 2053951715622512.
8. Ángel Cabrera, Fred Hohman, Jason Lin, and Duen Horng Chau. 2018. Interactive Classification for Deep Learning Interpretation. arXiv preprint arXiv:1806.05660 (2018).
9. Doina Caragea, Dianne Cook, and Vasant G Honavar. 2001. Gaining insights into support vector machine pattern classifiers using projection-based tour methods. In Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 251--256.
10. Alexandra Chouldechova, Diana Benavides-Prado, Oleksandr Fialko, and Rhema Vaithianathan. 2018. A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions. In Conference on Fairness, Accountability and Transparency. 134--148.
11. Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. 2017. Algorithmic decision making and the cost of fairness. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 797--806.
12. Cynthia L Corritore, Beverly Kracher, and Susan Wiedenbeck. 2003. On-line trust: concepts, evolving themes, a model. International journal of human-computer studies 58, 6 (2003), 737--758.
13. Mark Craven and Jude Shavlik. 1999. Rule extraction: Where do we go from here. University of Wisconsin Machine Learning Research Group working Paper 99 (1999).
14. Michael A DeVito, Jeremy Birnholtz, and Jeffery T Hancock. 2017. Platforms, people, and perception: Using affordances to understand self-presentation on social media. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, 740--754.
15. Michael A DeVito, Jeffrey T Hancock, Megan French, Jeremy Birnholtz, Judd Antin, Karrie Karahalios, Stephanie Tong, and Irina Shklovski. 2018. The Algorithm and the User: How Can HCI Use Lay Understandings of Algorithmic Systems?. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, panel04.
16. Nicholas Diakopoulos. 2016. Accountability in algorithmic decision making. Commun. ACM 59, 2 (2016), 56--62.
17. Berkeley J Dietvorst, Joseph P Simmons, and Cade Massey. 2015. Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General 144, 1 (2015), 114.
18. Finale Doshi-Velez and Been Kim. 2017. A roadmap for a rigorous science of interpretability. arXiv preprint. arXiv preprint arXiv:1702.08608 (2017).
19. Julia Dressel and Hany Farid. 2018. The accuracy, fairness, and limits of predicting recidivism. Science advances 4, 1 (2018), eaao5580.
20. Motahhare Eslami, Karrie Karahalios, Christian Sandvig, Kristen Vaccaro, Aimee Rickman, Kevin Hamilton, and Alex Kirlik. 2016. First i like it, then i hide it: Folk theories of social feeds. In Proceedings of the 2016 cHI conference on human factors in computing systems. ACM, 2371--2382.
21. Megan French and Jeff Hancock. 2017. What's the folk theory? Reasoning about cyber-social systems. (2017).
22. Nahum Gershon. 1998. Visualization of an imperfect world. IEEE Computer Graphics and Applications 4 (1998), 43--45.
23. Bryce Goodman and Seth Flaxman. 2016. European Union regulations on algorithmic decision-making and a "right to explanation". arXiv preprint arXiv:1606.08813 (2016).
24. Spence Green, Jason Chuang, Jeffrey Heer, and Christopher D Manning. 2014. Predictive Translation Memory: A mixed-initiative system for human language translation. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 177--187.
25. Paul Hitlin. 2016. Research in the crowdsourcing age, a case study. Pew Research Center 11 (2016).
26. Fred Matthew Hohman, Minsuk Kahng, Robert Pienta, and Duen Horng Chau. 2018. Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers. IEEE Transactions on Visualization and Computer Graphics (2018).
27. Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. ACM, 159--166.
28. Aleks Jakulin, Martin Moina, Janez Demsar, Ivan Bratko, and Blaz Zupan. 2005. Nomograms for visualizing support vector machines. In Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining. ACM, 108--117.
29. Mary Beth Kery, Amber Horvath, and Brad A Myers. 2017. Variolite: Supporting Exploratory Programming by Data Scientists.. In CHI. 1265-- 1276.
30. Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. 2017. Human decisions and machine predictions. The quarterly journal of economics 133, 1 (2017), 237--293.
31. Josua Krause, Adam Perer, and Kenney Ng. 2016. Interacting with predictions: Visual inspection of black-box machine learning models. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 5686--5697.
32. John Lee and Neville Moray. 1992. Trust, control strategies and allocation of function in human-machine systems. Ergonomics 35, 10 (1992), 1243--1270.
33. Min Kyung Lee and Su Baykal. 2017. Algorithmic Mediation in Group Decisions: Fairness Perceptions of Algorithmically Mediated vs. Discussion-Based Social Division.. In CSCW. 1035--1048.
34. Brian Y Lim. 2012. Improving understanding and trust with intelligibility in context-aware applications. (2012).
35. Zachary C Lipton. 2016. The mythos of model interpretability. arXiv preprint arXiv:1606.03490 (2016).
36. David Martens and Foster Provost. 2013. Explaining data-driven document classifications. (2013).
37. Adam W Meade and S Bartholomew Craig. 2012. Identifying careless responses in survey data. Psychological methods 17, 3 (2012), 437.
38. Tim Miller. 2017. Explanation in artificial intelligence: insights from the social sciences. arXiv preprint arXiv:1706.07269 (2017).
39. Julian D Olden and Donald A Jackson. 2002. Illuminating the "black box": a randomization approach for understanding variable contributions in artificial neural networks. Ecological modelling 154, 1--2 (2002), 135--150.
40. Cathy O'Neil. 2016. Weapons of math destruction: How big data increases inequality and threatens democracy. Broadway Books.
41. Kayur Patel, Steven M Drucker, James Fogarty, Ashish Kapoor, and Desney S Tan. 2011. Using multiple models to understand data. In IJCAI Proceedings-International Joint Conference on Artificial Intelligence, Vol. 22. 1723.
42. Yi Peng, Guoxun Wang, Gang Kou, and Yong Shi. 2011. An empirical study of classification algorithm evaluation for financial risk prediction. Applied Soft Computing 11, 2 (2011), 2906--2915.
43. Tony A Plate, Joel Bert, John Grace, and Pierre Band. 2000. Visualizing the function computed by a feedforward neural network. Neural computation 12, 6 (2000), 1337--1353.
44. Donghao Ren, Saleema Amershi, Bongshin Lee, Jina Suh, and Jason D Williams. 2017. Squares: Supporting interactive performance analysis for multiclass classifiers. IEEE transactions on visualization and computer graphics 23, 1 (2017), 61--70.
45. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. Why should i trust you?: Explaining the predictions of any classifier. In CHI 2019, May 4--9, 2019, Glasgow, Scotland Uk Hao-Fei Cheng et al. Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. ACM, 1135--1144.
46. Daniel Smilkov, Shan Carter, D Sculley, Fernanda B Viégas, and Martin Wattenberg. 2017. Direct-manipulation visualization of deep networks. arXiv preprint arXiv:1708.03788 (2017).
47. Simone Stumpf, Vidya Rajaram, Lida Li, Weng-Keen Wong, Margaret Burnett, Thomas Dietterich, Erin Sullivan, and Jonathan Herlocker. 2009. Interacting meaningfully with machine learning systems: Three experiments. International Journal of Human-Computer Studies 67, 8 (2009), 639--662.
48. Sebastian Thrun. 1995. Extracting rules from artificial neural networks with distributed representations. In Advances in neural information processing systems. 505--512.
49. Meng-Jung Tsai, Ching-Yeh Wang, and Po-Fen Hsu. {n. d.}. Developing the Computer Programming Self-Efficacy Scale for Computer Literacy Education. Journal of Educational Computing Research ({n. d.}), 0735633117746747.
50. F-Y Tzeng and K-L Ma. 2002. Opening the black box-data driven visualization of neural networks. 383--390 pages.
51. Sandra Wachter, Brent Mittelstadt, and Luciano Floridi. 2017. Why a right to explanation of automated decision-making does not exist in the general data protection regulation. International Data Privacy Law 7, 2 (2017), 76--99.
52. Lauren Weber and RE Silverman. 2012. Your resume vs. oblivion. The Wall Street Journal 24 (2012).
53. Daniel S Weld and Gagan Bansal. 2018. Intelligible Artificial Intelligence. arXiv preprint arXiv:1803.04263 (2018).
54. Ann Wilkinson, Alison E While, and Julia Roberts. 2009. Measurement of information and communication technology experience and attitudes to e-learning of students in the healthcare professions: integrative review. Journal of advanced nursing 65, 4 (2009), 755--772.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • Empirical studies in HCI

    • Interactive systems and tools