To explain or not to explain: the effects of personal characteristics when explaining music recommendations

ABSTRACT

Recommender systems have been increasingly used in online services that we consume daily, such as Facebook, Netflix, YouTube, and Spotify. However, these systems are often presented to users as a "black box", i.e. the rationale for providing individual recommendations remains unexplained to users. In recent years, various attempts have been made to address this black box issue by providing textual explanations or interactive visualisations that enable users to explore the provenance of recommendations. Among other things, results demonstrated benefits in terms of precision and user satisfaction. Previous research had also indicated that personal characteristics such as domain knowledge, trust propensity and persistence may also play an important role on such perceived benefits. Yet, to date, little is known about the effects of personal characteristics on explaining recommendations. To address this gap, we developed a music recommender system with explanations and conducted an online study using a within-subject design. We captured various personal characteristics of participants and administered both qualitative and quantitative evaluation methods. Results indicate that personal characteristics have significant influence on the interaction and perception of recommender systems, and that this influence changes by adding explanations. For people with a low need for cognition are the explained recommendations the most beneficial. For people with a high need for cognition, we observed that explanations could create a lack of confidence. Based on these results, we present some design implications for explaining recommendations.

References

1. Azzah Al-Maskari and Mark Sanderson. 2011. The effect of user characteristics on search effectiveness in information retrieval. Information Processing & Management 47, 5 (2011), 719--729.
2. Ivana Andjelkovic, Denis Parra, and John O'Donovan. 2016. Moodplay: Interactive Mood-based Music Discovery and Recommendation. In Proc. of UMAP '16. ACM, 275--279.
3. Nuray M Aykin and Turgut Aykin. 1991. Individual differences in human-computer interaction. Computers & industrial engineering 20, 3 (1991), 373--379.
4. Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015. Fitting Linear Mixed-Effects Models Using lme4. Journal of Statistical Software 67, 1 (2015), 1--48.
5. Svetlin Bostandjiev, John O'Donovan, and Tobias Höllerer. 2012. TasteWeights: a visual interactive hybrid recommender system. In Proc. of RecSys'12. ACM, 35--42.
6. Jeremy Boy, Ronald A Rensink, Enrico Bertini, and Jean-Daniel Fekete. 2014. A principled way of assessing visualization literacy. IEEE transactions on visualization and computer graphics 20, 12 (2014), 1963--1972.
7. Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77--101.
8. Peter Brusilovsky and Eva Millán. 2007. User models for adaptive hypermedia and adaptive educational systems. In The adaptive web. Springer, 3--53.
9. Andrea Bunt, Joanna McGrenere, and Cristina Conati. 2007. Understanding the utility of rationale in a mixed-initiative system for GUI customization. In International Conference on User Modeling. Springer, 147--156.
10. John T Cacioppo, Richard E Petty, and Chuan Feng Kao. 1984. The efficient assessment of need for cognition. Journal of personality assessment 48, 3 (1984), 306--307.
11. Giuseppe Carenini, Cristina Conati, Enamul Hoque, Ben Steichen, Dereck Toker, and James Enns. 2014. Highlighting interventions and user differences: informing adaptive information visualization support. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems. ACM, 1835--1844.
12. Li Chen and Pearl Pu. 2005. Trust building in recommender agents. In Proceedings of the Workshop on Web Personalization, Recommender Systems and Intelligent User Interfaces at the 2nd International Conference on E-Business and Telecommunication Networks. Citeseer, 135--145.
13. Mei C Chuah. 1998. Dynamic aggregation with circular visual designs. In Information Visualization, 1998. Proceedings. IEEE Symposium on. IEEE, 35--43.
14. Cristina Conati, Giuseppe Carenini, Enamul Hoque, Ben Steichen, and Dereck Toker. 2014. Evaluating the impact of user characteristics and different layouts on an interactive visualization for decision making. In Computer Graphics Forum, Vol. 33. Wiley Online Library, 371--380.
15. Cristina Conati, Giuseppe Carenini, Dereck Toker, and Sébastien Lallé. 2015. Towards user-adaptive information visualization. In Proc. of AAAI '15. AAAI Press, 4100--4106.
16. Ipshita Dewan and Pierre Benckendorff. 2013. Impact of Tech Savviness and impulsiveness on the mobile information search behaviour of young travellers. Information and communications technologies in tourism (2013).
17. Gitta O Domik and Bernd Gutkauf. 1994. User modeling for adaptive visualization systems. In Visualization, 1994., Visualization'94, Proceedings., IEEE Conference on. IEEE, 217--223.
18. G. Fournier. 2018 (Retrieved on September 16, 2018). Locus of Control. Psych Central. https://psychcentral.com/encyclopedia/locus-of-control/
19. Roger W Geyer. 2009. Developing the internet-savviness (IS) scale: Investigating the relationships between internet use and academically talented middle school youth. RMLE Online 32, 5 (2009), 1--20.
20. Liang Gou, Fang You, Jun Guo, Luqi Wu, and Xiaolong Luke Zhang. 2011. Sfviz: interest-based friends exploration and recommendation in social networks. In Proc. VINCI '11. ACM, 15.
21. David M Greenberg, Daniel Müllensiefen, Michael E Lamb, and Peter J Rentfrow. 2015. Personality predicts musical sophistication. Journal of Research in Personality 58 (2015), 154--158.
22. Brynjar Gretarsson, John O'Donovan, Svetlin Bostandjiev, Christopher Hall, and Tobias Höllerer. 2010. Smallworlds: visualizing social recommendations. In Computer Graphics Forum, Vol. 29. Wiley Online Library, 833--842.
23. Jonathan L Herlocker, Joseph A Konstan, and John Riedl. 2000. Explaining collaborative filtering recommendations. In Proceedings of the 2000 ACM conference on Computer supported cooperative work. ACM, 241--250.
24. Yosef Hochberg and Yoav Benjamini. 1990. More powerful procedures for multiple significance testing. Statistics in medicine 9, 7 (1990), 811--818.
25. Yucheng Jin, Karsten Seipp, Erik Duval, and Katrien Verbert. 2016. Go with the flow: effects of transparency and user control on targeted advertising using flow charts. In Proc. of AVI '16. ACM, 68--75.
26. Yucheng Jin, Nava Tintarev, and Katrien Verbert. 2018. Effects of individual traits on diversity-aware music recommender user interfaces. In Proceedings of the 26th Conference on User Modeling, Adaptation and Personalization. ACM, 291--299.
27. Yucheng Jin, Nava Tintarev, and Katrien Verbert. 2018. Effects of personal characteristics on music recommender systems with different levels of controllability. In Proceedings of the 12th ACM Conference on Recommender Systems. ACM, 13--21.
28. Roy PC Kessels, Martine JE Van Zandvoort, Albert Postma, L Jaap Kappelle, and Edward HF De Haan. 2000. The Corsi block-tapping task: standardization and normative data. Applied neuropsychology 7, 4 (2000), 252--258.
29. Bart P Knijnenburg, Niels JM Reijmer, and Martijn C Willemsen. 2011. Each to his own: how different users call for different interaction methods in recommender systems. In Proc. of RecSys'11. ACM, 141--148.
30. Bart P Knijnenburg, Martijn C Willemsen, Zeno Gantner, Hakan Soncu, and Chris Newell. 2012. Explaining the user experience of recommender systems. User Modeling and User-Adapted Interaction 22, 4-5 (2012), 441--504.
31. Alexandra Kuznetsova, Per B. Brockhoff, and Rune H. B. Christensen. 2017. lmerTest Package: Tests in Linear Mixed Effects Models. Journal of Statistical Software 82, 13 (2017), 1--26.
32. Sébastien Lallé, Cristina Conati, and Giuseppe Carenini. 2017. Impact of Individual Differences on User Experience with a Visualization Interface for Public Engagement. In Proc. of UMAP '17. ACM, 247--252.
33. Moira Maguire and Brid Delahunt. 2017. Doing a thematic analysis: A practical, step-by-step guide for learning and teaching scholars. AISHE-J: The All Ireland Journal of Teaching and Learning in Higher Education 9, 3 (2017).
34. Martijn Millecamp, Nyi Nyi Htun, YuchengJin, and Katrien Verbert. 2018. Controlling Spotify recommendations: effects of personal characteristics on music recommender user Interfaces. In Proceedings of the 26th Conference on User Modeling, Adaptation and Personalization. ACM, 101--109.
35. Akira Miyake and Priti Shah. 1999. Models of working memory: Mechanisms of active maintenance and executive control. Cambridge University Press.
36. Daniel Müllensiefen, Bruno Gingras, Jason Musil, and Lauren Stewart. 2014. The musicality of non-musicians: an index for assessing musical sophistication in the general population. PloS one 9, 2 (2014), e89642.
37. John O'Donovan, Barry Smyth, Brynjar Gretarsson, Svetlin Bostandjiev, and Tobias Höllerer. 2008. PeerChooser: visual interactive recommendation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1085--1088.
38. Thomas V Perneger. 1998. What's wrong with Bonferroni adjustments. Bmj 316, 7139 (1998), 1236--1238.
39. Pearl Pu, Li Chen, and Rong Hu. 2011. A user-centric evaluation framework for recommender systems. In Proceedings of the fifth ACM conference on Recommender systems. ACM, 157--164.
40. R Core Team. 2017. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/
41. Julian B Rotter. 1966. Generalized expectancies for internal versus external control of reinforcement. Psychological monographs: General and applied 80, 1 (1966), 1.
42. Rashmi Sinha and Kirsten Swearingen. 2002. The role of transparency in recommender systems. In CHI'02 extended abstracts on Human factors in computing systems. ACM, 830--831.
43. Nava Tintarev. 2017. Presenting Diversity Aware Recommendations: Making Challenging News Acceptable. In Proc. of FATREC 17'.
44. Nava Tintarev and Judith Masthoff. 2007. A survey of explanations in recommender systems. In Data Engineering Workshop, 2007 IEEE 23rd International Conference on. IEEE, 801--810.
45. Nava Tintarev and Judith Masthoff. 2016. Effects of Individual Differences in Working Memory on Plan Presentational Choices. Frontiers in psychology 7 (2016).
46. Dereck Toker, Cristina Conati, Giuseppe Carenini, and Mona Haraty. 2012. Towards adaptive information visualization: on the influence of user characteristics. In International Conference on User Modeling, Adaptation, and Personalization. Springer, 274--285.
47. Stephanie Tom Tong, Elena F Corriero, Robert G Matheny, and Jeffrey T Hancock. 2018. Online Daters' Willingness to Use Recommender Technology for Mate Selection Decisions. In Proceedings of the 5th Joint Workshop on Interfaces and Human Decision Making for Recommender Systems co-located with ACM Conference on Recommender Systems (RecSys 2018). ACM, 45--52.
48. Chun-Hua Tsai and Peter Brusilovsky. 2017. Enhancing Recommendation Diversity Through a Dual Recommendation Interface. In Proc. of RecSys IntRS'17. 10.
49. Chun-Hua Tsai and Peter Brusilovsky. 2018. Beyond the Ranked List: User-Driven Exploration and Diversification of Social Recommendation. In 23rd International Conference on Intelligent User Interfaces. ACM, 239--250.
50. AJAM Van Deursen, Ellen J Helsper, and R Eynon. 2014. Measuring digital skills. From digital skills to tangible outcomes. Project Report. Recuperado de: www.oii.ox.ac.uk/research/projects (2014).
51. Katrien Verbert, Denis Parra, Peter Brusilovsky, and Erik Duval. 2013. Visualizing recommendations to support exploration, transparency and controllability. In Proceedings of the 2013 international conference on Intelligent user interfaces. ACM, 351--362.
52. Bodo Winter. 2013. A very basic tutorial for performing linear mixed effects analyses. arXiv preprint arXiv:1308.5499 (2013).

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI design and evaluation methods

      • User models

      • User studies

  • Interaction design

    • Interaction design process and methods

      • User interface design

  • Visualization

    • Visualization application domains

      • Information visualization

    • Visualization design and evaluation methods

• Information systems

  • Information retrieval

    • Retrieval tasks and goals

      • Recommender systems

  • World Wide Web

    • Web searching and information discovery

      • Personalization

• Social and professional topics

  • User characteristics