skip to main content
10.1145/3320435.3320465acmconferencesArticle/Chapter ViewAbstractPublication PagesumapConference Proceedingsconference-collections
research-article

Evaluating Visual Explanations for Similarity-Based Recommendations: User Perception and Performance

Published: 07 June 2019 Publication History

Abstract

Recommender system helps users to reduce information overload. In recent years, enhancing explainability in recommender systems has drawn more and more attention in the field of Human-Computer Interaction (HCI). However, it is not clear whether a user-preferred explanation interface can maintain the same level of performance while the users are exploring or comparing the recommendations. In this paper, we introduced a participatory process of designing explanation interfaces with multiple explanatory goals for three similarity-based recommendation models. We investigate the relations of user perception and performance with two user studies. In the first study (N=15), we conducted card-sorting and semi-interview to identify the user preferred interfaces. In the second study (N=18), we carry out a performance-focused evaluation of six explanation interfaces. The result suggests that the user-preferred interface may not guarantee the same level of performance.

References

[1]
Svetlin Bostandjiev, John O'Donovan, and Tobias Höllerer. 2012. TasteWeights: a visual interactive hybrid recommender system. In Proceedings of the sixth ACM conference on Recommender systems. ACM, 35--42.
[2]
Peter Brusilovsky, Jung Sun Oh, Claudia López, Denis Parra, and Wei Jeng. 2016. Linking information and people in a social system for academic conferences. New Review of Hypermedia and Multimedia (2016), 1--31.
[3]
Cecilia di Sciascio, Peter Brusilovsky, and Eduardo Veas. 2018. A Study on User-Controllable Social Exploratory Search. In 23rd International Conference on Intelligent User Interfaces (IUI '18). ACM, New York, NY, USA, 353--364.
[4]
Jianguang Du, Jing Jiang, Dandan Song, and Lejian Liao. 2015. Topic modeling with document relative similarities. IJCAI.
[5]
Malin Eiband, Hanna Schneider, Mark Bilandzic, Julian Fazekas-Con, Mareike Haug, and Heinrich Hussmann. 2018. Bringing Transparency Design into Practice. In 23rd International Conference on Intelligent User Interfaces. ACM, 211--223.
[6]
Gerhard Friedrich and Markus Zanker. 2011. A taxonomy for generating explanations in recommender systems. AI Magazine, Vol. 32, 3 (2011), 90--98.
[7]
Fatih Gedikli, Mouzhi Ge, and Dietmar Jannach. 2011. Understanding recommendations by reading the clouds. In International Conference on Electronic Commerce and Web Technologies. Springer, 196--208.
[8]
Wael H Gomaa and Aly A Fahmy. 2013. A survey of text similarity approaches. International Journal of Computer Applications, Vol. 68, 13 (2013), 13--18.
[9]
Julio Guerra-Hollstein, Jordan Barria-Pineda, Christian D Schunn, Susan Bull, and Peter Brusilovsky. 2017. Fine-Grained Open Learner Models: Complexity Versus Support. In Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization. ACM, 41--49.
[10]
Chen He, Denis Parra, and Katrien Verbert. 2016. Interactive recommender systems: A survey of the state of the art and future research challenges and opportunities. Expert Systems with Applications, Vol. 56 (2016), 9--27.
[11]
Jonathan L Herlocker, Joseph A Konstan, and John Riedl. 2000. Explaining collaborative filtering recommendations. In Proceedings of the 2000 ACM conference on Computer supported cooperative work. ACM, 241--250.
[12]
Bart P. Knijnenburg, Svetlin Bostandjiev, John O'Donovan, and Alfred Kobsa. 2012. Inspectability and Control in Social Recommenders. In 6th ACM Conference on Recommender System . 43--50.
[13]
Pigi Kouki, James Schaffer, Jay Pujara, John O'Donovan, and Lise Getoor. 2017. User preferences for hybrid explanations. In Proceedings of the Eleventh ACM Conference on Recommender Systems. ACM, 84--88.
[14]
Danielle Lee and Peter Brusilovsky. 2017. How to Measure Information Similarity in Online Social Networks: A Case Study of Citeulike. Information Sciences, Vol. 418--419 (2017), 46--60.
[15]
Julian McAuley and Jure Leskovec. 2013. Hidden factors and hidden topics: understanding rating dimensions with review text. In Proceedings of the 7th ACM conference on Recommender systems. ACM, 165--172.
[16]
Conference Navigator. 2018. Paper Tuner. http://halley.exp.sis.pitt.edu/cn3/portalindex.php
[17]
Alexis Papadimitriou, Panagiotis Symeonidis, and Yannis Manolopoulos. 2012. A generalized taxonomy of explanations styles for traditional and social recommender systems. Data Mining and Knowledge Discovery, Vol. 24, 3 (2012), 555--583.
[18]
Denis Parra and Peter Brusilovsky. 2015. User-controllable personalization: A case study with SetFusion. International Journal of Human-Computer Studies, Vol. 78 (2015), 43--67.
[19]
Pearl Pu and Li Chen. 2007. Trust-inspiring explanation interfaces for recommender systems. Knowledge-Based Systems, Vol. 20, 6 (2007), 542--556.
[20]
Amit Sharma and Dan Cosley. 2013. Do social explanations work?: studying and modeling the effects of social explanations in recommender systems. In Proceedings of the 22nd international conference on World Wide Web. ACM, 1133--1144.
[21]
Julia Silge and David Robinson. 2016. tidytext: Text mining and analysis using tidy data principles in r. The Journal of Open Source Software, Vol. 1, 3 (2016), 37.
[22]
Kirsten Swearingen and Rashmi Sinha. 2001. Beyond algorithms: An HCI perspective on recommender systems. In ACM SIGIR 2001 Workshop on Recommender Systems, Vol. 13. Citeseer, 1--11.
[23]
Nava Tintarev and Judith Masthoff. 2011. Designing and evaluating explanations for recommender systems. In Recommender systems handbook . Springer, 479--510.
[24]
Nava Tintarev and Judith Masthoff. 2012. Evaluating the effectiveness of explanations for recommender systems. User Modeling and User-Adapted Interaction, Vol. 22, 4--5 (1 Oct. 2012), 399--439.
[25]
Nava Tintarev and Judith Masthoff. 2015. Explaining recommendations: Design and evaluation. In Recommender systems handbook . Springer, 353--382.
[26]
Chun-Hua Tsai and Peter Brusilovsky. 2017. Providing Control and Transparency in a Social Recommender System for Academic Conferences. In Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization. ACM, 313--317.
[27]
Chun-Hua Tsai and Peter Brusilovsky. 2018a. Beyond the Ranked List: User-Driven Exploration and Diversification of Social Recommendation. In 23rd International Conference on Intelligent User Interfaces. ACM, 239--250.
[28]
Chun-Hua Tsai and Peter Brusilovsky. 2018b. Explaining Social Recommendations to Casual Users: Design Principles and Opportunities. In Proceedings of the 23rd International Conference on Intelligent User Interfaces Companion. ACM, 59.
[29]
Chun-Hua Tsai and Peter Brusilovsky. 2019 a. Designing Explanation Interfaces for Transparency and Beyond. In Workshop on Intelligent User Interfaces for Algorithmic Transparency in Emerging Technologies .
[30]
Chun-Hua Tsai and Peter Brusilovsky. 2019 b. Explaining Recommendations in an Interactive Hybrid Social Recommender. In Proceedings of the 2019 Conference on Intelligent User Interface. ACM, 1--12.
[31]
Chun-Hua Tsai and Peter Brusilovsky. 2019 c. Exploring Social Recommendations with Visual Diversity-Promoting Interfaces. TiiS, Vol. 1, 1 (2019), 1--1.
[32]
Jesse Vig, Shilad Sen, and John Riedl. 2009. Tagsplanations: explaining recommendations using tags. In Proceedings of the 14th international conference on Intelligent user interfaces. ACM, 47--56.
[33]
Yao Wu and Martin Ester. 2015. Flame: A probabilistic model combining aspect based opinion mining and collaborative filtering. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining. ACM, 199--208.
[34]
Qian Zhao, Gediminas Adomavicius, F Maxwell Harper, Martijn C Willemsen, and Joseph A Konstan. 2017. Toward Better Interactions in Recommender Systems: Cycling and Serpentining Approaches for Top-N Item Lists. In CSCW. 1444--1453.
[35]
Zoomdata. 2018. Real-time Interactive Zoomdata Wordcloud. https://visual.ly/community/interactive-graphic/social-media/real-time-interactive-zoomdata-wordcloud

Cited By

View all
  • (2024)On the Negative Perception of Cross-domain Recommendations and ExplanationsProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657735(2102-2113)Online publication date: 10-Jul-2024
  • (2024)What Is the Focus of XAI in UI Design? Prioritizing UI Design Principles for Enhancing XAI User ExperienceArtificial Intelligence in HCI10.1007/978-3-031-60606-9_13(219-237)Online publication date: 29-Jun-2024
  • (2023)Co-Designing with Users the Explanations for a Proactive Auto-Response Messaging AgentProceedings of the ACM on Human-Computer Interaction10.1145/36042487:MHCI(1-23)Online publication date: 13-Sep-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
UMAP '19: Proceedings of the 27th ACM Conference on User Modeling, Adaptation and Personalization
June 2019
377 pages
ISBN:9781450360210
DOI:10.1145/3320435
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 June 2019

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. recommendation
  2. similarity-based
  3. visual explanation

Qualifiers

  • Research-article

Conference

UMAP '19
Sponsor:

Acceptance Rates

UMAP '19 Paper Acceptance Rate 30 of 122 submissions, 25%;
Overall Acceptance Rate 162 of 633 submissions, 26%

Upcoming Conference

UMAP '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)68
  • Downloads (Last 6 weeks)0
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)On the Negative Perception of Cross-domain Recommendations and ExplanationsProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657735(2102-2113)Online publication date: 10-Jul-2024
  • (2024)What Is the Focus of XAI in UI Design? Prioritizing UI Design Principles for Enhancing XAI User ExperienceArtificial Intelligence in HCI10.1007/978-3-031-60606-9_13(219-237)Online publication date: 29-Jun-2024
  • (2023)Co-Designing with Users the Explanations for a Proactive Auto-Response Messaging AgentProceedings of the ACM on Human-Computer Interaction10.1145/36042487:MHCI(1-23)Online publication date: 13-Sep-2023
  • (2023)Modeling Adaptive Expression of Robot Learning Engagement and Exploring Its Effects on Human TeachersACM Transactions on Computer-Human Interaction10.1145/357181330:5(1-48)Online publication date: 23-Sep-2023
  • (2023)Comparison of Attention Models and Post-hoc Explanation Methods for Embryo Stage Identification: A Case StudyPattern Recognition, Computer Vision, and Image Processing. ICPR 2022 International Workshops and Challenges10.1007/978-3-031-37731-0_17(216-230)Online publication date: 10-Aug-2023
  • (2022)How to Support Users in Understanding Intelligent Systems? An Analysis and Conceptual Framework of User Questions Considering User Mindsets, Involvement, and Knowledge OutcomesACM Transactions on Interactive Intelligent Systems10.1145/351926412:4(1-27)Online publication date: 5-Nov-2022
  • (2022)Understanding the Role of Explanation Modality in AI-assisted Decision-makingProceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization10.1145/3503252.3531311(223-233)Online publication date: 4-Jul-2022
  • (2022)Explaining Recommendations in E-Learning: Effects on Adolescents' TrustProceedings of the 27th International Conference on Intelligent User Interfaces10.1145/3490099.3511140(93-105)Online publication date: 22-Mar-2022
  • (2022)Metrics for Saliency Map Evaluation of Deep Learning Explanation MethodsPattern Recognition and Artificial Intelligence10.1007/978-3-031-09037-0_8(84-95)Online publication date: 1-Jun-2022
  • (2021)Educating Computer Science Students about Algorithmic Fairness, Accountability, Transparency and EthicsProceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education V. 110.1145/3430665.3456311(484-490)Online publication date: 26-Jun-2021
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media