Tell me more?: the effects of mental model soundness on personalizing an intelligent agent

ABSTRACT

What does a user need to know to productively work with an intelligent agent? Intelligent agents and recommender systems are gaining widespread use, potentially creating a need for end users to understand how these systems operate in order to fix their agent's personalized behavior. This paper explores the effects of mental model soundness on such personalization by providing structural knowledge of a music recommender system in an empirical study. Our findings show that participants were able to quickly build sound mental models of the recommender system's reasoning, and that participants who most improved their mental models during the study were significantly more likely to make the recommender operate to their satisfaction. These results suggest that by helping end users understand a system's reasoning, intelligent agents may elicit more and better feedback, thus more closely aligning their output with each user's intentions.

References

1. Amershi, S., Fogarty, J., Kapoor, A., and Tan, D. Examining multiple potential models in end-user interactive concept learning. In Proc. CHI, ACM (2010), 1357--1360.
2. Bandura, A. Self-efficacy: Toward a unifying theory of behavioral change. Psychological Review 8, 2 (1977).
3. Benedek, J. and Miner, T. Measuring desirability: New methods for evaluating desirability in a usability lab setting. In Proc. Usability Professionals Association International Conference (2002).
4. Bozionelos, N. The relationship of instrumental and expressive traits with computer anxiety. Personality and Individual Differences 31 (2001), 955--974.
5. Compeau, D. and Higgins, C. Application of social cognitive theory to training for computer skills. Information Systems Research, 6,2 (1995), 118--143.
6. Echo Nest, The. http://the.echonest.com (July, 2011).
7. Fiebrink, R., Cook, P., and Trueman, D. Human model evaluation in interactive supervised learning. In Proc. CHI, ACM (2011), 147--156.
8. Hart, S. and Staveland, L. Development of a NASA-TLX (Task load index): Results of empirical and theoretical research, Hancock, P. and Meshkati, N. (Eds.), Human Mental Workload (1988), 139--183.
9. Herlocker, J., Konstan, J., Riedl, J. Explaining collaborative filtering recommendations. In Proc. CSCW, ACM (2000), 241--250.
10. Johnson-Laird, P.N. Mental Models: Towards a Cognitive Science of Language, Inference, and Consciousness. Cambridge University Press (1983).
11. Kapoor, A., Lee, B., Tan, D., and Horvitz, E. Interactive optimization for steering machine classification. In Proc. CHI, ACM (2010), 1343--1352.
12. Kolb, D. A. Experiential Learning. Prentice-Hall Englewood Cliffs, NJ (1984).
13. Kulesza, T., Wong, W.-K., Stumpf, S., Perona, S., White, R., Burnett, M., Oberst, I., and Ko, A. J. Fixing the program my computer learned: barriers for end users, barriers for the machine. In Proc. IUI, ACM (2009), 187--196.
14. Kulesza, T., Stumpf, S., Burnett, M., Wong, W., Riche, Y., Moore, T., Oberst, I., Shinsel, A., McIntosh, K. Explanatory debugging: Supporting end-user debugging of machine-learned programs. In Proc. VL/HCC, IEEE (2010), 41--48.
15. Lim, B. Y., Dey, A. K., and Avrahami, D. Why and why not explanations improve the intelligibility of contextaware intelligent systems. Proc. CHI, ACM (2009), 2119- 2128.
16. Lim, B. Y. and Dey, A. K. Toolkit to support intelligibility in context-aware applications. In Proc. UbiComp, ACM (2010), 13--22.
17. McNee, S. M., Lam, S. K., Guetzlaff, C., Konstan, J. A., and Riedl, J. Confidence displays and training in recommender systems. In Proc. INTERACT, IFIP (2003), 176--183.
18. Norman, D. Some observations on mental models, Gentner, D. and Stevens, A. (Eds.), Mental Models (1983), 7--14.
19. Pandora Media, Inc. Initial Public Offering Form S-1 (2011).
20. Rosson, M. B., Carroll, J. M., and Bellamy, R. K. E. Smalltalk scaffolding: a case study of minimalist instruction. In Proc. CHI, ACM (1990), 423--430.
21. Sharp, H., Rogers, Y., and Preece, J. Interaction Design: Beyond Human-Computer Interaction (3rd edition), John Wiley (2011).
22. Sinha, R. R. and Swearingen, K. The role of transparency in recommender systems. In Proc. CHI Extended Abstracts, ACM (2002), 830--831.
23. Stone, P., Dunphy, D., Smith, M., Ogilvie, D., and associates. The General Inquirer: A Computer Approach to Content Analysis. The MIT Press (1966).
24. Stumpf, S., Rajaram, V., Li, L., Burnett, M., Dietterich, T., Sullivan, E., Drummond, R., and Herlocker, J. Toward harnessing user feedback for machine learning. In Proc. IUI, ACM (2007), 82--91.
25. Stumpf, S., Rajaram, V., Li, L., Burnett, M., Wong, W.K., Dietterich, T., Sullivan, E., Drummond, R., and Herlocker, J. Interacting meaningfully with machine learning systems: Three experiments. International Journal of Human-Computer Studies 67, 8 (2009).
26. Talbot, J., Lee, B., Tan, D., and Kapoor, A. EnsembleMatrix: Interactive visualization to support machine learning with multiple classifiers. In Proc. CHI, ACM (2009), 1283--1292.
27. Tintarev, N., and Masthoff, J. Effective explanations of recommendations: User-centered design. In Proc. Recommender Systems (2007), 153--156.
28. Tullio, J., Dey, A.K., Chalecki, J., and Fogarty, J. How it works: A field study of non-technical users interacting with an intelligent system. In Proc CHI, ACM (2007).
29. Wilfong, J. Computer anxiety and anger: The impact of computer use, computer experience, and self-efficacy beliefs. Computers in Human Behavior 22 (2006).

Index Terms

• Human-centered computing