Mixed methods for evaluating user satisfaction

ABSTRACT

Evaluation is a fundamental part of a recommendation system. Evaluation typically takes one of three forms: (1) smaller lab studies with real users; (2) batch tests with offline collections, judgements, and measures; (3) large-scale controlled experiments (e.g. A/B tests) looking at implicit feedback. But it is rare for the first to inform and influence the latter two; in particular, implicit feedback metrics often have to be continuously revised and updated as assumptions are found to be poorly supported.

Mixed methods research enables practitioners to develop robust evaluation metrics by combining strengths of both qualitative and quantitative approaches. In this tutorial, we will show how qualitative research on user behavior provides insight on the relationship between implicit signals and satisfaction. These insights can inform and augment quantitative modeling and analysis for online and offline metrics and evaluation.

References

1. Eugene Agichtein, Eric Brill, and Susan Dumais. Improving web search ranking by incorporating user behavior information. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 19--26, 2006.
2. Ben Carterette. The best published result is random: sequential testing and its effect on reported effectiveness. In Proceedings of the 38th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 2015.
3. Ben Carterette. Statistical significance testing in information retrieval: theory and practice. In Proceedings of the 40th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 2017.
4. Benjamin A Carterette. Multiple testing in statistical analysis of systems-based information retrieval experiments. ACM Transactions on Information Systems (TOIS), 30(1):4, 2012.
5. Marco Creatura. What we saw at recsys 2017 conference, Sep 2017.
6. John W Creswell, Ann Carroll Klassen, Vicki L Plano Clark, and Katherine Clegg Smith. Best practices for mixed methods research in the health sciences. Bethesda (Maryland): National Institutes of Health, 2013:541--545, 2011.
7. John W Creswell, Vicki L Plano Clark, Michelle L Gutmann, and William E Hanson. An expanded typology for classifying mixed methods research info design. Handbook of Mixed Methods in Social and Behavioural Research, 2003.
8. Steve Fox, Kuldeep Karnawat, Mark Mydland, Susan Dumais, and Thomas White. Evaluating implicit measures to improve web search. TOIS, 23(2), 2005.
9. Asela Gunawardana and Guy Shani. Evaluating recommender systems. In Recommender Systems Handbook, pages 265--308. Springer, 2015.
10. Qi Guo, Haojian Jin, Dmitry Lagun, Shuai Yuan, and Eugene Agichtein. Mining touch interaction data on mobile devices to predict web search result relevance. In Proceedings of the 36th Annual International Conference on Research and Development in Information Retrieval, 2013.
11. Martyn Hammersley and Paul Atkinson. Ethnography: Principles in practice. Routledge, 2007.
12. Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. Accurately interpreting clickthrough data as implicit feedback. In ACM SIGIR Forum, volume 51, pages 4--11. ACM, 2017.
13. Richard A Johnson and Dean W Wichern. Multivariate analysis. Encyclopedia of Statistical Sciences, 8, 2004.
14. Bart P Knijnenburg. Conducting user experiments in recommender systems. In Proceedings of the 6th ACM Conference on Recommender Systems, pages 3--4, 2012.
15. Jon A Krosnick. Survey research. Annual review of psychology, 50(1):537--567, 1999.
16. Mike Kuniavsky. Observing the user experience: a practitioner's guide to user research. Elsevier, 2003.
17. Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser. Research methods in human-computer interaction. Morgan Kaufmann, 2017.
18. Nils Brede Moe, Torgeir Dingsøyr, and Tore Dybå. A teamwork model for understanding an agile team: A case study of a scrum project. Information and Software Technology, 52(5):480--491, 2010.
19. Tetsuya Sakai. Conducting laboratory experiments properly with statistical tools: an easy hands-on tutorial. In Proceedings of the 41st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 2018.
20. Steven J Taylor, Robert Bogdan, and Marjorie DeVault. Introduction to qualitative research methods: A guidebook and resource. John Wiley & Sons, 2015.

Index Terms

• General and reference

  • Cross-computing tools and techniques

    • Evaluation

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI design and evaluation methods

      • User studies

• Information systems

  • Information retrieval

    • Retrieval tasks and goals

      • Recommender systems