A Systematic Mapping Study on Research Contributions on UX Evaluation Technologies

ABSTRACT

User eXperience (UX) refers to quality attributes related to the feelings and emotions of the users. In order to evaluate UX, several technologies (tools, methods, techniques, others) have been proposed that range from using questionnaires to employing biometrics. However, it is essential to characterize UX evaluation technologies to obtain evidence on the contexts in which using a specific technology can provide better evaluation results. In this paper, we present the state of the art on how to evaluate the UX of software applications. We performed a systematic mapping study with an initial sample of 2101 papers from which 227 relevant papers describing UX evaluation technologies have been identified. The results suggest that there is a need for specific UX evaluation technologies that are easy and comfortable to use from the point of view of users, while supporting practitioners in the correction of the aspects that cause poor experiences.

References

1. Stephanie Balters, and Steinert Martin. 2015. Capturing emotion reactivity through physiology measurement as a foundation for affective engineering in engineering design science and engineering practices. Journal of Intelligent Manufacturing 28, 1: 1--23.
2. Simone Barbosa, and Bruno Silva. 2010. Human Computer Interaction. Elsevier Brasil. (In Portuguese)
3. Javier A. Bargas-Avila, and Kasper Hornbæk. 2011. Old wine in new bottles or novel challenges: a critical analysis of empirical studies of user experience. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2689--2698.
4. Javier A. Bargas-Avila, and Kasper Hornbæk. 2012. Foci and blind spots in user experience research. Interactions 19, 6: 24--27.
5. Pieter Desmet. 2003. Measuring emotion: Development and application of an instrument to measure emotional responses to products. In Funology, 111--123.
6. Adrian Fernandez, Emilio Insfran, and Silvia Abrahão. 2011. Usability evaluation methods for the web: A systematic mapping study. Information and Software Technology 53, 8: 789--817.
7. Jérémy Frey, Christian Mühl, Fabien Lotte, and Martin Hachet. 2013. Review of the use of electroencephalography as an evaluation method for human-computer interaction. In arXiv preprint arXiv, 1311.2222.
8. Bill Gaver, and Heather Martin. 2000. Alternatives: exploring information appliances through conceptual design proposals. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems, 209--216.
9. Marc Hassenzahl. 2003. The thing and I: understanding the relationship between user and product. In Funology. 31--42.
10. Marc Hassenzahl, Sarah Diefenbach, and Anja Göritz. 2010. Needs, affect, and interactive products--Facets of user experience. Interacting with computers 22, 5: 353--362.
11. International Organization for Standardization, ISO DIS 9241-210. 2010. Ergonomics of human system interaction - Part 210. International Standardization Organization (ISO).
12. Minna Isomursu, Marika Tähti, Soili Väinämä, and Kari Kuutti. 2007. Experimental evaluation of five methods for collecting emotions in field settings with mobile applications. International Journal of Human-Computer Studies 65, 4: 404--418.
13. Patrick W. Jordan. 2002. Designing pleasurable products: An introduction to the new human factors. CRC press.
14. Barbara Kitchenham, and Stuart Chartes. 2007. Guidelines for performing systematic literature reviews in software engineering. In Technical report, Ver. 2.3 EBSE Technical Report. EBSE.
15. Barbara Kitchenham, Rialette Pretorius, David Budgen, O. Pearl Brereton, Mark Turner, Mahmood Niazi, and Stephen Linkman. 2010. Systematic literature reviews in software engineering--a tertiary study. Information and Software Technology 52, 8: 792--805.
16. Sari Kujala, Virpi Roto, Kaisa Väänänen-Vainio-Mattila, Evangelos Karapanos, and Arto Sinnelä. 2011. UX Curve: A method for evaluating long-term user experience. Interacting with Computers 23, 5: 473--483.
17. Effie Lai-Chong Law. 2011. The measurability and predictability of user experience. In Proceedings of the 3rd ACM SIGCHI symposium on Engineering interactive computing systems, 1--10.
18. Elizabete Munzlinger, Fabricio Batista Narcizo, and José Eustáquio Rangel De Queiroz. 2012. Systematization of literature reviews in the HCI area. In Companion Proceedings of the 11th Brazilian Symposium on Human Factors in Computing Systems, 51--54. (In Portuguese)
19. S. Rajeshkumar, Omar Ridha, and Murni Mahmud. 2013. Taxonomies of User Experience (UX) evaluation methods. In International Conference on Research and Innovation in Information Systems (ICRIIS), 533--538.
20. Luis Rivero, and Tayana Conte. 2016. TR-USES-2016-002 Systematic Mapping on UX Technologies: Extraction of User eXperience Evaluation Methods. USES Research Group Technical Report. Available at: http://uses.icomp.ufam.edu.br/relatorios-tecnicos/
21. Patricia Tavares, Luiz Agner, and Simone Bacellar Leal Ferreira. 2010. Usability Evaluation of mobile devices for the collection of neighborhood data through interviews based on scenarios and tasks. In Proceedings of the IX Symposium on Human Factors in Computing Systems, 11--20. (In Portuguese)
22. Arnold P. O. S. Vermeeren, Effie Lai-Chong Law, Virpi Roto, Marianna Obrist, Jettie Hoonhout, and Kaisa Väänänen-Vainio-Mattila. 2010. User experience evaluation methods: current state and development needs. In Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries, 521--530.
23. Le An, Songfan Yang, and Bir Bhanu. 2015. Efficient smile detection by extreme learning machine. Neurocomputing 149, 1: 354--363.
24. Carol M. Barnum, and Laura A. Palmer. 2010. More than a feeling: understanding the desirability factor in user experience. In CHI'10 Extended Abstracts on Human Factors in Computing Systems, 4703--4716.
25. Marios Belk, Panagiotis Germanakos, Panagiotis Zaharias, and George Samaras. 2012. Adaptivity Considerations for Enhancing User-Centric Web Experience. In The Fifth International Conference on Advances in Computer-Human Interactions (ACHI 2012), 348--353.
26. Luca Colombo, and Monica Landoni. 2014. A diary study of children's user experience with EBooks using flow theory as framework. In Proceedings of the 2014 conference on Interaction design and children, 135--144.
27. Reeva Lederman, Greg Wadley, John Gleeson, Sarah Bendall, and Mario Álvarez-Jiménez. 2014. Moderated online social therapy: Designing and evaluating technology for mental health. ACM Transactions on Computer-Human Interaction 21, 1: 5.
28. Philip Lew, Luis Olsina, and Li Zhang. 2010. Integrating quality, quality in use, actual usability and user experience. In 6th Central and Eastern European Software Engineering Conference (CEE-SECR), 117--123.
29. Llúcia Masip, Marta Oliva, and Toni Granollers. 2012. The open repository of heuristics. In Proceedings of the 13th International Conference on Interacción Persona-Ordenador, 4.
30. Pejman Mirza-Babaei, and Lennart Nacke. 2014. Introducing the biometric storyboards tool for games user research. In Games Media Entertainment (GEM), 1--7.
31. Georg Regal, Marc Busch, Christina Hochleitner, Peter Wokerstorfer, and Manfred Tscheligi. 2014. TACTUX: a tactile user experience assessment board. In Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational, 983--986.
32. Jacopo Staiano, María Menéndez, Alberto Battocchi, Antonella De Angeli, and Nicu Sebe. 2012. UX_Mate: from facial expressions to UX evaluation. In Proceedings of the Designing Interactive Systems Conference, 741--750.
33. Jeroen Vanattenhoven, and Dries De Roeck. 2009. Investigating the Use of the Experience Clip Method. In International Conference on User Centric Media, 79--86.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

• Software and its engineering

  • Software creation and management

    • Designing software

    • Software verification and validation

      • Software defect analysis

        • Software testing and debugging