research-article

User experience evaluation methods: current state and development needs

Authors:

Arnold P. O. S. Vermeeren,

Effie Lai-Chong Law,

Marianna Obrist,

Jettie Hoonhout,

Kaisa Väänänen-Vainio-MattilaAuthors Info & Claims

NordiCHI '10: Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries

Pages 521 - 530

https://doi.org/10.1145/1868914.1868973

Published: 16 October 2010 Publication History

Abstract

The recent shift of emphasis to user experience (UX) has rendered it a central focus of product design and evaluation. A multitude of methods for UX design and evaluation exist, but a clear overview of the current state of the available UX evaluation methods is missing. This is partly due to a lack of agreement on the essential characteristics of UX. In this paper, we present the results of our multi-year effort of collecting UX evaluation methods from academia and industry with different approaches such as literature review, workshops, Special Interest Groups sessions and an online survey. We have collected 96 methods and analyzed them, among other criteria, based on the product development phase and the studied period of experience. Our analysis reveals development needs for UX evaluation methods, such as early-stage methods, methods for social and collaborative UX evaluation, establishing practicability and scientific quality, and a deeper understanding of UX.

References

[1]

ENGAGE, Report on the evaluation of generative tools and methods for 'emotional design'. Deliverable D15.3. EU project Engage 520998 (2006).

[2]

Ganglbauer, E., Schrammel, J., Deutsch, S., and Tscheligi, M. Applying Psychophysiological Methods for Measuring User Experience: Possibilities, Challenges and Feasibility. Workshop on User Experience Evaluation Methods in Product Development. August 25, 2009. Uppsala, Sweden.

[3]

Gaver, B., Dunne, T., and Pacenti, E. Design: Cultural probes. Interactions 6, 1 (Jan. 1999), 21--29.

Digital Library

[4]

Green, W., Dunn, G., and Hoonhout, J. Developing the scale adoption framework for evaluation (SAFE). In: Proc. of the 5th COST294-MAUSE Open Workshop "Meaningful Measures: Valid Useful User Experience Measurement (VUUM)", Iceland June 2008. Also published in the ACM Library.

[5]

Hassenzahl, M., and Tractinsky, N., User Experience - a research agenda. In: Behavior & Information Technology, 25(2), (2006) pp. 91--97.

[6]

Hassenzahl, M. 2008. User experience (UX): towards an experiential perspective on product quality. In Proc. of the 20th international Conference of the Association Francophone D'interaction Homme-Machine. IHM '08, vol. 339. (2008) ACM, New York, NY, 11--15.

Digital Library

[7]

Hoonhout, H. C. M. Let the game tester do the talking: think aloud and interviewing to learn about the game experience, In: Isbister, K., Schaffer, N. (eds.), Game Usability: Advice from the Experts for Advancing the Player Experience, San Fransisco, CA: Morgan Kaufmann Publishers, (2008) 65--77.

[8]

HUMAINE D9j: Final report on WP9. 30^th January, 2008.

[9]

ISO DIS 9241-210:2010. Ergonomics of human system interaction - Part 210: Human-centred design for interactive systems (formerly known as 13407). International Standardization Organization (ISO). Switzerland.

[10]

Isomursu M, Tähti, M., Väinämö, S., and Kuutti, K. Experimental evaluation of five methods for collecting emotions in field settings with mobile applications. International Journal of Human-Computer Studies, Volume 65(4), April 2007, pp. 404--418.

Digital Library

[11]

Isomursu, M. (2008). User experience evaluation with experimental pilots. In: Väänänen-Vainio-Mattila, K.; Roto, V.; Hassenzahl, M. Now Let's Do It in Practice: User Experience Evaluation Methods in Product Development, Workshop at CHI2008 (2008).

Digital Library

[12]

Jordan, P. Designing Pleasurable Products. (2000) Taylor & Francis, London.

[13]

Karapanos, E., Zimmerman, J., Forlizzi, J., and Martens, J. User experience over time: an initial framework. In Proc. CHI '09. (2009) ACM, New York, NY, 729--738.

Digital Library

[14]

Kaye, J. Evaluating experience-focused HCI. In CHI '07 Extended Abstracts on Human Factors in Computing Systems CHI '07. (2007) ACM, New York, NY, 1661--1664.

Digital Library

[15]

Kujala, S., and Väänänen-Vainio-Mattila, K. Value of Information Systems and Products: Understanding the Users' Perspective and Values, Journal of Information Technology Theory and Application (JITTA), 9, 4, 2009.

[16]

Law, E., Roto, V., Hassenzahl, M., Vermeeren, A., and Kort, J. (2009). Understanding, Scoping and Defining User eXperience: A Survey Approach. Proc. CHI'09, ACM SIGCHI conference on Human Factors in Computing Systems.

Digital Library

[17]

Law, E., Scapin, D., Cockton, G., Stary, M., and Winckler, M. Maturation of Usability Evaluation Methods: Retrospect and Prospect. COST294-MAUSE Closing Conference Proceedings (2009). http://141.115.28.2/cost294/upload/533.pdf

[18]

Lewin, K. Field theory in social science; selected theoretical papers. D. Cartwright (ed.). (1951) New York: Harper & Row.

[19]

Mäkelä, A., Fulton Suri, J. Supporting Users' Creativity: Design to Induce Pleasurable Experiences. Proc. of the Int. Conf. on Affective Human Factors Design, (2001) pp. 387--394.

[20]

Monahan, K., Lahteenmaki, M., McDonald, S., and Cockton, G. 2008. An investigation into the use of field methods in the design and evaluation of interactive systems. In Proc. of the 22nd British HCI Group Annual Conf. on HCI 2008: People and Computers Xxii: Culture, Creativity, interaction - Volume 1. (2008) British Computer Society, Swinton, UK, 99--108.

Digital Library

[21]

Nisbett, R. E. and Wilson, T., Telling more than we can know: verbal reports on mental processes. Psychological Review, 1977, 84(3), 231--259.

[22]

Obrist, M., Roto, V., and Väänänen-Vainio-Mattila, K. User Experience Evaluation -- Do You Know Which Method to Use? Special Interest Group in CHI2009 Conference, Boston, USA, 5--9 April, 2009.

Digital Library

[23]

Roto, V., Väänänen-Vainio-Mattila, K., Law, E., and Vermeeren, A. User Experience Evaluation Methods in Product Development. Workshop in INTERACT'09, August 25, 2009. Uppsala, Sweden.

Digital Library

[24]

Shami, N. S., Hancock, J. T., Peter, C., Muller, M., and Mandryk, R. Measuring affect in HCI: going beyond the individual. In CHI '08 Extended Abstracts on Human Factors in Computing Systems. CHI '08. (2008) ACM, New York, NY, 3901--3904.

Digital Library

[25]

Stone, D., Jarrett, C., Woodroffe, M., and Minocha, S. User Interface Design and Evaluation (The Morgan Kaufmann Series in Interactive Technologies). (2005) Morgan Kaufmann.

Digital Library

[26]

Tullis, T., and Albert, B. Measuring the User Experience. Collecting, Analyzing, and Presenting Usability Metrics. (2008) Morgan Kaufmann.

Digital Library

[27]

Väänänen-Vainio-Mattila, K., Roto, V., and Hassenzahl, M. Now Let's Do It in Practice: User Experience Evaluation Methods in Product Development. In Extended Abstract of CHI 2008, ACM Press (2008), pp. 3961--3964.

Digital Library

Cited By

Zhong SGatti EHardwick JRibul MCho YObrist M(2025) LLM-mediated domain-specific voice agents: the case of TextileBot Behaviour & Information Technology10.1080/0144929X.2025.2456667(1-33)Online publication date: 3-Feb-2025
https://doi.org/10.1080/0144929X.2025.2456667
Tang RHerrli GTang RHerrli G(2025)UX Research Models and MethodsThe Domain of UX in Information Studies: Bridging Theories, Research, and Professional Practice10.1007/978-3-031-83529-2_3(37-71)Online publication date: 4-Mar-2025
https://doi.org/10.1007/978-3-031-83529-2_3
Ghosh SDubey S(2024)The Field of Usability and User Experience (UX) DesignNavigating Usability and User Experience in a Multi-Platform World10.4018/979-8-3693-2337-3.ch005(117-154)Online publication date: 15-Nov-2024
https://doi.org/10.4018/979-8-3693-2337-3.ch005
Show More Cited By

Index Terms

User experience evaluation methods: current state and development needs
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods

Recommendations

Game User Experience Evaluation
CHI EA '16: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems

In a nutshell: This course comprehensively covers important user experience (UX) evaluation methods as well as opportunities and challenges of UX evaluation in the area of entertainment and games. The course is an ideal forum for attendees to gain ...
Let users tell the story: evaluating user experience with experience reports
CHI EA '10: CHI '10 Extended Abstracts on Human Factors in Computing Systems

User experience (UX) has been under extensive research in recent years. One of the key questions has been how to evaluate user experience. Several methods such as diaries, experience sampling and questionnaires have been used for collecting data on user ...
The experience of interactive storytelling: comparing “fahrenheit” with “façade”
ICEC'11: Proceedings of the 10th international conference on Entertainment Computing

At the intersection of multimedia, artificial intelligence, and gaming technology, new visions of future entertainment media arise that approximate the “Holodeck” ® idea of interactive storytelling. We report exploratory experiments on the user ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

NordiCHI '10: Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries

October 2010

889 pages

ISBN:9781605589343

DOI:10.1145/1868914

General Chairs:
Ebba Þóra Hvannberg
University of Iceland, Iceland
,
Marta Kristín Lárusdóttir
Reykjavik University, Iceland
,
Program Chairs:
Ann Blandford
University College London
,
Jan Gulliksen
Royal Institute of Technology, Sweden

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Reykjavik University
University of Iceland

In-Cooperation

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 October 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

NordiCHI '10

Sponsor:

NordiCHI '10: 6th Nordic Conference on Human-Computer Interaction

October 16 - 20, 2010

Reykjavik, Iceland

Acceptance Rates

Overall Acceptance Rate 379 of 1,572 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

415
Total Citations
View Citations
12,880
Total Downloads

Downloads (Last 12 months)1,040
Downloads (Last 6 weeks)97

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhong SGatti EHardwick JRibul MCho YObrist M(2025) LLM-mediated domain-specific voice agents: the case of TextileBot Behaviour & Information Technology10.1080/0144929X.2025.2456667(1-33)Online publication date: 3-Feb-2025
https://doi.org/10.1080/0144929X.2025.2456667
Tang RHerrli GTang RHerrli G(2025)UX Research Models and MethodsThe Domain of UX in Information Studies: Bridging Theories, Research, and Professional Practice10.1007/978-3-031-83529-2_3(37-71)Online publication date: 4-Mar-2025
https://doi.org/10.1007/978-3-031-83529-2_3
Ghosh SDubey S(2024)The Field of Usability and User Experience (UX) DesignNavigating Usability and User Experience in a Multi-Platform World10.4018/979-8-3693-2337-3.ch005(117-154)Online publication date: 15-Nov-2024
https://doi.org/10.4018/979-8-3693-2337-3.ch005
Lebzar BAit el Bour D(2024)The impact of humanizing digital interfaces on UX. A study of human-Chatbot interactionQuestion(s) de management10.3917/qdm.229.0143n° 49:2(143-160)Online publication date: 3-Jul-2024
https://doi.org/10.3917/qdm.229.0143
Marin-Alvarez ECarcelen-Garcia SGalmes-Cerezo M(2024)A Holistic and Multidimensional Methodology Proposal for a Persona with Total Visual Impairment Archetype on the WebSocieties10.3390/soc1407010214:7(102)Online publication date: 27-Jun-2024
https://doi.org/10.3390/soc14070102
Hsueh NLin HLai L(2024)Applying Large Language Model to User Experience TestingElectronics10.3390/electronics1323463313:23(4633)Online publication date: 24-Nov-2024
https://doi.org/10.3390/electronics13234633
Rojas LQuiñones DCubillos C(2024)FRAMUX-EV: A Framework for Evaluating User Experience in Agile Software DevelopmentApplied Sciences10.3390/app1419899114:19(8991)Online publication date: 6-Oct-2024
https://doi.org/10.3390/app14198991
An SPark JKim HKang HBeck D(2024)An Ergonomic Study on the Operation Method and In-Vehicle Location of an Automotive Electronic GearshiftApplied Sciences10.3390/app1402067214:2(672)Online publication date: 12-Jan-2024
https://doi.org/10.3390/app14020672
Tinterri APelizzari Fdi Padova MPalladino FVignoli GDipace A(2024)Automating board-game based learning. A comprehensive study to assess reliability and accuracy of AI in game evaluationIntelligenza Artificiale10.3233/IA-240030(1-17)Online publication date: 2-Jul-2024
https://doi.org/10.3233/IA-240030
Miyairi T(2024)Design of sensory experiences using sound symbolism: A study on rotary switch operationAcoustical Science and Technology10.1250/ast.e24.3145:5(245-251)Online publication date: 1-Sep-2024
https://doi.org/10.1250/ast.e24.31
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten