short-paper

Analysis in usability evaluations: an exploratory study

Authors:

Asbjørn Følstad,

Effie Lai-Chong Law,

Kasper HornbækAuthors Info & Claims

NordiCHI '10: Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries

Pages 647 - 650

https://doi.org/10.1145/1868914.1868995

Published: 16 October 2010 Publication History

Abstract

While the planning and implementation of usability evaluations are well described in the literature, the analysis of the evaluation data is not. We present interviews with 11 usability professionals on how they conduct analysis, describing the resources, collaboration, creation of recommendations, and prioritization involved. The interviews indicate a lack of structure in the analysis process and suggest activities, such as generating recommendations, that are unsupported by existing methods. We discuss how to better support analysis, and propose four themes for future research on analysis in usability evaluations.

References

[1]

Bias, R. and Mayhew, D. Cost-Justifying Usability: An Update for the Internet Age. Morgan Kauffman, San Fransisco, CA, 2005.

Digital Library

[2]

Cockton, G. and Lavery, D. A Framework for Usability Problem Extraction. In Proc. Interact 1999. IOS Press (1999), 344--352.

[3]

Cockton, G., Lavery, D. and Woolrych, A. Inspection-based Evaluations. In A. Sears and J. Jacko (eds), The Human-Computer Interaction Handbook, 2nd edition. CRC, 2008.

[4]

Cockton, G., Woolrych, A. and Hindmarch, M. Reconditioned merchandise: extended structured report formats in usability inspection. In Proc. CHI 2004 (2004), 1433--1436.

Digital Library

[5]

Dumas, J. S. and Loring, B. A. Moderating Usability Tests: Principles and Practices for Interacting. Morgan Kaufman, 2008.

Digital Library

[6]

Ezzy, D. Qualitative Analysis: Practice and Innovation. Routledge, London, 2002.

[7]

Hartson, H. R., Andre, T. S. and Williges, R. C. Criteria for evaluating usability evaluation methods. International Journal of Human-Computer Interaction, 13, 4 (2001), 373--410.

[8]

Hassenzahl, M. Prioritising usability problems: Data-driven and judgement-driven severity estimates. Behaviour & Information Technology, 19(2000), 29--42.

[9]

Hertzum, M. and Jacobsen, N. E. The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction, 13, 1 (2001), 421--443.

[10]

Hornbæk, K. and Frøkjær, E. Comparison of techniques for matching of usability problem descriptions. Interacting with computers, 20, 6 (2008), 505--514.

Digital Library

[11]

Howarth, J., Smith-Jackson, T., Hartson, R. Supporting novice usability practitioners with usability engineering tools. Int. Journal of Human-Computer Studies, 67 (2009), 533--549.

Digital Library

[12]

Kjeldskov, J., Skov, M. and Stage, J. Instant Data Analysis. In Proc. Nordichi 2004 (2004), 233--240.

Digital Library

[13]

Law, E. L.-C. and Hvannberg, E. T. Consolidating usability problems with novice evaluators. In Proc. NordiCHI 2008. ACM (2008), 495--498.

Digital Library

[14]

Nørgaard, M. and Hornbæk, K. What Do Usability Evaluators Do in Practice? An Explorative Study of Think-Aloud Testing. In Proc. DIS 2006. ACM Press (2006), 209--218.

Digital Library

[15]

Redish, J., Bias, R., Bailey, R., Molich, R., Dumas, J. and Spool, J. Usability in Practice: Formative Usability Evaluations - Evolution and Revolution. In Proc. CHI 2002, ext. abstracts (2002), 885--890.

Digital Library

[16]

Rubin, J. and Chisnell, D. Handbook of usability testing (2nd. edition). Wiley Publishing, Indianapolis, IN, 2008.

[17]

Sears, A. Heuristic Walkthroughs: Finding the Problems Without the Noise. International Journal of Human-Computer Interaction, 9, 3 (1997), 213--234.

[18]

Smith, H. L., and Hyman, H. The biasing effect of interviewer expectations on survey results. Public Opinion, 14, 3 (1950), 491--506.

[19]

Skov, M. and Stage, J. Supporting Problem Identification in Usability Evalutions. In Proc. OZCHI 2005 (2005).

Digital Library

[20]

Woolrych, A., Cockton, G. and Hindmarch, M. Knowledge Resources in Usability Inspection. In Proc. HCI 2005 (2005).

Cited By

Dutronc ECaffiau SDupuy-Chessa S(2024)Usability inspections based on heuristics: towards a state of practices and requirements for supportCompanion Proceedings of the 16th ACM SIGCHI Symposium on Engineering Interactive Computing Systems10.1145/3660515.3661330(49-56)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3660515.3661330
Kuang ELi MFan MShinohara K(2024)Enhancing UX Evaluation Through Collaboration with Conversational AI Assistants: Effects of Proactive Dialogue and TimingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642168(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642168
Kuang E(2023)Crafting Human-AI Collaborative Analysis for User Experience EvaluationExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3577042(1-6)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544549.3577042
Show More Cited By

Index Terms

Analysis in usability evaluations: an exploratory study
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
2. Software and its engineering
  1. Software creation and management
    1. Designing software
      1. Software implementation planning
        Software design techniques
    2. Software development process management

Recommendations

What do usability evaluators do in practice?: an explorative study of think-aloud testing
DIS '06: Proceedings of the 6th conference on Designing Interactive systems

Think-aloud testing is a widely employed usability evaluation method, yet its use in practice is rarely studied. We report an explorative study of 14 think-aloud sessions, the audio recordings of which were examined in detail. The study shows that ...
Metaphors of human thinking for usability inspection and design

Usability inspection techniques are widely used, but few focus on users' thinking and many are appropriate only for particular devices and use contexts. We present a new technique (MOT) that guides inspection by metaphors of human thinking. The metaphors ...
Scrutinising usability evaluation: does thinking aloud affect behaviour and mental workload?

Thinking aloud is widely used for usability evaluation. The validity of the method is, however, debatable because it is generally used in a relaxed way that conflicts with the prescriptions of the classic model for obtaining valid verbalisations of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

NordiCHI '10: Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries

October 2010

889 pages

ISBN:9781605589343

DOI:10.1145/1868914

General Chairs:
Ebba Þóra Hvannberg
University of Iceland, Iceland
,
Marta Kristín Lárusdóttir
Reykjavik University, Iceland
,
Program Chairs:
Ann Blandford
University College London
,
Jan Gulliksen
Royal Institute of Technology, Sweden

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Reykjavik University
University of Iceland

In-Cooperation

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 October 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

NordiCHI '10

Sponsor:

NordiCHI '10: 6th Nordic Conference on Human-Computer Interaction

October 16 - 20, 2010

Reykjavik, Iceland

Acceptance Rates

Overall Acceptance Rate 379 of 1,572 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

18
Total Citations
View Citations
447
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)0

Reflects downloads up to 09 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Dutronc ECaffiau SDupuy-Chessa S(2024)Usability inspections based on heuristics: towards a state of practices and requirements for supportCompanion Proceedings of the 16th ACM SIGCHI Symposium on Engineering Interactive Computing Systems10.1145/3660515.3661330(49-56)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3660515.3661330
Kuang ELi MFan MShinohara K(2024)Enhancing UX Evaluation Through Collaboration with Conversational AI Assistants: Effects of Proactive Dialogue and TimingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642168(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642168
Kuang E(2023)Crafting Human-AI Collaborative Analysis for User Experience EvaluationExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3577042(1-6)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544549.3577042
Kuang EJahangirzadeh Soure EFan MZhao JShinohara K(2023)Collaboration with Conversational AI Assistants for UX Evaluation: Questions and How to Ask them (Voice vs. Text)Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581247(1-15)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581247
Kuang EJin XFan M(2022)“Merging Results Is No Easy Task”: An International Survey Study of Collaborative Data Analysis Practices Among UX PractitionersProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517647(1-16)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3517647
Soure EKuang EFan MZhao J(2022)CoUX: Collaborative Visual Analysis of Think-Aloud Usability Test Videos for Digital InterfacesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2021.311482228:1(643-653)Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1109/TVCG.2021.3114822
Hwang HLee Y(2021)Usability Problem Identification Based on Explainable Neural Network in Asynchronous Testing EnvironmentInteracting with Computers10.1093/iwc/iwab01833:2(155-166)Online publication date: 5-Aug-2021
https://doi.org/10.1093/iwc/iwab018
Kim TArconada-Alvarez SChoi Y(2021)Challenges and Workarounds of Conducting Augmented Reality Usability Tests Remotely a Case StudyHCI International 2021 - Late Breaking Papers: Design and User Experience10.1007/978-3-030-90238-4_6(63-71)Online publication date: 20-Nov-2021
https://doi.org/10.1007/978-3-030-90238-4_6
de Lima Salgado AAgostini do Amaral Lde Mattos Fortes RHortes Nisihara Chagas MJoyce G(2017)Addressing Mobile Usability and Elderly Users: Validating Contextualized HeuristicsDesign, User Experience, and Usability: Theory, Methodology, and Management10.1007/978-3-319-58634-2_28(379-394)Online publication date: 14-May-2017
https://doi.org/10.1007/978-3-319-58634-2_28
Hvannberg E(2015)Identifying and explicating knowledge on method transferUniversal Access in the Information Society10.1007/s10209-013-0340-114:2(187-202)Online publication date: 1-Jun-2015
https://dl.acm.org/doi/10.1007/s10209-013-0340-1
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten