What do usability evaluators do in practice?: an explorative study of think-aloud testing

ABSTRACT

Think-aloud testing is a widely employed usability evaluation method, yet its use in practice is rarely studied. We report an explorative study of 14 think-aloud sessions, the audio recordings of which were examined in detail. The study shows that immediate analysis of observations made in the think-aloud sessions is done only sporadically, if at all. When testing, evaluators seem to seek confirmation of problems that they are already aware of. During testing, evaluators often ask users about their expectations and about hypothetical situations, rather than about experienced problems. In addition, evaluators learn much about the usability of the tested system but little about its utility. The study shows how practical realities rarely discussed in the literature on usability evaluation influence sessions. We discuss implications for usability researchers and professionals, including techniques for fast-paced analysis and tools for capturing observations during sessions.

REFERENCES

• 1: Jonathan Arnowitz , Monica Heidelberg , Diana Gray , Michael Arent , Naomi Dorsch, The stakeholder forest: designing an expenses application for the enterprise, CHI '05 extended abstracts on Human factors in computing systems, April 02-07, 2005, Portland, OR, USA [doi>10.1145/1056808.1056810]
• 2: Beyer, H. & Holtzblatt, K. Contextual Design, Morgan Kaufman Publishers, San Francisco, 1998.
• 3: Boivie, I., Åborg, C., Persson, J., & Lööfberg, M. Why Usability for Lost or Usability in in-House Software Development, Interacting with Computers, 15 (2003), 623--639.
• 4: Boren, M. T. & Ramey, J. Thinking Aloud: Reconciling Theory and Practice, IEEE Transactions on Professional Communication, 43, 3 (2000), 261--277.
• 5: Carter, L. & Yeats, D. The Role of Highlights Video in Usability Testing: Rhetorical and Generic Expectations, Technical Communications, 52, 2 (2005), 1--7.
• 6: Chi, M. T. H. Quantifying Qualitative Analyses of Verbal Data: A Practical Guide, The Journal of the Learning Sciences, 6, 3 (1997), 271--315.
• 7: Gilbert Cockton , Darryn Lavery , Alan Woolrych, Inspection-based evaluations, The human-computer interaction handbook: fundamentals, evolving technologies and emerging applications, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 2002
• 8: Cockton, G., Woolrych, A., Hall, L., & Hidemarch, M. Changing Analysts' Tunes: The Surprising Impact of a New Instrument for Usability Inspection Method Assessment, Proc. HCI 2003, Springer Verlag (2003), 145--162.
• 9: Joseph S. Dumas, User-based evaluations, The human-computer interaction handbook: fundamentals, evolving technologies and emerging applications, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 2002
• 10: Joseph S. Dumas , Rolf Molich , Robin Jeffries, Describing usability problems: are we sending the right message?, interactions, v.11 n.4, July + August 2004 [doi>10.1145/1005261.1005274]
• 11: Joseph S. Dumas , Janice C. Redish, A Practical Guide to Usability Testing, Intellect Books, Exeter, UK, 1999
• 12: Ericsson, K. A. & Simon, H. Protocol Analysis: Verbal Reports As Data, Revised Edition, MIT Press, Cambridge, MA, 1993.
• 13: Erik Frøkjær , Kasper Hornbæk, Cooperative usability testing: complementing usability tests with user-supported interpretation sessions, CHI '05 extended abstracts on Human factors in computing systems, April 02-07, 2005, Portland, OR, USA [doi>10.1145/1056808.1056922]
• 14: Jan Gulliksen , Inger Boivie , Jenny Persson , Anders Hektor , Lena Herulf, Making a difference: a survey of the usability profession in Sweden, Proceedings of the third Nordic conference on Human-computer interaction, p.207-215, October 23-27, 2004, Tampere, Finland [doi>10.1145/1028014.1028046]
• 15: Hertzum, M. User Testing in Industry: A Case Study of Laboratory, Workshop, and Field Tests, Proc. ERCIM Workshop on User Interfaces for All, (1999), 59--72.
• 16: Hertzum, M. & Jacobsen, N. E. The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods, International Journal of Human-Computer Interaction, 13 (2001), 421--443.
• 17: Kasper Hornbæk , Erik Frøkjær, Comparing usability problems and redesign proposals as input to practical systems development, Proceedings of the SIGCHI conference on Human factors in computing systems, April 02-07, 2005, Portland, Oregon, USA [doi>10.1145/1054972.1055027]
• 18: Kasper Hornbæk , Erik Frøkjær, Two psychology-based usability inspection techniques studied in a diary experiment, Proceedings of the third Nordic conference on Human-computer interaction, p.3-12, October 23-27, 2004, Tampere, Finland [doi>10.1145/1028014.1028016]
• 19: Iivari, N. Usability Specialists - 'a Mommy Mob', 'Realistic Humanists' or 'Staid Researchers'? An Analysis of Usability Work in Software Product Development, Proc. Interact 2005, Edizioni Guiseppe Laterza, (2005), 418--430.
• 20: Jacobsen, N. E. & John, B. E. Two Case Studies in Using Cognitive Walkthroughs for Interface Evaluation, CMU-CS-00-132 (2000).
• 21: Robin Jeffries , James R. Miller , Cathleen Wharton , Kathy Uyeda, User interface evaluation in the real world: a comparison of four techniques, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.119-124, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108862]
• 22: Bonnie E. John, Beyond the UI: product, process and passion, Proceedings of the third Nordic conference on Human-computer interaction, p.285-286, October 23-27, 2004, Tampere, Finland [doi>10.1145/1028014.1028058]
• 23: Bonnie E. John , Matthew M. Mashyna, Evaluating a multimedia authoring tool, Journal of the American Society for Information Science, v.48 n.11, p.1004-1022, Nov. 1997 [doi>10.1002/(SICI)1097-4571(199711)48:11<1004::AID-ASI4>3.3.CO;2-T]
• 24: Bonnie E. John , Hilary Packer, Learning and using the cognitive walkthrough method: a case study approach, Proceedings of the SIGCHI conference on Human factors in computing systems, p.429-436, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223962]
• 25: Claire-Marie Karat , Robert Campbell , Tarra Fiegel, Comparison of empirical testing and walkthrough methods in user interface evaluation, Proceedings of the SIGCHI conference on Human factors in computing systems, p.397-404, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142873]
• 26: Molich, Rolf, User testing, Discount user testing, 2003, www.dialogdesign.dk.
• 27: Rolf Molich , Meghan R. Ede , Klaus Kaasgaard , Barbara Karyukin, Comparative usability evaluation, Behaviour & Information Technology, v.23 n.1, p.65-74, January-February 2004 [doi>10.1080/0144929032000173951]
• 28: Jakob Nielsen, Usability Engineering, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993
• 29: Jakob Nielsen, Finding usability problems through heuristic evaluation, Proceedings of the SIGCHI conference on Human factors in computing systems, p.373-380, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142834]
• 30: Steven Pace, A grounded theory of the flow experiences of web users, International Journal of Human-Computer Studies, v.60 n.3, p.327-363, March 2004 [doi>10.1016/j.ijhcs.2003.08.005]
• 31: Paul Sawyer , Alicia Flanders , Dennis Wixon, Making a difference—the impact of inspections, Proceedings of the SIGCHI conference on Human factors in computing systems: common ground, p.376-382, April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/238386.238579]
• 32: Rick Spencer, The streamlined cognitive walkthrough method, working around social constraints encountered in a software development company, Proceedings of the SIGCHI conference on Human factors in computing systems, p.353-359, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332456]
• 33: Strauss, A. & Corbin, J. Basics of Qualitative Research - Techniques and Procedures for Developing Grounded Theory, Sage Publications, California, (1998).
• 34: Szczur, M. Usability Testing - on a Budget: a NASA Usability Test Case Study, Behaviour & Information Technology, 13 (1994), 106--118.
• 35: Karel Vredenburg , Ji-Ye Mao , Paul W. Smith , Tom Carey, A survey of user-centered design practice, Proceedings of the SIGCHI conference on Human factors in computing systems: Changing our world, changing ourselves, April 20-25, 2002, Minneapolis, Minnesota, USA [doi>10.1145/503376.503460]
• 36: Stephanie Wilson , Mathilde Bekker , Peter Johnson , Hilary Johnson, Helping and hindering user involvement — a tale of everyday design, Proceedings of the SIGCHI conference on Human factors in computing systems, p.178-185, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258699]
• 37: Dennis Wixon, Evaluating usability methods: why the current literature fails the practitioner, interactions, v.10 n.4, July + August 2003 [doi>10.1145/838830.838870]
• 38: Zirkler, D. & Ballman, D. R. Usability Testing in a Competive Market: Lessons Learned, Behaviour and Information Technology, 13, 1&2 (1994), 191--197.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.2 Design Tools and Techniques

Keywords:
industrial software development, think aloud testing, usability evaluation, user-centered design