Supporting problem identification in usability evaluations

ABSTRACT

Identification of usability problems is a key element in a usability evaluation of an interactive system. This paper presents and discusses results from an empirical study of problem identification. The study includes an experiment, where it was examined to what extent a conceptual tool can support problem identification in a usability evaluation. A group of novice evaluators was divided into two, and one of the groups received a conceptual tool and a related presentation of ideas and examples. Both groups conducted a usability evaluation based on the same recording of a user applying a web-based system to solve a series of tasks. It is concluded that the conceptual tool and training in using the tool improve the problem identification performance of a group of inexperienced usability evaluators.

REFERENCES

• 1: Gilbert Cockton , Alan Woolrych , Mark Hindmarch, Reconditioned merchandise: extended structured report formats in usability inspection, CHI '04 extended abstracts on Human factors in computing systems, April 24-29, 2004, Vienna, Austria [doi>10.1145/985921.986083]
• 2: Joseph F. Dumas , Janice C. Redish, A Practical Guide to Usability Testing, Greenwood Publishing Group Inc., Westport, CT, 1993
• 3: Fath, J. L., Mann, T. L., and Holzman, T. G. (1994) A Practical Guide to Using Software Usability Labs: Lessons Learned at IBM. Behaviour & Information Technology 13, 1--2, 25--35.
• 4: Frøkjær, E., Hertzum, M. and Hornbæk, K. (2000) Measuring Usability: Are Effectiveness, Efficiency, and Satisfaction Really Correlated? CHI Letters, 2, 1, 345--352.
• 5: Hertzum, M. and Jacobsen, N. E. (2001). The evaluator effect: A chilling fact about us-ability evaluation methods. International Journal of Human-Computer Interaction, 13, 4, 421--443.
• 6: Hornbæk, K. & Frøøkjær, E. (2004) Usability Inspection by Metaphors of Human Thinking Compared to Heuristic Evaluation. International Journal of Human-Computer Interaction, 17(3) 357--374.
• 7: ISO 9241-11. Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs) Part 11: Guidance on usability. ISO 1997
• 8: Niels Ebbe Jacobsen , Morten Hertzum , Bonnie E. John, The evaluator effect in usability tests, CHI 98 conference summary on Human factors in computing systems, p.255-256, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/286498.286737]
• 9: Robin Jeffries , James R. Miller , Cathleen Wharton , Kathy Uyeda, User interface evaluation in the real world: a comparison of four techniques, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.119-124, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108862]
• 10: Claire-Marie Karat , Robert Campbell , Tarra Fiegel, Comparison of empirical testing and walkthrough methods in user interface evaluation, Proceedings of the SIGCHI conference on Human factors in computing systems, p.397-404, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142873]
• 11: Lavery, D. Cockton, G. and Atkinson, M. P. (1997) Comparison of Evaluation Methods Using Structured Usability Problem Reports. Behaviour and Information Technology, 16(4), 246--266.
• 12: Rolf Molich , Jakob Nielsen, Improving a human-computer dialogue, Communications of the ACM, v.33 n.3, p.338-348, March 1990 [doi>10.1145/77481.77486]
• 13: Molich, R. (2000) User-Friendly Web Design (in Danish). Copenhagen: Ingeniøren Books.
• 14: Jakob Nielsen, Finding usability problems through heuristic evaluation, Proceedings of the SIGCHI conference on Human factors in computing systems, p.373-380, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142834]
• 15: Jakob Nielsen, Usability Engineering, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993
• 16: Rohn, J. A. (1994) The Usability Engineering Laboratories at Sun Microsystems. Behaviour & Information Technology 13, 1--2, 25--35.
• 17: Jeffrey Rubin , Theresa Hudson, Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests, John Wiley & Sons, Inc., New York, NY, 1994
• 18: Skov, M. B. and Stage, J. (2001) A Simple Approach to Web-Site Usability Testing. C. Stephanidis (Ed.), Universal Access in HCI. Towards an Information Society for All, pp. 737--741. London: Lawrence Erlbaum.
• 19: Skov, M. B. and Stage, J. (2003a) Enhancing Usability Testing Skills of Novice Testers: A Longitudinal Study. Proceedings of the 2nd Conference on Universal Access in Computer-Human Interaction. London: Lawrence-Erlbaum.
• 20: Skov, M. B. and Stage, J. (2003b) Supporting Novices in Detecting Web Site Usability Problems: A Comparison of the Think-Aloud and Questionnaire Methods. Proceedings of the 10th Conference on Human-Computer Interaction. London: Lawrence-Erlbaum.
• 21: Skov, M. B. and Stage, J. (2004) Integrating Usability Design and Evaluation: Training Novice Evaluators in Usability Testing. K. Hornbæk and J. Stage (Eds.), Proceedings of the Workshop on Improving the Interplay between Usability Evaluation and User Interface Design, NordiCHI 2004, pp. 31--35. Aalborg University, Department of Computer Science, HCI-Lab Report no. 2004/2.

INDEX TERMS

Keywords:
novice evaluators, think-aloud, usability