How UX Practitioners Produce Findings in Usability Testing

Abstract

Usability testing has long been a core interest of HCI research and forms a key element of industry practice. Yet our knowledge of it harbours striking absences. There are few, if any detailed accounts of the contingent, material ways in which usability testing is actually practiced. Further, it is rare that industry practitioners’ testing work is treated as indigenous and particular (instead subordinated as a ‘compromised’ version). To service these problems, this article presents an ethnomethodological study of usability testing practices in a design consultancy. It unpacks how findings are produced in and as the work of observers analysing the test as it unfolds between moderators taking participants through relevant tasks. The study nuances conventional views of usability findings as straightforwardly ‘there to be found’ or ‘read off’ by competent evaluators. It explores how evaluators/observers collaboratively work to locate relevant troubles in the test's unfolding. However, in the course of doing this work, potential candidate troubles may also routinely be dissipated and effectively ‘ignored’ in one way or another. The implications of the study suggest refinements to current understandings of usability evaluations, and affirm the value to HCI in studying industry practitioners more deeply.

References

1. Carol M. Barnum. 2010. Usability Testing Essentials: Ready, Set ... Test! (1st ed.). Morgan Kaufmann.
2. Jordan Beck and Hamid R. Ekbia. 2018. The theory-practice gap as generative metaphor. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI’18). ACM, New York, NY, Paper 620, 11 pages.
3. Martin Belam. 2010. 10 tips for ‘ambush guerilla user testing’. (June 2010). Retrieved October 29, 2018 from http://www.currybet.net/cbet_blog/2010/06/10-tips-for-ambush-guerilla-us.php.
4. Douglas Benson and John Hughes. 1991. Method: Evidence and inference—evidence and inference in ethnomethodology. In Ethnomethodology and the Human Sciences, Graham Button (Ed.), chapter 6. Cambridge University Press, 109--136.
5. Herbert Blumer. 1986. Symbolic Interactionism: Perspective and Method. University of California Press, Berkley.
6. M. Ted Boren and Judith Ramey. 2000. Thinking aloud: Reconciling theory and practice. IEEE Transactions on Professional Communication 43, 3 (2000), 261--278.
7. Thea Borgholm and Kim Halskov Madsen. 1999. Cooperative usability practices. Communications of the ACM 42, 5 (1999), 91--97.
8. John Bowers and James Pycock. 1994. Talking through design: Requirements and resistance in cooperative prototyping. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’94), Beth Adelson, Susan Dumais, and Judith Olson (Eds.). ACM, New York, NY, 299--305.
9. Anders Bruun and Jan Stage. 2012. Training software development practitioners in usability testing: An assessment acceptance and prioritization. In Proceedings of the 24th Australian Computer-Human Interaction Conference (OzCHI’12), Vivienne Farrell, Graham Farrell, Caslon Chua, Weidong Huang, Raj Vasa, and Clinton Woodward (Eds.). ACM, New York, NY, 52--60.
10. Anders A. Bruun and Jan Stage. 2015. An empirical study of the effects of three think-aloud protocols on identification of usability problems. In Human-Computer Interaction (INTERACT’15), J. Abascal, S. Barbosa, M. Fetter, T. Gross, P. Palanque, and M. Winckler (Eds.), Lecture Notes in Computer Science, vol. 9297. Springer, Cham.
11. Graham Button (Ed.). 1991. Ethnomethodology and the Human Sciences. Cambridge University Press.
12. Stuart K. Card, Thomas P. Moran, and Allen Newell. 1980. The keystroke-level model for user performance time with interactive systems. Communications of the ACM 23, 7 (1980), 396--410.
13. Stuart K. Card, Allen Newell, and Thomas P. Moran. 1983. The Psychology of Human-Computer Interaction. L. Erlbaum Assoc. Inc., Hillsdale, NJ.
14. Aaron V. Cicourel. 1964. Method and Measurement in Sociology. Free Press, New York.
15. Torkil Clemmensen, Morten Hertzum, Kasper Hornbæk, Qingxin Shi, and Pradeep Yammiyavar. 2009. Cultural cognition in usability evaluation. Interacting with Computers 21, 3 (2009), 212--220.
16. Gilbert Cockton. 2014. A critical, creative UX community: CLUF. Journal of Usability Studies 10, 1 (2014), 1--16.
17. Gilbert Cockton. 2017. Usability evaluation. In The Encyclopedia of Human-Computer Interaction (2nd ed.). Interaction Design Foundation. Retrieved September 20, 2017 from https://www.interaction-design.org/literature/book/the-encyclopedia-of-human-computer-interaction-2nd-ed/usability-evaluation.
18. Gilbert Cockton and Alan Woolrych. 2002. Sale must end: Should discount methods be cleared off HCI's shelves? Interactions 9, 5 (2002), 13--18.
19. Gillbert Cockton, Alan Woolrych, Lynne Hall, and Mark Hindmarch. 2004. Changing analysts’ tunes: The surprising impact of a new instrument for usability inspection method assessment. In People and Computers XVII — Designing for Society, E. O'Neill, P. Palanque, and P. Johnson (Eds.). Springer, London.
20. Lucas Colusso, Cynthia L. Bennett, Gary Hsieh, and Sean A. Munson. 2017. Translational resources: Reducing the gap between academic research and HCI practice. In Proceedings of the 2017 Conference on Designing Interactive Systems (DIS’17). 957--968.
21. Andy Crabtree, Peter Tolmie, and Mark Rouncefield. 2013. ‘How many bloody examples do you want?’—Fieldwork and generalisation. In Proceedings of the 13th European Conference on Computer Supported Cooperative Work (ECSCW’13). Springer, 1--20.
22. Susan M. Dray. 2009. Engaged scholars, thoughtful practitioners: The interdependence of academics and practitioners in user-centered design and usability. Journal of Usability Studies 5, 1 (2009), 1--7.
23. Joseph S. Dumas and Beth A. Loring. 2008. Moderating Usability Tests: Principles and Practices for Interacting (1st ed.). Morgan Kaufmann Publishers Inc., San Francisco, CA.
24. Joseph S. Dumas and Janice C. Redish. 1999. A Practical Guide to Usability Testing (Revised ed.). University of Chicago Press.
25. Jodi Forlizzi. 2018. Moving beyond user-centered design. Interactions 25, 5 (2018), 22--23.
26. Erin Friess. 2012. Do usability evaluators do what we think usability evaluators do? Communication Design Quarterly Review 13, 1 (2012), 9--13.
27. Erik Frøkjær and Kasper Hornbæk. 2005. Cooperative usability testing: Complementing usability tests with user-supported interpretation sessions. In Proceedings of CHI’05 Extended Abstracts on Human Factors in Computing Systems (CHI EA’05). ACM, New York, NY, 1383--1386.
28. Asbjørn Følstad and Kasper Hornbæk. 2010. Work-domain knowledge in usability evaluation: Experiences with cooperative usability testing. Journal of Systems and Software 83, 11 (2010), 2019--2030.
29. Asbjørn Følstad, Effie Law, and Kasper Hornbæk. 2012. Analysis in practical usability evaluation: A survey study. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’12). ACM, New York, NY, 2127--2136.
30. Asbjørn Følstad, Effie Lai-Chong Law, and Kasper Hornbæk. 2012. Outliers in usability testing: How to treat usability problems found for only one test participant? In Proceedings of the 7th Nordic Conference on Human-Computer Interaction: Making Sense through Design (NordiCHI'12). ACM, New York, NY, 257--260.
31. Dominic Furniss. 2008. Beyond Problem Identification: Valuing Methods in a ‘System of Usability Practice’. Ph.D. thesis, University College London, London, UK.
32. Julia Gardner. 1999. Strengthening the focus on users' working practices. Communications of the ACM 42, 5 (1999), 78--82.
33. Harold Garfinkel. 1967. Studies in Ethnomethodology. Englewood Cliffs, Prentice Hall.
34. Harold Garfinkel. 2002. Ethnomethodology's Program. Rowman and Littlefield, New York.
35. Harold Garfinkel, Michael Lynch, and Eric Livingston. 1981. The work of a discovering science construed with materials from the optically discovered pulsar. Philosophy of the Social Sciences 11, 2 (1981), 121--158.
36. Harold Garfinkel, Harvey Sacks. 1986. On formal structures of practical actions. In Ethnomethodological Studies of Work, Harold Garfinkel (Ed.). Routledge 8 Kegan Paul, London, UK, 160--193.
37. Igor Garnik, Marcin Sikorski, and Gilbert Cockton. 2014. Creative sprints: An unplanned broad agile evaluation and redesign process. In Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational (NordiCHI’14). ACM, New York, NY, 1125--1130.
38. Elizabeth Goodman. 2013. Delivering Design: Performance and Materiality in Professional Interaction Design. Ph.D. Thesis. University of California, Berkeley, CA.
39. Charles Goodwin. 1980. Restarts, pauses, and the achievement of a state of mutual gaze at turn-beginning. Sociological Inquiry 50, 3--4 (1980), 272--302.
40. Colin M. Gray. 2016. What is the nature and intended use of design methods? In Proceedings of the Design Research Society. Design Research Society, Brighton, UK.
41. Colin M. Gray. 2016. “It's more of a mindset than a method”: UX practitioners’ conception of design methods. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI’16). ACM, New York, NY, 4044--4055.
42. Colin M. Gray, Erik Stolterman, and Martin A. Siegel. 2014. Reprioritizing the relationship between HCI research and practice: Bubble-up and trickle-down effects. In Proceedings of the 2014 Conference on Designing Interactive Systems (DIS’14). ACM, New York, NY, 725--734.
43. Colin M. Gray, Austin L. Toombs, and Shad Gross. 2015. Flow of competence in UX design practice. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI’15). ACM, New York, NY, 3285--3294.
44. Wayne D. Gray and Marilyn C. Salzman. 1998. Damaged merchandise? A review of experiments that compare usability evaluation methods. Human-Computer Interaction 13, 3 (1998), 203--261.
45. Tovi Grossman, George Fitzmaurice, and Ramtin Attar. 2009. A survey of software learnability: Metrics, methodologies and guidelines. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’09). ACM, New York, NY, 649--658.
46. Steven Harrison, Maribeth Back, and Deborah Tatar. 2006. “It's just a method!”: A pedagogical experiment in interdisciplinary design. In Proceedings of the 6th Conference on Designing Interactive systems (DIS’06). ACM, New York, NY, 261--270.
47. Christian Heath and Paul Luff. 2000. Technology in Action. Cambridge University Press, Cambridge, UK, New York.
48. Christian Heath, Jon Hindmarsh, and Paul Luff. 2010. Video in Qualitative Research. Sage, London.
49. Alexa Hepburn and Galena B. Bolden. 2012. The conversation analytic approach to transcription. In The Handbook of Conversation Analysis, J. Sidnell and T. Stivers (Eds.). John Wiley 8 Sons, Ltd, Chichester, UK.
50. Kasper Hornbæk. 2008. Usability evaluation as idea generation. In Maturing Usability, Effie Lai-Chong Law, Ebba Thora Hvannberg, and Gilbert Cockton (Eds.), Human-Computer Interaction Series, chapter 12. Springer, London, 267--286.
51. Kasper Hornbæk. 2010. Dogmas in the assessment of usability evaluation methods. Behaviour 8 Information Technology 29, 1 (2010), 97--111.
52. Morten Hertzum. 2016. A usability test is not an interview. Interactions 23, 2 (2016), 82--84.
53. Morten Hertzum, Kristin D. Hansen, and Hans H. K. Andersen. 2009. Scrutinising usability evaluation: Does thinking aloud affect behaviour and mental workload? Behaviour 8 Information Technology 28, 2 (2009), 165--18.
54. Morten Hertzum and Neils Ebbe Jacobsen. 2001. The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction 13, 4 (2001), 421--443.
55. Robin Jeffries, James R. Miller, Cathleen Wharton, and Kathy Uyeda. 1991. User interface evaluation in the real world: A comparison of four techniques. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’91), Scott P. Robertson, Gary M. Olson, and Judith S. Olson (Eds.). ACM, New York, NY, 119--124.
56. Jesper Kjeldskov, Mikael B. Skov, and Jan Stage. 2004. Instant data analysis: Conducting usability evaluations in a day. In Proceedings of the 3rd Nordic Conference on Human-Computer Interaction (NordiCHI’04). ACM, New York, NY, 233--240.
57. Steve Krug. 2009. Rocket Surgery Made Easy: The Do-it-yourself Guide to Finding and Fixing Usability Problems (1st ed.). New Riders Publishing, Thousand Oaks, CA.
58. Mike Kuniavsky. 2003. Observing the User Experience. Morgan Kaufman, San Francisco, CA.
59. Carine Lallemand and Vincent Koenig. 2017. Lab testing beyond usability: Challenges and recommendations for assessing user experiences. Journal of Usability Studies 12, 3 (2017), 133--154.
60. Darryn Lavery, Gilbert Cockton, and Malcolm P. Atkinson. 1997. Comparison of evaluation methods using structured usability problem reports. Behaviour and Information Technology 16, 4--5 (1997), 246--266.
61. Gitte Lindgaard. 2006. Notions of thoroughness, efficiency, and validity: Are they valid in HCI practice? International Journal of Industrial Ergonomics 36, 12 (2006), 1069--1074.
62. Eric Livingston. 1987. Making Sense of Ethnomethodology. Routledge 8 Kegan Paul, London.
63. Michael Lynch. 2000. Against reflexivity as an academic virtue and source of privileged knowledge. Theory, Culture 8 Society 17, 3 (2000), 26--54.
64. Ritch Macefield. 2009. How to specify the participant group size for usability studies: A practitioner's guide. Journal of Usability Studies 5, 1 (2009), 34--45.
65. Michael Mair, Chris Elsey, Paul V. Smith, and Patrick G. Watson. 2018. War on video: Combat footage, vernacular video analysis and military culture from within. Ethnographic Studies 15 (2018), 83--105.
66. Tara Matthews, Tejinder Judge, and Steve Whittaker. 2012. How do designers and user experience professionals actually perceive and use personas? In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’12). ACM, New York, NY, 1219--1228.
67. Sharon McDonald, Helen Edwards, and Tingting Zhao. 2012. Exploring think-alouds in usability testing: The findings of an international survey. IEEE Transactions on Professional Communication 55, 1 (2012), 2--19.
68. Michael Medlock, Dennis Wixon, Bill Fulton, Mark Terrano, and Ramon Romero. 2002. Using the RITE Method to Improve Products: A Definition and a Case Study. Usability Professionals Association.
69. Rolf Molich, Meghan R. Ede, Klaus Kaasgaard, and Barbara Karyukin. 2004. Comparative usability evaluation. Behaviour and Information Technology 23, 1 (2004), 65--74.
70. Rolf Molich and Joseph S. Dumas. 2008. Comparative usability evaluation (CUE-4). Behaviour and Information Technology 27, 3 (2008), 263--281.
71. Mike Monteiro. 2012. Design Is a Job. A Book Apart.
72. Jakob Nielsen and Thomas K. Landauer. 1993. A mathematical model of the finding of usability problems. In Proceedings of the INTERACT’93 and CHI’93 Conference on Human Factors in Computing Systems (CHI’93). ACM, New York, NY, 206--213.
73. Mie Nørgaard and Kasper Hornbæk. 2006. What do usability evaluators do in practice?: An explorative study of think-aloud testing. In Proceedings of the 6th Conference on Designing Interactive Systems (DIS’06). ACM, New York, NY, 209--218.
74. Gary M. Olson and Thomas P. Moran. 1998. Commentary on “Damaged merchandise?”. Human-Computer Interaction 13, 3 (1998), 263--323.
75. Trevor Pinch. 1993. “Testing -- one, two, three ... testing!”: Toward a sociology of testing. Science, Technology, 8 Human Values 18, 1 (1993), 25--41.
76. Melvin Pollner. 1991. Left of ethnomethodology: The rise and decline of radical reflexivity. American Sociological Review 56, 3 (1991), 370--380.
77. Stuart Reeves. 2017. Usability in vivo. Human-Computer Interaction 33 (2017), 190--194. Commentary on The Usability Construct: A Dead End?
78. Stephanie Rosenbaum. 2008. The future of usability evaluation: Increasing impact on value. In Maturing Usability, Effie Lai-Chong Law, Ebba Thora Hvannberg, and Gilbert Cockton (Eds.), Human-Computer Interaction Series, chapter 15. Springer, London, 344--378.
79. Harvey Sacks. 1992. Lectures on Conversation. Oxford, Blackwell.
80. Harvey Sacks, Emanuel A. Schegloff, and Gail Jefferson. 1974. A simplest systematics for the organization of turn-taking for conversation. Language 50, 4, Part 1 (1974), 696--735.
81. Harvey Sacks. 1984. Notes on methodology. In Structures of Social Action: Studies in Conversation Analysis, John Heritage and J. Maxwell Atkinson (Eds.). Cambridge, Cambridge University Press, 2--27.
82. Emanuel A. Schegloff. 1987. Analyzing single episodes of interaction: An exercise in conversation analysis. Social Psychology Quarterly 50, 2 (1987), 101--114.
83. Emanuel A. Schegloff. 2007. A tutorial on membership categorization. Journal of Pragmatics 39, 3 (2007), 462--482.
84. Emmanuel A. Schegloff and Harvey Sacks. 1973. Opening Up Closings. Semiotica 8, 4 (1973), 289--327.
85. Wes Sharrock and Robert J. Anderson. 1986. The Ethnomethodologists. Ellis Horwood, Chichester.
86. Qingxin Shi. 2008. A field study of the relationship and communication between Chinese evaluators and users in thinking aloud usability tests. In Proceedings of the 5th Nordic Conference on Human-Computer Interaction: Building Bridges (NordiCHI’08). ACM, New York, NY, 344--352.
87. Janet M. Six and Ritch Macefield. 2016. How to determine the right number of participants for usability studies. UXmatters (2016). Retrieved from https://www.uxmatters.com/mt/archives/2016/01/how-to-determine-the-right-number-of-participants-for-usability-studies.php.
88. Jared Spool and Will Schroeder. 2001. Testing web sites: Five users is nowhere near enough. In Proceedings of Extended Abstracts on Human Factors in Computing Systems (CHI EA’01). ACM, New York, NY, 285--286.
89. Philippe Sormani, Esther González-Martínez, and Alain Bovet. 2011. Discovering work: A topical introduction. Ethnographic Studies 12 (2011), 1--11.
90. Lucy A. Suchman. 1987. Plans and Situated Actions: The Problem of Human-Machine Communication. Cambridge University Press, New York, NY.
91. Margaret H. Szymanski, Erik Vinkhuyzen, Paul M. Aoki, and Allison Woodruff. 2006. Organizing a remote state of incipient talk: Push-to-talk mobile radio interaction. Language in Society 35, 3 (2006), 393--418.
92. Maryam Tohidi, William Buxton, Ronald Baecker, and Abigail Sellen. 2006. User sketches: A quick, inexpensive, and effective way to elicit more reflective user feedback. In Proceedings of the 4th Nordic Conference on Human-Computer Interaction: Changing Roles (NordiCHI’06), Anders Mørch, Konrad Morgan, Tone Bratteteig, Gautam Ghosh, and Dag Svanaes (Eds.). ACM, New York, NY, 105--114.
93. Noam Tractinsky. 2018. The usability construct: A dead end? Human--Computer Interaction 33, 2 (2018), 131--177.
94. Thomas Tullis and William Albert. 2013. Measuring User Experience: Collecting, Analyzing, and Presenting Usability Metrics (2nd ed.). Morgan Kaufmann.
95. Arnold Vermeeren. 2009. What's the Problem? Studies on Identifying Usability Problems in User Tests. Ph.D. Thesis. Technische Universiteit Delft, Delft, Netherlands.
96. Dirk vom Lehn. 2014. Harold Garfinkel: The Creation and Development of Ethnomethodology. Routledge, New York.
97. Stephanie Wilson, Mathilde Bekker, Peter Johnson, and Hilary Johnson. 1997. Helping and hindering user involvement — A tale of everyday design. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’97). ACM, New York, NY, 178--185.
98. Dennis Wixon. 2003. Evaluating usability methods: Why the current literature fails the practitioner. Interactions 10, 4 (2003), 28--34.
99. Tracee Vetting Wolf, Jennifer A. Rode, Jeremy Sussman, and Wendy A. Kellogg. 2006. Dispelling “design” as the black art of CHI. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’06), Rebecca Grinter, Thomas Rodden, Paul Aoki, Ed Cutrell, Robin Jeffries, and Gary Olson (Eds.). ACM, New York, NY, 521--530.
100. Ludwig Wittgenstein. 2009. Philosophical Investigations, P. M. S. Hacker and Joachim Schulte (Eds.). Blackwell, Oxford, UK.
101. Steve Woolgar. 1990. Configuring the user: The case of usability trials. The Sociological Review 38, 1 (1990), 58--99.
102. Alan Woolrych, Kasper Hornbæk, Erik Frøjær, and Gilbert Cockton. 2011. Ingredients and meals rather than recipes: A proposal for research that does not treat usability evaluation methods as indivisible wholes. International Journal of Human-Computer Interaction 27, 10 (2011), 940--970.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • Empirical studies in HCI

    • HCI design and evaluation methods

      • Usability testing