skip to main content
10.1145/1753326.1753685acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Think-aloud protocols: a comparison of three think-aloud protocols for use in testing data-dissemination web sites for usability

Published: 10 April 2010 Publication History

Abstract

We describe an empirical, between-subjects study on the use of think-aloud protocols in usability testing of a federal data-dissemination Web site. This double-blind study used three different types of think-aloud protocols: a traditional protocol, a speech-communication protocol, and a coaching protocol. A silent condition served as the control. Eighty participants were recruited and randomly pre-assigned to one of four conditions. Accuracy and efficiency measures were collected, and participants rated their subjective satisfaction with the site. Results show that accuracy is significantly higher in the coaching condition than in the other conditions. The traditional protocol and the speech-communication protocol are not statistically different from each other with regard to accuracy. Participants in the coaching condition are more satisfied with the Web site than participants in the traditional or speech-communication condition. In addition, there are no significant differences with respect to efficiency (time-on-task). This paper concludes with recommendations for usability practitioners.

References

[1]
Boren, T., and Ramey, J. Thinking aloud: Reconciling theory and practice. IEEE Transactions on Professional Communication 43, 3 (2000), 261--278.
[2]
Bowers, V., and Snyder, H. Concurrent versus retrospective verbal protocol for comparing window usability. In Proc Human Factors Society 34th Annual Meeting, (1990), 1270--1274.
[3]
Chin, J.P., Diehl, V., and Norman, K.L. (1988). Development of an instrument measuring user satisfaction of the human-computer interface. In Proc CHI 88, ACM Press (1988), 213--218.
[4]
Deffner, G. Verbal protocols as a research tool in human factors. In Proc Human Factors Society 34th Annual Meeting, (1990), 1263--1264.
[5]
Dumas, J and Redish, J. A Practical Guide to Usability Testing. Intellect Press, Portland, OR, USA, 1999.
[6]
Ericsson, K.A. and Simon, H.A. Protocol Analysis: Verbal Reports As Data. (Revised ed.) MIT Press, Cambridge, MA, USA, 1996.
[7]
Hair, J.F., Anderson, R.E., Tatham, R.L., and Black, W.C. Multivariate data analysis. (5th ed.). Prentice Hall, Englewood Cliffs, NJ, USA, 1998.
[8]
Hertzum, M, Hansen, K., and Anderson, H. Scrutinising usability evaluation: does thinking aloud affect behaviour and mental workload? Behaviour & Information Technology 28, 2 (2009), 165--181.
[9]
Hughes, J., and Parkes, S. Trends in the use of verbal protocol analysis in software engineering research. Behaviour & Information Technology 22, 2 (2003), 127--140.
[10]
Krahmer, E., and Ummelen, N. Thinking about thinking aloud: A comparison of two verbal protocols for usability testing. IEEE Transactions on Professional Communication 47, 2 (2004), 105--117.
[11]
Khasawneh, S. A. Construct validation of an Arabic version of the learning transfer system of the learning transfer system for use in Jordan. (2004), Dissertation: Louisiana State University.
[12]
Nielsen, J. Usability Engineering. Morgan Kaufmann, New York, NY, USA, 1993.
[13]
Norgaard, M., and Hornbaek, K., What do usability evaluators do in Practice? An explorative study of think-aloud testing. In DIS 2006, ACM Press (2006), 209--219.
[14]
Novotny, E., and Cahoy, E. If we teach, do they learn?: The impact of instruction on online catalog search strategies. Portal: Libraries and the Academy, 6, 2 (2006), 155--167.
[15]
Rhenius, D., and Deffner, G. Evaluation of concurrent thinking aloud using eye-tracking data. In Proc Human Factors Society 34th Annual Meeting, (1990), 1265--1269.
[16]
Romaine, S. Communicating Gender. Erlbaum, Mahwah, NJ, USA, 1999.
[17]
Rubin, J. Handbook of usability testing: How to plan, design, and conduct effective tests. Wiley, New York: NY, USA, 1994.
[18]
SAS Institute, Inc. Statistical Analysis Software (SAS 8.02), 2001, Software package.
[19]
Tabachnick, B.G and Fidell, L.S. Using Multivariate Statistics (3rd Ed.). HarperCollins, New York, NY, USA, 1996.
[20]
Van Den Haak, M., De Jong, M., and Schellens, P. Retrospective vs. concurrent think-aloud protocols: Testing the usability of an online library catalogue. Behaviour & Information Technology 22, 5 (2003), 339--351.
[21]
Van Den Haak, M., De Jong, M., and Schellens, P. Employing think--aloud protocols and constructive interaction to test the usability of online library catalogues: A methodological comparison. Interacting with Computers 16, 6 (2004), 1153--1170.
[22]
Van Den Haak, M., De Jong, M., and Schellens, P. Evaluation of an informational Web site: Three variants of the think-aloud method compared. Technical Communication 54, 1 (2007), 58--71.
[23]
Ward, N and Tsukahara, W. Prosodic features which cue back-channel feedback in English and Japanese. Journal of Pragmatics 32, (2000), 1177--1207.
[24]
Wright, R., and Converse, S. Method bias and concurrent verbal protocol in software usability testing. Proc Human Factors Society 36th Annual Meeting, (1992), 1220--1224.
[25]
Yngve, V. On getting a word in edgewise. Papers from the sixth regional meeting {of the} Chicago Linguistic Society, (1970), 567--577.

Cited By

View all
  • (2025)Novel Procedures for Evaluating Autism Online in a Culturally Diverse Population of Children: Protocol for a Mixed Methods Pathway Development StudyJMIR Research Protocols10.2196/5574114(e55741)Online publication date: 11-Feb-2025
  • (2024)Evaluating the Usability and Quality of a Clinical Mobile App for Assisting Physicians in Head Computed Tomography Scan Ordering: Mixed Methods StudyJMIR Human Factors10.2196/5579011(e55790)Online publication date: 9-Sep-2024
  • (2024)Usability Study of Security Features in Programmable Logic ControllersProceedings of the 2024 European Symposium on Usable Security10.1145/3688459.3688471(200-219)Online publication date: 30-Sep-2024
  • Show More Cited By

Index Terms

  1. Think-aloud protocols: a comparison of three think-aloud protocols for use in testing data-dissemination web sites for usability

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      CHI '10: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
      April 2010
      2690 pages
      ISBN:9781605589299
      DOI:10.1145/1753326
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 10 April 2010

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. experimental design
      2. think aloud
      3. usability testing
      4. user testing
      5. user-centered design
      6. verbalization

      Qualifiers

      • Research-article

      Conference

      CHI '10
      Sponsor:

      Acceptance Rates

      Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

      Upcoming Conference

      CHI 2025
      ACM CHI Conference on Human Factors in Computing Systems
      April 26 - May 1, 2025
      Yokohama , Japan

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)266
      • Downloads (Last 6 weeks)31
      Reflects downloads up to 16 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2025)Novel Procedures for Evaluating Autism Online in a Culturally Diverse Population of Children: Protocol for a Mixed Methods Pathway Development StudyJMIR Research Protocols10.2196/5574114(e55741)Online publication date: 11-Feb-2025
      • (2024)Evaluating the Usability and Quality of a Clinical Mobile App for Assisting Physicians in Head Computed Tomography Scan Ordering: Mixed Methods StudyJMIR Human Factors10.2196/5579011(e55790)Online publication date: 9-Sep-2024
      • (2024)Usability Study of Security Features in Programmable Logic ControllersProceedings of the 2024 European Symposium on Usable Security10.1145/3688459.3688471(200-219)Online publication date: 30-Sep-2024
      • (2024)Concurrent or Retrospective Thinking Aloud in Usability Tests: A Meta-Analytic ReviewACM Transactions on Computer-Human Interaction10.1145/366532731:3(1-29)Online publication date: 17-May-2024
      • (2024)Evaluating Search System Explainability with Psychometrics and CrowdsourcingProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657796(1051-1061)Online publication date: 10-Jul-2024
      • (2024)‘We Do Not Have the Capacity to Monitor All Media’: A Design Case Study on Cyber Situational Awareness in Computer Emergency Response TeamsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642368(1-16)Online publication date: 11-May-2024
      • (2024)HCI/UX/UI Education Issues and Learning Challenges for Thai Undergraduate Students2024 8th International Conference on Information Technology (InCIT)10.1109/InCIT63192.2024.10810605(132-137)Online publication date: 14-Nov-2024
      • (2024)Evaluating both usability and accessibility: the case of access@tour by action – a digital solution for accessible tourismDisability and Rehabilitation: Assistive Technology10.1080/17483107.2024.2409810(1-17)Online publication date: 30-Sep-2024
      • (2024)Methodological Challenges in Victimisation StudiesUnderstanding Prisoner Victimisation10.1007/978-3-031-54350-0_6(143-173)Online publication date: 17-Apr-2024
      • (2023)Talking about Thinking Aloud: Perspectives from Interactive Think-Aloud PractitionersJournal of User Experience 10.5555/3605526.360552818:3(113-132)Online publication date: 20-Jun-2023
      • Show More Cited By

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media