skip to main content
10.1145/1124772.1124961acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
Article

The validity of the stimulated retrospective think-aloud method as measured by eye tracking

Published: 22 April 2006 Publication History

Abstract

Retrospective Think aloud (RTA) is a usability method that collects the verbalization of a user's performance after the performance is over. There has been little work done to investigate the validity and reliability of RTA. This paper reports on an experiment investigating these issues with a form of the method called stimulated RTA. By comparing subjects' verbalizations with their eye movements, we support the validity and reliability of stimulated RTA: the method provides a valid account of what people attended to in completing tasks, it has a low risk of introducing fabrications, and its validity isn't affected by task complexity. More detailed analysis of RTA shows that it also provides additional information about user's inferences and strategies in completing tasks. The findings of this study provide valuable support for usability practitioners to use RTA and to trust the users' performance information collected by this method in a usability study.

References

[1]
Bell, B., et al. Usability testing of a graphical programming system: things we missed in a programming walkthrough. In Proc. CHI'91. ACM Press (1991), 7--12.
[2]
Bowers, V.A.&H.L. Snyder. Concurrent versus Retrospective Verbal Protocol for Comparing Window Usability. In Proc. of the Human Factors Society 34th Annual Meeting. (1990), 1270--1274.
[3]
Branch, J.L. Investigating the Information-Seeking Processes of Adolescents: The Value of Using Think Alouds and Think Afters. Library & Information Science Research. 22,4 (2000), 371--392.
[4]
Campbell, D.J. Task Complexity: A review and analysis. The Academy of Management Review. 13,1 (1988), 40--52.
[5]
Capra, M.G. Contemporaneous versus Retrospective User-Reported Critical Incidents in Usability Evaluation. In Proc. of Human Factors Society, 46 th Annual Meeting. (2002), 1973--1977.
[6]
Card, S.K., et al. Information scent as a driver of web behavior graphs: results of a protocol analysis method for web usability. In Proc. CHI'01. ACM Press (2001), 498--505.
[7]
Choi, B., et al. A Qualitative Cross-National Study of Cultural Influences on Mobile Data Service Design. In Proc. CHI 2005. ACM Press (2005), 661--670.
[8]
Ericsson, K.A.&H.A. Simon, Protocol analysis: Verbal Reports as Data. 1993: Cambridge, MA: MIT Press.
[9]
Gapra, M.G. Comtemporaneous versus Retrospective User-reported Critical Incidents in Usability Evaluation. In Proceedings of the Human Factors and Ergonomics Society, 46th Annual Meeting. (2002), 1973--1977.
[10]
Geiselman, R.E.&F.S. Bellezza. Eye-movements and overt rehearsal in word recall. Journal of Experimental Psychology: Human Learning and Memory. 3,3 (1977), 305--315.
[11]
Gero, J.S.&H.-h. Tang. Differences between retrospective and concurrent protocols in revealing the process-oriented aspects of the design process. Design Studies. 21,3 (2001), 283--295.
[12]
Goldberg, J.H.&A.M. Wichansky, Eye tracking in usability evlauation: A practitioner's guide., in The Mind's Eyes: Cognitive and Applied Aspects of Eye Movements, R. Radach, et al., Editors.(2003), Elsevier Science: Oxford. 493--516.
[13]
Gray, W.D.&M.C. Salzman. Damaged merchandise? A review of experiments that compare usability evaluation methods. Human-Computer Interaction. 13,3 (1998), 203--261.
[14]
Kensing, F. Prompted Reflections: A Technique for Understanding complex work. Interactions. Jan-Feb.,(1998), 7--15.
[15]
Kjeldskov, J.&M.B. Skov. Creating Realistic Laboratory Settings: Comparative Studies of Three Think aloud Usability Evaluations of a Mobile System. In Proc. of the 9th IFIP TC13 INTERACT 2003. (2003), 663 -- 670.
[16]
Mankoff, J., et al. Is Your Web Page Accessible? A Comparative Study of Methods for Assessing Web Page Accessibility for the Blind. In Proc. of CHI'05. ACM Press (2005), 41--50.
[17]
Nielson, J., Usability Engineering. 1993: Cambridge, MA: AP Professional.
[18]
Page, C.&M. Rahimi. Concurrent and Retrospective Verbal Protocols in Usability Testing: Is There Value Added In Collecting Both? In Proc. of the Human Factors and Ergonomics Society, 39th Annual Meeting. (1995), 223--227.
[19]
Preece, J., Human-Computer Interaction. 1994: Addison-Wesley, England.
[20]
Preece, J., et al., Interaction Design: Beyond Human-Computer Interaction. 2002: John Wiley & Sons.
[21]
Ramey, J., et al., Adaptation of an Ethnographic Method for Investigation the Task Domain in Diagnostic Radiology, in A Field Methods Casebook for Software Design, e. D. Wixon and J. Ramey, Editor.(1996), John Wiley and Sons. 1--15.
[22]
Rhenius, D.&G. Deffner. Evaluation of Concurrent Thinking Aloud using Eye-tracking Data. Proc. of the Human Factors and Ergonomics Society 34th Annual Meeting. (1990), 1265--1269.
[23]
Rowley, D.E. Usability Testing in the field: bringing the laboratory to the user. In Proc. CHI'94. ACM Press (1994), 252 -- 257.
[24]
Russo, J.E., et al. The Validity of Verbal Protocols. Memory and Cognition. 17,6 (1989), 759--769.
[25]
Sankoff, D.&J.B. Kruskal, An overview of sequence comparison, in Time Warps, String Edits, and Macro-Molecules: The Theory and Practice of Sequence Comparison.(1983), Addison-Wesley.
[26]
Soukoreff, R.W.&I.S. MacKenzie. Measuring errors in text entry tasks: An application of the Levenshtein string distance statistic. In Proc. CHI'01. ACM Press (2001), 319--320.
[27]
St. Amant, R.&M.O. Riedl. A perception/action substrate for cognitive modeling in HCI. International Journal of Human-Computer Studies. 55,1 (2001), 15--39.
[28]
Suwa, M.&B. Tversky. What architects see in their sketches: implications for design tools. In Proc. CHI'96. ACM Press (1996), 191--192.
[29]
Teague, R., et al. Concurrent vs. Post-Task Usability Test Ratings. In Proc. CHI'01. ACM Press (2001), 289--290.
[30]
Van den Haak, M.J., et al. Retrospective vs. concurrent think-aloud protocols: testing the usability of an online library catalogue. Behaviour& Information Technology. 22,5 (2003), 339--351.
[31]
Waes, L.V. Thinking Aloud as a Method for Testing the Usability of Websites: The influence of Task Variation on the Evaluation of Hypertext. IEEE Transactions on Professional Communication. 43,3 (2000), 279--291.
[32]
Williams, T.R., et al. Does Isolating a Visual Element Call Attention to It? Results of an Eye-tracking Investigation of the Effects of Isolation on Emphasis. Technical Communication. 52,1 (2005), 21--26.
[33]
Wood, R.E. Task Complexity: Definition of the construct. Organizational Behavior and Human Decision Processes. 37,(1986), 60--82.

Cited By

View all
  • (2025)A framework for testing theories of criminal decision-making using VR scenarios and process tracing and its application to situational action theoryPsychology, Crime & Law10.1080/1068316X.2025.2466073(1-31)Online publication date: 18-Feb-2025
  • (2024)Learning from (re)experience: What mobile eye-tracking video can help us learn about the cognitive processes of teachingFrontiers in Education10.3389/feduc.2024.12998969Online publication date: 13-Mar-2024
  • (2024)Concurrent or Retrospective Thinking Aloud in Usability Tests: A Meta-Analytic ReviewACM Transactions on Computer-Human Interaction10.1145/366532731:3(1-29)Online publication date: 17-May-2024
  • Show More Cited By

Index Terms

  1. The validity of the stimulated retrospective think-aloud method as measured by eye tracking

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '06: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
    April 2006
    1353 pages
    ISBN:1595933727
    DOI:10.1145/1124772
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 22 April 2006

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. eye tracking
    2. reliability
    3. retrospective think aloud
    4. usability research
    5. validity
    6. verbalization

    Qualifiers

    • Article

    Conference

    CHI06
    Sponsor:
    CHI06: CHI 2006 Conference on Human Factors in Computing Systems
    April 22 - 27, 2006
    Québec, Montréal, Canada

    Acceptance Rates

    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)127
    • Downloads (Last 6 weeks)17
    Reflects downloads up to 16 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)A framework for testing theories of criminal decision-making using VR scenarios and process tracing and its application to situational action theoryPsychology, Crime & Law10.1080/1068316X.2025.2466073(1-31)Online publication date: 18-Feb-2025
    • (2024)Learning from (re)experience: What mobile eye-tracking video can help us learn about the cognitive processes of teachingFrontiers in Education10.3389/feduc.2024.12998969Online publication date: 13-Mar-2024
    • (2024)Concurrent or Retrospective Thinking Aloud in Usability Tests: A Meta-Analytic ReviewACM Transactions on Computer-Human Interaction10.1145/366532731:3(1-29)Online publication date: 17-May-2024
    • (2024)Using Worked Examples for Engaging in Epistemic Programming ProjectsProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630961(443-449)Online publication date: 7-Mar-2024
    • (2024)I see an IC: A Mixed-Methods Approach to Study Human Problem-Solving Processes in Hardware Reverse EngineeringProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642837(1-20)Online publication date: 11-May-2024
    • (2024)Usability Evaluation of the Mountrash Application Using Retrospective Think Aloud and Performance Measurement Techniques2024 2nd International Conference on Software Engineering and Information Technology (ICoSEIT)10.1109/ICoSEIT60086.2024.10497501(132-137)Online publication date: 28-Feb-2024
    • (2024)Individual and collaborative infographic design and cognitive strategies in visual literacy support process: a qualitative research perspectiveJournal of Visual Literacy10.1080/1051144X.2024.234938343:2(147-172)Online publication date: 7-May-2024
    • (2024)Eye tracking-based evaluation of accessible and usable interactive systems: tool set of guidelines and methodological issuesUniversal Access in the Information Society10.1007/s10209-023-01083-xOnline publication date: 11-Jan-2024
    • (2023)The Think-Aloud Method for Evaluating the Usability of a Regional AtlasISPRS International Journal of Geo-Information10.3390/ijgi1203009512:3(95)Online publication date: 26-Feb-2023
    • (2023)Biometric Analysis in Design Cognition Studies: A Systematic Literature ReviewBuildings10.3390/buildings1303063013:3(630)Online publication date: 27-Feb-2023
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media