skip to main content
10.1145/1109540.1109542acmotherconferencesArticle/Chapter ViewAbstractPublication PagesidcConference Proceedingsconference-collections
Article

Comparison of think-aloud and constructive interaction in usability testing with children

Published: 08 June 2005 Publication History

Abstract

Constructive interaction provides natural thinking-aloud as test subjects collaborate to solve tasks. Since children may face difficulties in following instructions for a standard think-aloud test, constructive interaction has been suggested as evaluation method when conducting usability testing with children. However, the relationship between think-aloud and constructive interaction is still poorly understood.We present an experiment that compares think-aloud and constructive interaction in usability testing. The experiment involves 60 children with three setups where children apply think-aloud, and constructive interaction in acquainted and non-acquainted pairs. Our results showed only minor significant differences between the setups, but the pairing of the children had impact on identification of usability problems as acquainted dyads identified more problems both in total and of the most severe than non-acquainted dyads and individual testers. Finally, the acquainted pairs reported that they had to put less effort into the testing than the think-aloud and non-acquainted children.

References

[1]
Benford, S., Bederson, B. B., Åkesson, K-P, Bayon, V., Druin, A., Hansson, P., Hourcade, J. P., Ingram, R., Neale, H., O'Malley, C., Simsarian, K. T., Stanton, D., Sundblad, Y., and Taxén, G. (2000) Designing storytelling technologies to encouraging collaboration between young children. In Proceedings of the Human Factors and Computing Systems CHI'OO, ACM Press, pp. 556--563
[2]
Bers, M. U., Ackermann, E., Cassell, J., Donegan, B., Gonzalez-Heydrich, J., DeMaso, D. R., Strohecker, C., Lualdi, S., Bromley, D., and Karlin, J. (1998) Interactive Storytelling Environments: Coping with Cardiac Illness at Boston's Children's Hospital. In Proceedings of the Human Factors and Computing Systems CHI'98, ACM Press, pp. 603--610
[3]
Bers, M. U., Gonzalez-Heydrich, J., and DeMaso, D. R. (2001) Identity Construction Environments: Supporting a Virtual Therapeutic Community of Pediatric Patients Undergoing Dialysis. In Proceedings of the Human Factors and Computing Systems CHI'01, ACM Press, pp. 380--387
[4]
Cassell, J. (2002) Genderizing. The Handbook of Human-Computer Interaction
[5]
Cassell, J. and Ryokai, K. (2001) Making Space for Voice: Technologies for Supporting Children's Fantasy and Storytelling. In Personal and Ubiquitous Computing, Springer-Verlag, vol.5(3), pp. 169--190
[6]
Danesh, A., Inkpen, K. M., Lau, F., Shu, K., Booth, K. S. (2001) Geney: Designing a collaborative activity for the Palm handheld computer. In Proceedings of the Human Factors and Computing Systems CHI'01, ACM Press, pp. 388--395
[7]
Donker, A. and Reitsma, P. (2004) Usability testing with young children. In Proceedings of the 4th International Conference on Interaction Design and Children (IDC), ACM Press, pp. 43--48
[8]
Druin, A. and Solomon, C. (1996) Designing Multimedia Environments for Children. Wiley & Sons, New York
[9]
Druin, A. (1999) The Role of Children in the Design of New Technology. HCIL Technical Report No. 99--23, University of Maryland, USA
[10]
Druin, A. (1999) The Design of Children's Technology. Morgan Kaufmann Publishers, Inc., San Francisco, CA
[11]
Ellis, J. B. and Bruckman, A. S. (2001) Designing Palaver Tree Online: Supporting Social Roles in a Community of Oral History. In Proceedings of the Human Factors and Computing Systems CHI'01, ACM Press, pp. 474--481
[12]
Gray, W. D. and Salzman, M. C. (1998) Damaged Merchandise? A Review of Experiments That Compare Usability Evaluation Methods. In Human-Computer Interaction, vol. 13, pp. 203--261
[13]
Gorriz, C. M. and Medina, C. (2000) Engaging Girls with Computers through Software Games. Communications of the ACM, vol. 43, No. 1, pp. 42 -- 49
[14]
Hanna, L., Risden, K., and Alexander, K. J. (1997) Guidelines for Usability Testing with Children. In interactions, September + October, pp. 9 -- 14
[15]
Höök, K., Bullock, A., Paiva, A., Vala, M., Chaves, R., and Prada, R. (2003) FantasyA and SenToy. In Proceedings of the Human Factors and Computing Systems CHI'03, ACM Press, pp. 474 -- 481
[16]
Inkpen, K. (1997) Three Important Research Agendas for Educational Multimedia: Learning, Children, and Gender. In Proceedings of Educational MultiMedia '97
[17]
Jeffries, R., Miller, J. R., Wharton, C., and Uyeda, K. M. User Interface Evaluation in the Real World: A Comparison of Four Techniques. In Proceedings of the Human Factors and Computing Systems CHI'91, ACM Press, 119--124
[18]
Kahler, H. (2000) Constructive Interaction and Collaborative Work. interactions, May + June, pp. 27 -- 34
[19]
Karat, C.-M., Campbell, R., and Fiegel, T. (1992) Comparison of Empirical Testing and Walkthrough Methods in User Interface Evaluation. In Proceedings of the Human Factors and Computing Systems CHI'92, ACM Press, 397--404
[20]
Kennedy, S. (1989) Using Video in the BNR Usability Lab. ACM SIGCHI Bulletin, vol. 21(2), pp. 92 -- 95
[21]
van Kesteren, I. E. H., Bekker, M. M., Vermeeren, A. P. O. S., and Lloyd, P. A. (2003) Assessing usability evaluation methods on their effectiveness to elicit verbal comments from children subjects. In Proceeding of the 2003 conference on Interaction design and children (IDC'03), ACM Press, pp. 41 -- 49
[22]
Lester, J. C., Converse, S. A., Kahler, S. E., Barlow, S. T., Stone, B. A., and Bhogal, R. S. (1997) The Persona Effect: Affective Impact of Animated Pedagogical Agents. In Proceedings of the Human Factors and Computing Systems CHI'97, ACM Press, pp. 359 -- 366
[23]
Lumbreras, M. and Sánchez, J. (1999) Interactive 3D Sound Hyperstories for Blind Children. In Proceedings of the Human Factors and Computing Systems CHI'99, ACM Press, pp. 318 -- 325
[24]
Markopoulos, P. and Bekker, M. (2003) On the Assessment pf Usability Testing Methods for Children. Interacting with Computers, Elsevier, Vol. 15, pp. 227 -- 243
[25]
Miller R. C. and Hart, S. G. (1984) Assessing the Subjective Workload of Directional Orientation Tasks. In Proceedings of 20th Annual Conference on Manual Control, NASA Conference Publication, pp. 85 -- 95
[26]
Miyake, N. (1986) Constructive Interaction and the Iterative Process of Understanding. Cognitive Science, vol. 10(2), pp. 151 -- 177
[27]
Moher, T., Johnson, A., Ohlsson, S., and Gillingham, M. (1999) Bridging Strategies for VR-based Learning. In Proceedings of the Human Factors and Computing Systems CHI'99, ACM, pp. 536 -- 543
[28]
Molich, R. (1997) Usable IT Systems (In Danish). Teknisk Forlag, Copenhagen
[29]
Montemayor, J., Druin, A., Farber, A., Simms, S., Churaman, W., and D'Amour, A. (2002) Physical Programming: Designing Tools for Children to Create Physical Interactive Environments. In Proceedings of the Human Factors and Computing Systems CHI'02, ACM Press, pp. 299 -- 306
[30]
Nielsen, J. (1993) Usability Engineering. Academic Press
[31]
Nielsen, J. and Landauer, T. K. (1993) A Mathematical Model of the Finding of Usability Problems. In Proceedings of the Human Factors and Computing Systems INTERCHI'93, ACM Press, pp. 206 -- 213
[32]
O'Malley, C. E., Draper, S. W., and Riley, M. S. (1984) Constructive Interaction: A Method for Studying Human-Computer-Human Interaction. In Proceedings of IFIP Interact '84, pp. 269 -- 274
[33]
Rader, C., Brand, C., and Lewis, C. (1997) Degrees of Comprehension: Children's Understanding of a Visual Programming Environment. In Proceedings of the Human Factors and Computing Systems CHI'97, ACM Press, pp. 351 -- 358
[34]
Resnick, M., Martin, F., Berg, R., Borovoy, R., Colella, V., Kramer, K., and Silverman, B. (1998) Digital Manipulatives: New Toys to Think With. In Proceedings of the Human Factors and Computing Systems CHI'98, ACM, pp. 281 -- 287
[35]
Rubin, J. (1994) Handbook on Usability Testing -- How to Plan, Design, and Conduct Effective Tests. John Wiley & Son, Inc.
[36]
Scaife, M., Rogers, Y., Aldrich, F., and Davies, M. (1997) Designing for or Designing with? Informant Design for Interactive Learning Environments. In Proceedings of the Human Factors and Computing Systems CHI'97, ACM Press, pp. 343 -- 350
[37]
Stewart, J., Bederson, B. B., and Druin, A. (1999) Single Display Groupware: A Model for Co-Present Collaboration. In Proceedings of the Human Factors and Computing Systems CHI'99, ACM, pp. 286 -- 293
[38]
Strommen, E. (1998) When the Interface is a Talking Dinosaur: Learning across Media with ActiMates Barney. In Proceedings of the Human Factors and Computing Systems CHI'98, ACM Press, pp. 288 -- 295

Cited By

View all
  • (2024)Testing the Quality of the Mobile Application Interface Using Various Methods—A Case Study of the T1DCoach ApplicationApplied Sciences10.3390/app1415658314:15(6583)Online publication date: 27-Jul-2024
  • (2024)Using Psychophysiological Data to Facilitate Reflective Conversations with Children about their Player ExperiencesProceedings of the ACM on Human-Computer Interaction10.1145/36771128:CHI PLAY(1-33)Online publication date: 15-Oct-2024
  • (2024)Reflections on Data Collection during Toy Prototype Development in a Design Studio CourseProceedings of the 23rd Annual ACM Interaction Design and Children Conference10.1145/3628516.3659422(940-944)Online publication date: 17-Jun-2024
  • Show More Cited By

Index Terms

  1. Comparison of think-aloud and constructive interaction in usability testing with children

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    IDC '05: Proceedings of the 2005 conference on Interaction design and children
    June 2005
    128 pages
    ISBN:1595930965
    DOI:10.1145/1109540
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 08 June 2005

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. children
    2. constructive interaction
    3. subjective workload
    4. think-aloud
    5. usability testing and evaluation

    Qualifiers

    • Article

    Conference

    IDC05
    IDC05: Interaction Design and Children
    June 8 - 10, 2005
    Colorado, Boulder

    Acceptance Rates

    Overall Acceptance Rate 172 of 578 submissions, 30%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)67
    • Downloads (Last 6 weeks)6
    Reflects downloads up to 09 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Testing the Quality of the Mobile Application Interface Using Various Methods—A Case Study of the T1DCoach ApplicationApplied Sciences10.3390/app1415658314:15(6583)Online publication date: 27-Jul-2024
    • (2024)Using Psychophysiological Data to Facilitate Reflective Conversations with Children about their Player ExperiencesProceedings of the ACM on Human-Computer Interaction10.1145/36771128:CHI PLAY(1-33)Online publication date: 15-Oct-2024
    • (2024)Reflections on Data Collection during Toy Prototype Development in a Design Studio CourseProceedings of the 23rd Annual ACM Interaction Design and Children Conference10.1145/3628516.3659422(940-944)Online publication date: 17-Jun-2024
    • (2023)Emma stop that, it's my turn now - Comparing Peer Tutoring and Thinking Aloud for Usability-Testing with Children in a school settingProceedings of Mensch und Computer 202310.1145/3603555.3608542(442-447)Online publication date: 3-Sep-2023
    • (2023)How do Children Rate Their Health? An Investigation of Considered Health Dimensions, Health Factors, and Assessment StrategiesChild Indicators Research10.1007/s12187-023-10066-616:6(2545-2580)Online publication date: 13-Sep-2023
    • (2022)User experiences of structured stakeholder engagement to consider transferability: The TRANSFER approachCampbell Systematic Reviews10.1002/cl2.128418:4Online publication date: 17-Oct-2022
    • (2021)The Dutch CoronaMelder: Do users see added value? (Preprint)JMIR Formative Research10.2196/27882Online publication date: 11-Feb-2021
    • (2021)Field Playtesting with Experts' Constructive Interaction: An Evaluation Method for Mobile Games for Cultural Heritage2021 IEEE Conference on Games (CoG)10.1109/CoG52621.2021.9619049(1-9)Online publication date: 17-Aug-2021
    • (2021)Informing the design of a multisensory learning environment for elementary mathematics learningJournal on Multimodal User Interfaces10.1007/s12193-021-00382-y16:2(155-171)Online publication date: 11-Oct-2021
    • (2021)Learning About the Heritage of Tinian Marble Crafts with a Location-Based Mobile Game and Tour AppDigital Heritage. Progress in Cultural Heritage: Documentation, Preservation, and Protection10.1007/978-3-030-73043-7_31(377-388)Online publication date: 14-Apr-2021
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media