skip to main content
10.1145/1139073.1139095acmotherconferencesArticle/Chapter ViewAbstractPublication PagesidcConference Proceedingsconference-collections
Article

Assessing the applicability of the structured expert evaluation method (SEEM) for a wider age group

Published: 07 June 2006 Publication History

Abstract

This paper describes a study which examines whether a predictive evaluation method (SEEM) is suitable to assess products for an older age group than for which the evaluation method was originally developed. SEEM stands for Structured Expert Evaluation Method and is an analytical evaluation method especially developed for assessing the fun and usability of young children's educational computer games (children from 5 to 7 years old). In the present study SEEM was applied to assess educational computer games for children between 9 and 11 years old. Outcomes on scores for thoroughness (whether SEEM finds all problems), validity (whether SEEM makes predictions that are likely to be true) and appropriateness (whether SEEM is applied correctly) were compared. The results show that the trends for the thoroughness and the validity are the same for the two different age groups; however SEEM scores a bit better for the oldest age group. The appropriateness scores are about the same for the two age groups. The results indicate that SEEM can also be applied for assessing educational computer games for children between 9 and 11 years old.

References

[1]
Bio Mania - Een speelse toepassing op biologie (Bio Mania - a playful application to biology) {Computer software} Edison (1998)
[2]
Max en de toverstenen (Milo and the magical stones) {Computer software} MediaMix Benelux (2002)
[3]
Regenboog, de mooiste vis van de zee (Rainbow, the most beautiful fish in the ocean) {Computer software} MediaMix Benelux (2002)
[4]
Baauw, E. and Bekker, M. M.: A comparison of two analytical evaluation methods for children's computer games. Position paper for workshop: Child Computer Interaction: Methodological research, Interact 2005, 12 September 2005 Rome, Italy (2005)
[5]
Baauw, E., Bekker, M. M., and Barendregt, W.: A Structured Expert Evaluation Method for the Evaluation of Children's Computer Games. Proceedings of Human-Computer Interaction INTERACT 2005, 14 September 2005 Rome, Springer Verlag (2005)
[6]
Baauw, E. and Markopoulos, P.: A comparison of think aloud and post-task interview. 1 June 2004 Maryland, USA, ACM Press (2004) 115--117
[7]
Barendregt, W., Bekker, M. M., Bouwhuis, D., and Baauw, E.: Predicting effectiveness of children participants in user testing based on personality characteristics. Behaviour & Information Technology (in press)
[8]
Barendregt, W., Bekker, M. M., and Speerstra, M.: Empirical evaluation of usability and fun in computer games for children. Proceedings of Human-Computer Interaction INTERACT-03', 9 March 2003 Zürich, Switzerland, IOP Press (2003) 705--708
[9]
Chattratichart, J. and Brodie, J.: Applying User Testing Data to UEM Performance Metrics. Late Breaking Results Paper, 24 April 2004 Vienna, Austria (2004) 1119--1122
[10]
Cockton, G. and Woolrych, A.: Understanding Inspection Methods: Lessons from an Assessment of Heuristic Evaluation. In: Blandford, A., Vanderdonckt, J., and Gray, P. D. (Eds.): Springer-Verlag (2001) 171--192
[11]
Hartson, H. R., Andre, T. S., and Williges, R. C.: Criteria for evaluating usability evaluation methods. International Journal of Human-Computer Interaction: Special issue on Empirical Evaluation of Information Visualisations, 13(4) (2001) 373--410
[12]
Inkpen, K.: Three Important Research Agendas for Educational Multimedia: Learning, Children, and Gender. Proceedings of Educational MultiMedia '97, June 1997, Calgary (1997) 521--526
[13]
Lavery, D., Cockton, G., and Atkinson, M. P.: Comparison of Evaluation Methods Using Structured Usability Problem Reports. Behaviour and Information Technology, 16(4) (1997) 246--266
[14]
Malone, T. W.: What makes things fun to learn? A study of intrinsically motivating computer games Technical Report CIS-7, Xerox PARC, Palo Alto (1980)
[15]
Malone, T. W. and Lepper, M. R.: Making learning fun: a taxonomy of intrinsic motivations for learning. In: Snow, R. E. and Farr, M. J. (Eds.): Aptitude, Learning and Interaction III Cognitive and Affective Process Analysis. Hillsdale, N.J., Erlbaum (1987)
[16]
Nielsen, J.: Heuristic Evaluation. In: Nielsen, J. and Mack, R. L. (Eds.): Usability Inspection Methods. New York, USA, John Wiley & Sons, Inc. (1994) 25--62
[17]
Norman, D. A.: The design of everyday things. London, MIT Press (1998)
[18]
Piaget, J.: Science of Education and the Psychology of the Child. New York, Orion Press (1970)
[19]
Sears, A.: Heuristic Walkthroughs: Finding the problems without the noise. International Journal of Human-Computer Interaction, 9(3) (1997) 213--234

Cited By

View all
  1. Assessing the applicability of the structured expert evaluation method (SEEM) for a wider age group

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    IDC '06: Proceedings of the 2006 conference on Interaction design and children
    June 2006
    172 pages
    ISBN:9781450378086
    DOI:10.1145/1139073
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 07 June 2006

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. SEEM
    2. analytical evaluation methods
    3. children
    4. educational computer games
    5. fun
    6. usability

    Qualifiers

    • Article

    Conference

    IDC06

    Acceptance Rates

    Overall Acceptance Rate 172 of 578 submissions, 30%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)4
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 17 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2017)A method impact assessment framework for user experience evaluations with childrenProceedings of the 31st British Computer Society Human Computer Interaction Conference10.14236/ewic/HCI2017.82(1-9)Online publication date: 3-Jul-2017
    • (2017)Academic methods for usability evaluation of serious gamesMultimedia Tools and Applications10.1007/s11042-016-3845-976:4(5755-5784)Online publication date: 1-Feb-2017
    • (2015)From England to UgandaHuman-Computer Interaction10.1080/07370024.2014.98403430:3-4(263-293)Online publication date: 1-May-2015
    • (2014)Empowering Teenagers to Perform a Heuristic Evaluation of a GameProceedings of the 28th International BCS Human Computer Interaction Conference on HCI 2014 - Sand, Sea and Sky - Holiday HCI10.5555/2742941.2742993(353-358)Online publication date: 9-Sep-2014
    • (2012)Designing the anti-heuristic gameProceedings of the 11th International Conference on Interaction Design and Children10.1145/2307096.2307153(308-311)Online publication date: 12-Jun-2012
    • (2010)Work-domain knowledge in usability evaluationJournal of Systems and Software10.1016/j.jss.2010.02.02683:11(2019-2030)Online publication date: 1-Nov-2010
    • (2009)Evaluating a Tangible Game Video Console for KidsProceedings of the 12th IFIP TC 13 International Conference on Human-Computer Interaction: Part I10.1007/978-3-642-03655-2_17(141-144)Online publication date: 24-Aug-2009
    • (2008)A comparison of two analytical evaluation methods for educational computer games for young childrenCognition, Technology and Work10.1007/s10111-007-0068-x10:2(129-140)Online publication date: 26-Mar-2008
    • (2007)Design and evaluation of tangible interfaces for primary school childrenProceedings of the 6th international conference on Interaction design and children10.1145/1297277.1297331(209-212)Online publication date: 6-Jun-2007

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media