Assessing the applicability of the structured expert evaluation method (SEEM) for a wider age group

ABSTRACT

This paper describes a study which examines whether a predictive evaluation method (SEEM) is suitable to assess products for an older age group than for which the evaluation method was originally developed. SEEM stands for Structured Expert Evaluation Method and is an analytical evaluation method especially developed for assessing the fun and usability of young children's educational computer games (children from 5 to 7 years old). In the present study SEEM was applied to assess educational computer games for children between 9 and 11 years old. Outcomes on scores for thoroughness (whether SEEM finds all problems), validity (whether SEEM makes predictions that are likely to be true) and appropriateness (whether SEEM is applied correctly) were compared. The results show that the trends for the thoroughness and the validity are the same for the two different age groups; however SEEM scores a bit better for the oldest age group. The appropriateness scores are about the same for the two age groups. The results indicate that SEEM can also be applied for assessing educational computer games for children between 9 and 11 years old.

REFERENCES

• 1: Bio Mania - Een speelse toepassing op biologie (Bio Mania - a playful application to biology) {Computer software} Edison (1998)
• 2: Max en de toverstenen (Milo and the magical stones) {Computer software} MediaMix Benelux (2002)
• 3: Regenboog, de mooiste vis van de zee (Rainbow, the most beautiful fish in the ocean) {Computer software} MediaMix Benelux (2002)
• 4: Baauw, E. and Bekker, M. M.: A comparison of two analytical evaluation methods for children's computer games. Position paper for workshop: Child Computer Interaction: Methodological research, Interact 2005, 12 September 2005 Rome, Italy (2005)
• 5: Baauw, E., Bekker, M. M., and Barendregt, W.: A Structured Expert Evaluation Method for the Evaluation of Children's Computer Games. Proceedings of Human-Computer Interaction INTERACT 2005, 14 September 2005 Rome, Springer Verlag (2005)
• 6: Ester Baauw , Panos Markopoulous, A comparison of think-aloud and post-task interview for usability testing with children, Proceeding of the 2004 conference on Interaction design and children: building a community, p.115-116, June 01-03, 2004, Maryland [doi>10.1145/1017833.1017848]
• 7: Barendregt, W., Bekker, M. M., Bouwhuis, D., and Baauw, E.: Predicting effectiveness of children participants in user testing based on personality characteristics. Behaviour & Information Technology (in press)
• 8: Barendregt, W., Bekker, M. M., and Speerstra, M.: Empirical evaluation of usability and fun in computer games for children. Proceedings of Human-Computer Interaction INTERACT-03', 9 March 2003 Zürich, Switzerland, IOP Press (2003) 705--708
• 9: Chattratichart, J. and Brodie, J.: Applying User Testing Data to UEM Performance Metrics. Late Breaking Results Paper, 24 April 2004 Vienna, Austria (2004) 1119--1122
• 10: Cockton, G. and Woolrych, A.: Understanding Inspection Methods: Lessons from an Assessment of Heuristic Evaluation. In: Blandford, A., Vanderdonckt, J., and Gray, P. D. (Eds.): Springer-Verlag (2001) 171--192
• 11: Hartson, H. R., Andre, T. S., and Williges, R. C.: Criteria for evaluating usability evaluation methods. International Journal of Human-Computer Interaction: Special issue on Empirical Evaluation of Information Visualisations, 13(4) (2001) 373--410
• 12: Inkpen, K.: Three Important Research Agendas for Educational Multimedia: Learning, Children, and Gender. Proceedings of Educational MultiMedia '97, June 1997, Calgary (1997) 521--526
• 13: Lavery, D., Cockton, G., and Atkinson, M. P.: Comparison of Evaluation Methods Using Structured Usability Problem Reports. Behaviour and Information Technology, 16(4) (1997) 246--266
• 14: Malone, T. W.: What makes things fun to learn? A study of intrinsically motivating computer games Technical Report CIS-7, Xerox PARC, Palo Alto (1980)
• 15: Malone, T. W. and Lepper, M. R.: Making learning fun: a taxonomy of intrinsic motivations for learning. In: Snow, R. E. and Farr, M. J. (Eds.): Aptitude, Learning and Interaction III Cognitive and Affective Process Analysis. Hillsdale, N.J., Erlbaum (1987)
• 16: Jakob Nielsen, Heuristic evaluation, Usability inspection methods, John Wiley & Sons, Inc., New York, NY, 1994
• 17: Norman, D. A.: The design of everyday things. London, MIT Press (1998)
• 18: Piaget, J.: Science of Education and the Psychology of the Child. New York, Orion Press (1970)
• 19: Sears, A.: Heuristic Walkthroughs: Finding the problems without the noise. International Journal of Human-Computer Interaction, 9(3) (1997) 213--234

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Evaluation/methodology

Keywords:
SEEM, analytical evaluation methods, children, educational computer games, fun, usability