Comparison of playtesting and expert review methods in mobile game evaluation

ABSTRACT

Selecting an evaluation method for product evaluations depends on many issues, such as the development stage of the product, time schedule, resources, and money that can be invested on the evaluation. The user testing and expert review methods are probably the most common ones when productivity software is being evaluated. Conducting a playtesting with players is commonly used by game designers, but the expert review method has not received that much attention, although it has proven to be an efficient and useful method. In this paper, we present a comparison study of the playtesting and expert review methods in mobile game evaluation. Our objective is to compare the effectiveness of the expert review method with playtesting. Results indicate that the expert review method is able to identify playability problems as accurately as playtesting, but in addition, it identifies problems that are crucial for the playability of the game.

References

1. Adams, E., Rollings, A. Game Design and Development: Fundamentals of Game Design. Prentice Hall, 2007
2. Baauw, E., Bekker, M. M., Barendregt, W. A Structured Expert Evaluation Method for the Evaluation of Children's Computer Games. In proc INTERACT, (2005), 457--469
3. Desurvire, H., Caplan, M., Toth, J. A. Using heuristics to evaluate the playability of games. In proc CHI Extended Abstracts, ACM Press (2004), 1509--1512
4. Drachen, A., Canossa, A. Towards gameplay analysis via gameplay metrics. In proc MindTrek, ACM (2009), 202--209
5. Dumas, J. S., Loring, B. Moderating Usability Tests: Principles & Practices for Interacting. Morgan-Kaufmann, 2008
6. Federoff, M. A. Heuristics and Usability Guidelines for the Creation and Evaluation of Fun in Video Games. Indiana University, 2002
7. Fullerton, T., Swain, C., Hoffman, S. Game Design Workshop: Designing, Prototyping, and Playtesting Games. CMP Books, 2004
8. Gray, W. D., Salzman, M. C. Damaged Merchandise? A Review of Experiments That Compare Usability Evaluation Methods. 13, 3 (1998), 203--261
9. Kim, J. H., Gunn, D. V., Schuh, E., Phillips, B., Pagulayan, R. J., Wixon, D. Tracking real-time user experience (TRUE): a comprehensive instrumentation solution for complex systems. In proc CHI, ACM (2008), 443--452
10. Korhonen, H., Koivisto, E. M. I. Playability Heuristics for Mobile Game. In proc MobileHCI, ACM Press (2006), 9--16
11. Korhonen, H., Paavilainen, J., Saarenpää, H. Expert Review Method in Game Evaluations -- Comparison of Two Playability Heuristic Sets. In proc MindTrek, ACM Press (2009), 74--81
12. Laitinen, S. Do usability expert evaluation and test provide novel and useful data for game development? JUS 1, 2 (2006), 64--75
13. Molich, R., Dumas, J. S. Comparative Usability Evaluation (CUE-4). BIT 27, 3 (2008), 263--281
14. Molich, R., Kaasgaard, K., Karyukin, B. Comparative Usability Evaluation. BIT 23, 1 (2004), 65--74
15. Mulligan, J., Patrovsky, B. Developing Online Games: An Insider's Guide. New Riders, 2003
16. Nielsen, J. Finding usability problems through heuristic evaluation. In proc CHI, ACM Press (1992), 373--380
17. Nielsen, J. Heuristic Evaluation. in Usability Inspection Methods, Nielsen, J., Mack, R. (Eds) Wiley & Sons (1994), 25--62
18. Nielsen, J. Usability Engineering. Academic Press, 1994
19. Nielsen, J. Usability inspection methods. In proc CHI Extended Abstracts, ACM (1994).
20. Nielsen, J., Molich, R. Heuristic evaluation of user interfaces. In proc CHI, ACM (1990), 249--256
21. Nielsen, J., Phillips, V. L. Estimating the relative usability of two interfaces: heuristic, formal, and empirical methods compared. In proc CHI, ACM (1993), 214--221
22. Pagulayan, R. J., Keeker, K., Wixon, D., Romero, R. L., Fuller, T. User-centered Design in Games. in Handbook for Human-Computer Interaction in Interactive Systems., Jacko, J., Sears, A. (Eds) Mahwah, NJ: Lawrence Erlbaum Associates, Inc. (2003), 883--906
23. Pinelle, S., Wong, N., Stach, T. Heuristic evaluation for games: usability principles for video game design. In proc CHI, ACM Press (2008), 1453--1462
24. Rouse, R. Game Design: Theory and Practice. Wordware Publishing, 2001
25. Rubin, J. Handbook of Usability Testing: How to plan, design and conduct effective tests. John Wiley & Sons, 1994
26. Schell, J. The Art of Game Design. Morgan Kaufmann, 2008
27. Tan, W.-s., Liu, D., Bishu, R. Web evaluation: Heuristic evaluation vs. User Testing. 39, 4 (2009), 621--627

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI design and evaluation methods