Empirical Evaluation Methods for Pattern Languages: Sketches, Classification, and Network Analysis

ABSTRACT

After elaborating design patterns in a certain field or domain building a pattern language will be the next step. This can be a challenging task, because a large amount of text with numerous cross-references and dependencies needs to become inherently consistent. In order to improve the quality of a pattern language it is necessary to analyse and evaluate all individual patterns as well as the pattern language as a whole. This paper provides an overview of existing evaluation methods for pattern languages and contributes further empirical methods that aim to assess and analyse pattern languages from different perspectives. Methods for sketching patterns, pattern classification, and the network analysis of pattern interrelation will be introduced and explained. The three methods are applied and tested in a case study about a pattern language for video learning environments.

References

1. Alexander, C. 1979. The Timeless Way of Building. Oxford University Press, New York.
2. Alexander, C., Ishikawa, S., and Silverstein, M. 1977. A Pattern Language. Oxford University Press, New York.
3. Baxley, B. 2003. Universal Model of a User Interface. In Proceedings of the 2003 Conference on Designing for User Experiences.
4. DUX '03. ACM, New York, NY, USA, 1--14. Boroditsky, L. and Gaby, A. 2010. Remembrances of Times East. Psychological Science 21, 11, 1635--1639.
5. Chatzigeorgiou, A., Tsantalis, N., and Deligiannis, I. 2008. An empirical study on students' ability to comprehend design patterns. Computers & Education 51, 3, 1007--1016.
6. Coplien, J. O. 1997. A pattern language for writers' workshops. In Proceedings of the 2nd European Conference on Pattern Languages of Programming (EuroPLoP '97). Siemens Technical Report 120/SW1/FB. Siemens, Munich, Germany, 51--60.
7. Crumlish, C. and Malone, E. 2009. Designing Social Interfaces. O'Reilly Media, Sebastopol.
8. Cutumisu, M., Onuczko, C., Szafron, D., Schaeffer, J., McNaughton, M., Roy, T., Siegel, J., and Carbonaro, M. 2006. Evaluating Pattern Catalogs: The Computer Games Experience. In Proceedings of the 28th International Conference on Software Engineering. ICSE '06. ACM, New York, NY, USA, 132--141.
9. Cybulski, J. L. and Linden, T. 1998. Composing Multimedia Artefacts for Reuse. In Proceedings of Pattern Languages of Programs'98. 15.
10. Döring, N. and Bortz, J. 2006. Forschungsmethoden und Evaluation in den Sozial- und Humanwissenschaften 5 Ed. Springer-Verlag, Berlin Heidelberg.
11. Euler, D. 2014. Design Research -- a paradigm under development. In Design-Based Research, D. Euler and P. Sloane, Eds. Zeitschrift für Berufs- und Wirtschaftspädagogik Series, vol. 1. Franz Steiner, Stuttgart, 15--44.
12. Fincher, S. 2003. Perspectives on HCI patterns: concepts and tools (introducing PLML). Interfaces 56, 26--28.
13. Gamma, E., Helm, R., Johnson, R., and Vlissides, J. 1995. Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley.
14. Greenberg, S., Carpendale, S., Marquardt, N., and Buxton, B. 2012. Sketching user experiences. Elsevier Ltd, Waltham.
15. Guerra, E. and Fernandes, C. 2010. An Evaluation Process for Pattern Languages. In Proceedings of the 8th Latin American Conference on Pattern Languages of Programs. SugarLoafPLoP '10. ACM, New York, NY, USA, 18:1--18:11.
16. Harrison, N. 1999. The Language of Shepherds: A Pattern Language for Shepherding. In Proceedings of the 6th Annual Conference on the Pattern Languages of Programs (PLoP). Monticello, Illinois, USA.
17. Hevner, A. R., March, S. T., Park, J., and Ram, S. 2004. Design Science in Information Systems Research. MIS Q. 28, 1, 75--105.
18. Hübscher, C., Pauwels, S. L., Roth, S. P., Bargas-Avila, J. A., and Opwis, K. 2011. The Organization of Interaction Design Pattern Languages Alongside the Design Process. Interact. Comput. 23, 3, 189--201.
19. Iba, T. and Isaku, T. 2012. Holistic Pattern-Mining Patterns. In Proceedings of the 17th European Conference on Pattern Languages of Programs. Infragistics. 2013. Quince UX Paterns. http://quince.infragistics.com/html/.
20. Janeiro, J., Barbosa, S. D. J., Springer, T., and Schill, A. 2010. Improving the Search for User Interface Design Patterns through Typed Relationships. Springer Berlin Heidelberg, Berlin, Heidelberg, 3--14.
21. Kodituwakku, S. R. and Bertok, P. 2003. Pattern categories: a mathematical approach for organizing design patterns. In Proceedings of the 2002 Conference on Pattern Languages of Programs - Volume 13. CRPIT '02. Australian Computer Society, Inc., Darlinghurst, Australia, Australia, 63--73.
22. Kruschitz, C. and Hitz, M. 2010. Are Human-computer Interaction Design Patterns Really Used? In Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries. NordiCHI '10. ACM, New York, NY, USA, 711--714.
23. Kubo, A., Washizaki, H., Takasu, A., and Fukazawa, Y. 2005. Extracting relations among embedded software design patterns. Journal of Integrated Design and Process Science 9, 3, 39--52.
24. Kunert, T. 2009. User-Centered Interaction Design Patterns for Interactive Digital Televison Applications. Springer, Dordrecht / Heidelberg / London / New York.
25. Kunzmann, C., Schmidt, A., and Pirker, J. 2016. Pattern-oriented Approaches for Design-based Research in Collaborative Research Projects: A Knowledge Maturing Perspective. In Proceedings of the 21st European Conference on Pattern Languages of Programs. EuroPlop '16. ACM, New York, NY, USA, 19:1--19:8.
26. Leidel, M., Niemann, S., Saliha, H. A., Cullmann, J., Seidel, N., Borchardt, D., Bernhofer, C., and Krebs, P. 2012. International viewpoint and news -- IWRM education: e-learning module on Integrated Water Resources Management. Environmental Earth Sciences 68, 2, 609--613.
27. Lopes, R. and Carriico, L. 2007. Patterns for Time-based Hypermedia Artifacts. In Proceedings of the 11th European Conference on Pattern Languages of Programs, U. Zdun and L. B. Hvatum, Eds. Universitätsverlag Konstanz, Konstanz, 247--276.
28. Neil, T. 2012. Mobile Design Pattern Gallery: UI Patterns for Smartphone Apps. O'Reilly, Sebastopol.
29. Petter, S., Khazanchi, D., and Murphy, J. D. 2010. A Design Science Based Evaluation Framework for Patterns. SIGMIS Database 41, 3, 9--26.
30. Retalis, S., Georgia, P., and Dimitriadis, Y. 2006. Eliciting Design Patterns for E-Learning Systems. Computer Science Education 16, 2, 105--118.
31. Rosenberg, D. and Grafton, A. 2010. Cartographies of Time. Princeton Architectural Press.
32. Schümmer, T. and Lukosch, S. 2007. Patterns for computer-mediated interaction. Wiley, Hoboken, NJ.
33. Seidel, N. 2012. Collaborative hypervideo editing using MediaWiki. In Proceedings of the Eighth Annual International Symposium on Wikis and Open Collaboration. ACM, Linz, 22:1--22:4.
34. Seidel, N. 2013. Peer Assessment und Peer Annotation mit Hilfe eines videobasierten CSCL-Scripts. In DeLFI 2013 -- Die 11. e-Learning Fachtagung Informatik der Gesellschaft für Informatik, A. Breiter and C. Rensing, Eds. Gesellschaft für Informatik, Bonn, 83--94.
35. Seidel, N. 2014a. Interaction design patterns for design and development of video learning environments. In Proceedings of the 19th European Conference on Pattern Languages of Programs. ACM, New York, NY, USA, 20:1--20:12.
36. Seidel, N. 2014b. Interaction design patterns for interactive video players in video-based learning environments. In Proceedings of the 19th European Conference on Pattern Languages of Programs. ACM, New York, NY, USA, 19:1--19:14.
37. Seidel, N. 2015a. Interaction design patterns for spatio-temporal annotations in video learning environments. In Proceedings of the 20th European Conference on Pattern Languages of Programs. ACM, New York, NY, USA, 16:1--16:21.
38. Seidel, N. 2015b. Making web video accessible -- interaction design patterns for assistive video learning environments. In Proceedings of the 20th European Conference on Pattern Languages of Programs. ACM, New York, NY, USA, 17:1--17:16.
39. Seidel, N. 2017. Dissertation. Ph.D. thesis, Technische Universität Dresden, Dresden.
40. Tidwell, J. 2005. Designing Interfaces -- Patterns for Effective Interaction Design 1 Ed. O'Reilly Media, Sebastopol.
41. Tidwell, J. 2011. Designing Interfaces -- Patterns for Effective Interaction Design 2 Ed. O'Reilly Media, Sebastopol.
42. van Duyne, D. K., Landay, J. A., and Hong, J. I. 2006. The Design of Sites: Patterns for Creating Winning Web Sites 2 Ed. Prentice Hall.
43. van Welie, M. and Veer, G. C. v. D. 2003. Pattern Languages in Interaction Design: Structure and Organization. In Proceedings of Interact '03, 1-5 September, M. Rauterberg, Ed. IOS Press, Amsterdam, 527--534.
44. Vlissides, J. 1996. Seven Habits of Successful Pattern Writers. C++ Report 8, 10.
45. Washizaki, H., Kadoya, M., Fukazawa, Y., and Kawamura, T. 2014. Network Analysis for Software Patterns Including Organizational Patterns in Portland Pattern Repository. In Agile Conference (AGILE), 2014. 66--67.
46. Wellhausen, T. and Fiesser, A. 2012. How to Write a Pattern?: A Rough Guide for First-time Pattern Authors. In Proceedings of the 16th European Conference on Pattern Languages of Programs. EuroPLoP '11. ACM, New York, NY, USA, 5:1--5:9.
47. Yahoo! 2012. ypatterns.
48. Zimmer, W. 1995. Pattern Languages of Program Design. ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, Chapter Relationsh, 345--364.

Index Terms

• Applied computing

  • Education

    • Interactive learning environments