Layered Evaluation of a Personalized Interaction Approach

ABSTRACT

Evaluation of personalized systems is a complicated endeavor. First, evaluation goals, methods and criteria are manifold and have to be carefully selected to fit the actual application scenario and the scope of the evaluated system. Second, it is considerably harder to locate the source of problems, compared to non-adaptive systems where problems most often reside on the UI level. Thus, in the past, a layered evaluation approach for personalized systems has been proposed that distinguishes between five layers that can theoretically all be the source of problems (e.g., collection of input data or adaptation decision). This paper outlines a use case related to personalized interaction comprising i) modeling a user's interaction abilities, ii) recommending interaction methods and devices that fit the user's individual needs, and iii) personalized system behavior and reaction to user input. Next, the paper describes experiences with an evaluation process using the layered evaluation framework.

References

1. Mirjam Augstein, Daniel Kern, Thomas Neumayr, Werner Kurschl, and Josef Altmann 2015. Measuring Physical Pressure in Smart Phone Interaction for People with Impairments Mensch und Computer 2015, Workshopband. Oldenbourg Wissenschaftsverlag, Stuttgart, Germany, 207--214.
2. Mirjam Augstein, Thomas Neumayr, Daniel Kern, Werner Kurschl, Josef Altmann, and Thomas Burger. 2017. An Analysis and Modeling Framework for Personalized Interaction Proceedings of the 22nd International Conference on Intelligent User Interfaces. Limassol, Cyprus.
3. Mirjam Augstein, Thomas Neumayr, Werner Kurschl, Daniel Kern, Thomas Burger, and Josef Altmann. 2017. A Personalized Interaction Approach -- Motivation and Use Case. (2017). shownoteAccepted at HAAPIE17, to be held in conjunction with the 25th Intl. Conf. on User Modeling, Adaptation and Personalization.
4. Pradipta Biswas and Patrick Robinson, Peter ad Langdon. 2012. Designing Inclusive Interfaces Through User Modeling and Simulation. International Journal of Human-Computer Interaction Vol. 28 (2012), 1--33.
5. Krysztof Gajos, Daniel S. Weld, and Jacob O. Wobbrock. 2010. Automatically generating personalized user interfaces with SUPPLE. Artificial Intelligence Vol. 174, 12--13 (2010), 910--950.
6. Krzysztof Z. Gajos, Jacob O. Wobbrock, and Daniel S. Weld. 2007. Automatically generating user interfaces adapted to users' motor and vision capabilities Proc. UIST 2007. ACM Press, 231--240.
7. Charalampos Karagiannidis and Demetrios G. Sampson 2000. Layered Evaluation of Adaptive Applications and Services. Springer Berlin Heidelberg, 343--346.
8. Bart P. Knijnenburg, Martijn C. Willemsen, Zeno Gantner, Hakan Soncu, and Chris Newell. 2012. Explaining the user experience of recommender systems. User Modeling and User-Adapted Interaction Vol. 22, 4--5 (October 2012), 441--504.
9. Alexandros Paramythis, Stephan Weibelzahl, and Judith Masthoff 2010. Layered Evaluation of Interactive Adaptive Systems: Framework and Formative Methods. User Modeling and User-Adapted Interaction Vol. 20, 5 (2010), 383--453.
10. M Rabbi, A Pfammatter, M Zhang, B Spring, and T Choudhury 2015. Automated Personalized Feedback for Physical Activity and Dietary Behavior Change With Mobile Phones: A Randomized Controlled Trial on Adults. JMIR Mhealth Uhealth, Vol. 3, 2 (May 2015).
11. Constantine Stephanidis, Alexandros Paramythis, Michael Sfyrakis, A Stergiou, Napoleon Maou, A Leventis, George Paparoulis, and Charalampos Karagiannidis. 1998. Adaptable and adaptive user interfaces for disabled users in the AVANTI project Proc. of the Intl. Conf. on Intelligence in Services and Networks. 153--166.
12. Stephan Weibelzahl. 2003. Evaluation of Adaptive Systems. Ph.D. Dissertation. University of Education Freiburg.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI design and evaluation methods

      • User models

    • Interaction paradigms