Measuring Engagement with Online Forms

ABSTRACT

Online form-filling and transactions are extremely common, both for industry and government; and it is important to provide a satisfying user experience during these tasks if customers or citizens are to continue using online channels. However, reliable measures of experience in these cases are limited. Other areas of information interaction, e.g., online search, news, and shopping, are increasingly exploring and attempting to measure the concept of user engagement (UE). In this study, we ask whether UE is an appropriate outcome for the utilitarian activities of online form-filling and transactions.

We describe work in progress which measures UE using the User Engagement Scale (UES) with utilitarian tasks, and which looks for behaviours which correlate with the UES. Early results suggest that, first, the UES can be adapted to such situations; and second, that readily observable user behaviours including time on site, mouse movements, and keypresses correlate with UES sub-scales and can, to some extent, predict users' responses.

References

1. E. Agichtein, E. Brill, and S. Dumais. Improving web search ranking by incorporating user behaviour information. In Proc. SIGIR, pages 19--26, August 2006.
2. H. Akaike. A new look at the statistical model identification. IEEE Trans. Automatic Control, 19 (6): 716--723, 1974.
3. I. Arapakis, M. Lalmas, and G. Valkanas. Understanding within-content engagement through pattern analysis of mouse gestures. In Proc. CIKM, pages 1439--1448, 2014.
4. A. Broder. A taxonomy of web search. SIGIR Forum, 36 (2), Fall 2002.
5. Cabinet Office. Digital efficiency report, November 2012. Retrieved 21 August 2013 from http://publications.cabinetoffice.gov.uk/digital/efficiency/digital-efficiency-report.pdf.
6. J. M. Cortina. What is coefficient alpha? An examination of theory and applications. J Applied Psychology, 28 (1): 98--104, 1993.
7. M. Csikszentmihalyi. Flow: The Psychology of Optimal Experience. Harper & Row, New York, 1990.
8. R. F. DeVellis. Scale development: Theory and applications. Sage, Thousand Oaks, California, 2nd edition, 2003.
9. H. Feild, J. Allan, and R. Jones. Predicting searcher frustration. In Proc. SIGIR, pages 34--41, 2010.
10. Government Digital Service. Government service design manual: User satisfaction, 2014. Retrieved 17 August 2014 from https://www.gov.uk/service-manual/measurement/user-satisfaction.html.
11. T. Joachims, L. Granka, B. Pan, H. Hembrooke, F. Radlinksi, and G. Gay. Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search. ACM Trans. Info. Systems, 25 (2), April 2007.
12. G. Lindgaard, G. Fernandes, C. Dudek, and J. Brown. Attention web designers: You have 50 milliseconds to make a good first impression! Behaviour and Information Technology, 25: 115--126, 2006.
13. H. L. O'Brien. The influence of hedonic and utilitarian motivations on user engagement: the case of online shopping experiences. Interacting with Computers, 22 (4): 344--352, 2010.
14. H. L. O'Brien and P. Cairns. Generalizing the User Engagement Scale to online news websites. Information Processing and Management, 51 (4): 413--427, 2015.
15. H. L. O'Brien and E. G. Toms. The development and evaluation of a survey to measure user engagement. J American Society for Information Science and Technology, 61 (1): 50--69, 2010.
16. D. Parra and P. Brusilovsky. User-controllable personalization: A case study with set-fusion. Int. J Human-Computer Studies, 77: 43--67, 2014.
17. P. Thomas. Using interaction data to explain difficulty navigating online. ACM Transactions on the Web, 8 (4): 24, 2014.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI design and evaluation methods

      • User studies

• Information systems

  • Information retrieval

    • Users and interactive retrieval

      • Search interfaces

  • World Wide Web

    • Web applications