Mining touch interaction data on mobile devices to predict web search result relevance

ABSTRACT

Fine-grained search interactions in the desktop setting, such as mouse cursor movements and scrolling, have been shown valuable for understanding user intent, attention, and their preferences for Web search results. As web search on smart phones and tablets becomes increasingly popular, previously validated desktop interaction models have to be adapted for the available touch interactions such as pinching and swiping, and for the different device form factors. In this paper, we present, to our knowledge, the first in-depth study of modeling interactions on touch-enabled device for improving Web search ranking. In particular, we evaluate a variety of touch interactions on a smart phone as implicit relevance feedback, and compare them with the corresponding fine-grained interactions on a desktop computer with mouse and keyboard as the primary input devices. Our experiments are based on a dataset collected from two user studies with 56 users in total, using a specially instrumented version of a popular mobile browser to capture the interaction data. We report a detailed analysis of the similarities and differences of fine-grained search interactions between the desktop and the smart phone modalities, and identify novel patterns of touch interactions indicative of result relevance. Finally, we demonstrate significant improvements to search ranking quality by mining touch interaction data.

References

1. E. Agichtein, E. Brill, and S. Dumais. Improving web search ranking by incorporating user behavior information. SIGIR '06, pages 19--26, 2006.
2. J.-w. Ahn, P. Brusilovsky, D. He, J. Grady, and Q. Li. Personalized web exploration with task models. In Proceeding of the 17th international conference on World Wide Web, WWW '08, pages 1--10, 2008.
3. R. Biedert, A. Dengel, G. Buscher, and A. Vartan. Reading and estimating gaze on smart phones. ETRA '12, pages 385--388, New York, NY, USA, 2012. ACM.
4. L. Breiman. Bagging predictors. Mach. Learn., 24(2):123--140, Aug. 1996.
5. G. Buscher, A. Dengel, and L. van Elst. Query expansion using gaze-based feedback on the subdocument level. SIGIR '08, pages 387--394, 2008.
6. G. Buscher, R. W. White, S. Dumais, and J. Huang. Large-scale analysis of individual and task differences in search result page examination strategies. WSDM '12, pages 373--382, 2012.
7. K. Church and N. Oliver. Understanding mobile web and mobile search use in today's dynamic mobile landscape. In Proc. of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services, 2011.
8. K. Church and B. Smyth. Understanding the intent behind mobile information needs. In Proceedings of the 14th international conference on Intelligent user interfaces, IUI '09, pages 247--256, 2009.
9. M. Claypool, P. Le, M. Wased, and D. Brown. Implicit interest indicators. IUI '01, pages 33--40, 2001.
10. H. A. Feild, J. Allan, and R. Jones. Predicting searcher frustration. SIGIR '10, pages 34--41.
11. S. Fox, K. Karnawat, M. Mydland, S. Dumais, and T. White. Evaluating implicit measures to improve web search. ACM Transactions on Information Systems, 23(2), 2005.
12. G. Golovchinsky, M. N. Price, and B. N. Schilit. From reading to retrieval: freeform ink annotations as queries. SIGIR '99, pages 19--25.
13. Q. Guo and E. Agichtein. Exploring mouse movements for inferring query intent. SIGIR '08, pages 707--708, 2008.
14. Q. Guo and E. Agichtein. Ready to buy or just browsing?: detecting web searcher goals from interaction data. SIGIR '10, pages 130--137, 2010.
15. Q. Guo and E. Agichtein. Towards predicting web searcher gaze position from mouse movements. CHI EA '10, pages 3601--3606, 2010.
16. Q. Guo and E. Agichtein. Beyond dwell time: estimating document relevance from cursor movements and other post-click searcher behavior. WWW '12, 2012.
17. Q. Guo, S. Yuan, and E. Agichtein. Detecting success in mobile search from interaction. SIGIR '11, 2011.
18. A. Hassan, R. Jones, and K. L. Klinkner. Beyond dcg: user behavior as a predictor of a successful search. WSDM '10, pages 221--230.
19. A. Hassan, Y. Song, and L.-w. He. A task level user satisfaction metric and its application on improving relevance estimation. CIKM '11, 2011.
20. J. Huang and A. M. Diriye. Web user interaction mining from touch-enabled mobile devices. HCIR '12, 2012.
21. J. Huang, R. White, and G. Buscher. User see, user point: gaze and cursor alignment in web search. CHI '12, pages 1341--1350, 2012.
22. J. Huang, R. W. White, G. Buscher, and K. Wang. Improving searcher models using mouse cursor activity. SIGIR '12, pages 195--204, 2012.
23. J. Huang, R. W. White, and S. Dumais. No clicks, no problem: using cursor movements to understand and improve search. CHI '11, pages 1225--1234, 2011.
24. K. Järvelin and J. Kekäläinen. Ir evaluation methods for retrieving highly relevant documents. SIGIR '00, pages 41--48, 2000.
25. T. Joachims, L. Granka, B. Pan, H. Hembrooke, F. Radlinski, and G. Gay. Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search. ACM Trans. Inf. Syst., 25(2), 2007.
26. M. Kamvar, M. Kellar, R. Patel, and Y. Xu. Computers and iphones and mobile phones, oh my!: a logs-based comparison of search users on different devices. WWW '09, pages 801--810, 2009.
27. D. Kelly and N. J. Belkin. Reading time, scrolling and interaction: exploring implicit sources of user preferences for relevance feedback. SIGIR '01, pages 408--409, 2001.
28. D. Kelly and N. J. Belkin. Display time as implicit feedback: understanding task effects. SIGIR '04, pages 377--384, 2004.
29. M. Melucci and R. W. White. Discovering hidden contextual factors for implicit feedback. In CIR'07, pages 1--1, 2007.
30. M. Morita and Y. Shinoda. Information filtering based on user behavior analysis and best match text retrieval. SIGIR '94, pages 272--281, 1994.
31. V. Navalpakkam and E. Churchill. Mouse tracking: measuring and predicting users' experience of web-based content. CHI '12, pages 2963--2972, 2012.
32. R. Rafter and B. Smyth. Passive profiling from server logs in an online recruitment environment. In Proc. of ITWP, 2001.
33. K. Rodden, X. Fu, A. Aula, and I. Spiro. Eye-mouse coordination patterns on web search results pages. CHI EA '08, pages 2997--3002, 2008.
34. R. W. White and G. Buscher. Text selections as implicit relevance feedback. SIGIR '12, pages 1151--1152, 2012.
35. R. W. White and S. T. Dumais. Characterizing and predicting search engine switching behavior. CIKM '09, 2009.
36. R. W. White and D. Kelly. A study on the effects of personalization and task information on implicit feedback performance. CIKM '06, pages 297--306, 2006.

Index Terms

• Information systems

  • Information retrieval

    • Evaluation of retrieval results

      • Relevance assessment

    • Information retrieval query processing

    • Retrieval models and ranking

    • Retrieval tasks and goals