Periodicity in User Engagement with a Search Engine and Its Application to Online Controlled Experiments

Abstract

Nowadays, billions of people use the Web in connection with their daily needs. A significant part of these needs are constituted by search tasks that are usually addressed by search engines. Thus, daily search needs result in regular user engagement with a search engine. User engagement with web services was studied in various aspects, but there appears to be little work devoted to its regularity and periodicity. In this article, we study periodicity of user engagement with a popular search engine through applying spectrum analysis to temporal sequences of different engagement metrics. First, we found periodicity patterns of user engagement and revealed classes of users whose periodicity patterns do not change over a long period of time. In addition, we give an exhaustive analysis of the stability and quality of identified clusters. Second, we used the spectrum series as key metrics to evaluate search quality. We found that the novel periodicity metrics outperform the state-of-the-art quality metrics both in terms of significance level (p-value) and sensitivity to a large set of larges-scale A/B experiments conducted on real search engine users.

References

1. Eytan Adar, Jaime Teevan, and Susan T. Dumais. 2008. Large scale analysis of web revisitation patterns. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1197--1206.
2. Eytan Adar, Jaime Teevan, and Susan T. Dumais. 2009a. Resonance on the web: Web dynamics and revisitation patterns. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1381--1390.
3. Eytan Adar, Jaime Teevan, Susan T. Dumais, and Jonathan L. Elsas. 2009b. The web changes everything: Understanding the dynamics of web content. In Proceedings of the 2nd ACM International Conference on Web Search and Data Mining. ACM, 282--291.
4. Olga Arkhipova, Lidia Grauer, Igor Kuralenok, and Pavel Serdyukov. 2015. Search engine evaluation based on search engine switching prediction. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 723--726.
5. Sunandan Chakraborty, Filip Radlinski, Milad Shokouhi, and Paul Baecke. 2014. On correlation of absence time and search effectiveness. In Proceedings of the 37th International ACM SIGIR Conference on Research 8 Development in Information Retrieval. ACM, 1163--1166.
6. William T. Cochran, James W. Cooley, David L. Favin, Howard D. Helms, Reg A. Kaenel, William W. Lang, George C. Maling Jr., David E. Nelson, Charles M. Rader, and Peter D. Welch. 1967. What is the fast fourier transform? Proc. IEEE 55, 10 (1967), 1664--1674.
7. James W. Cooley and John W. Tukey. 1965. An algorithm for the machine calculation of complex Fourier series. Math. Comput. 19, 90 (1965), 297--301.
8. Thomas Crook, Brian Frasca, Ron Kohavi, and Roger Longbotham. 2009. Seven pitfalls to avoid when running controlled experiments on the web. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 1105--1114.
9. David L. Davies and Donald W. Bouldin. 1979. A cluster separation measure. IEEE Trans. Pattern Anal. Mach. Intell. 2 (1979), 224--227.
10. Alex Deng. 2015. Objective bayesian two sample hypothesis testing for online controlled experiments. In Proceedings of the 24th International Conference on World Wide Web Companion. International World Wide Web Conferences Steering Committee, 923--928.
11. Alex Deng and Victor Hu. 2015. Diluted treatment effect estimation for trigger analysis in online controlled experiments. In Proceedings of the 8th ACM International Conference on Web Search and Data Mining. ACM, 349--358.
12. Alex Deng, Tianxi Li, and Yu Guo. 2014. Statistical inference in two-stage online controlled experiments with treatment selection and validation. In Proceedings of the 23rd International Conference on World Wide Web. ACM, 609--618.
13. Alex Deng, Ya Xu, Ron Kohavi, and Toby Walker. 2013. Improving the sensitivity of online controlled experiments by utilizing pre-experiment data. In Proceedings of the 6th ACM International Conference on Web Search and Data Mining. ACM, 123--132.
14. Abdigani Diriye, Ryen White, Georg Buscher, and Susan Dumais. 2012. Leaving so soon?: Understanding and predicting web search abandonment rationales. In Proceedings of the 21st ACM International Conference on Information and Knowledge Management. ACM, 1025--1034.
15. Alexey Drutsa. 2015. Sign-aware periodicity metrics of user engagement for online search quality evaluation. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 779--782.
16. Alexey Drutsa, Gleb Gusev, and Pavel Serdyukov. 2015a. Engagement periodicity in search engine usage: Analysis and its application to search quality evaluation. In Proceedings of the 8th ACM International Conference on Web Search and Data Mining. ACM, 27--36.
17. Alexey Drutsa, Gleb Gusev, and Pavel Serdyukov. 2015b. Future user engagement prediction and its application to improve the sensitivity of online experiments. In Proceedings of the 24th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 256--266.
18. Alexey Drutsa, Anna Ufliand, and Gleb Gusev. 2015c. Practical aspects of sensitivity in online experimentation with user engagement metrics. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management. ACM, 763--772.
19. Georges Dupret and Mounia Lalmas. 2013. Absence time and user engagement: Evaluating ranking functions. In Proceedings of the Sixth ACM International Conference on Web Search and Data Mining. ACM, 173--182.
20. Henry A. Feild, James Allan, and Rosie Jones. 2010. Predicting searcher frustration. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 34--41.
21. David A. Freedman, David Collier, Jasjeet S. Sekhon, and Philip B. Stark. 2010. Statistical Models and Causal Inference: A Dialogue with the Social Sciences. Cambridge University Press.
22. Qi Guo, Ryen W. White, Yunqiao Zhang, Blake Anderson, and Susan T. Dumais. 2011. Why searchers switch: Understanding and predicting engine switching rationales. In Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 335--344.
23. Ahmed Hassan, Xiaolin Shi, Nick Craswell, and Bill Ramsey. 2013. Beyond clicks: Query reformulation as a predictor of search satisfaction. In Proceedings of the 22nd ACM International Conference on Conference on Information 8 Knowledge Management. ACM, 2019--2028.
24. Ahmed Hassan, Yang Song, and Li-wei He. 2011. A task level metric for measuring web search satisfaction and its application on improving relevance estimation. In Proceedings of the 20th ACM International Conference on Information and Knowledge Management. ACM, 125--134.
25. Ahmed Hassan and Ryen W. White. 2013. Personalized models of search satisfaction. In Proceedings of the 22nd ACM International Conference on Conference on Information 8 Knowledge Management. ACM, 2009--2018.
26. Henning Hohnhold, Deirdre O’Brien, and Diane Tang. 2015. Focusing on the long-term: It’s good for users and business. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 1849--1858.
27. Victor Hu, Maria Stone, Jan Pedersen, and Ryen W White. 2011. Effects of search success on search engine re-use. In Proceedings of the 20th ACM International Conference on Information and Knowledge Management. ACM, 1841--1846.
28. Bernard J. Jansen, Amanda Spink, and Vinish Kathuria. 2007. How to define searching sessions on web search engines. In Advances in Web Mining and Web Usage Analysis. Springer, 92--109.
29. Rosie Jones and Kristina Lisa Klinkner. 2008. Beyond the session timeout: Automatic hierarchical segmentation of search topics in query logs. In Proceedings of the 17th ACM Conference on Information and Knowledge Management. ACM, 699--708.
30. Eugene Kharitonov, Craig Macdonald, Pavel Serdyukov, and Iadh Ounis. 2015a. Optimised scheduling of online experiments. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 453--462.
31. Eugene Kharitonov, Aleksandr Vorobev, Craig Macdonald, Pavel Serdyukov, and Iadh Ounis. 2015b. Sequential testing for early stopping of online experiments. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 473--482.
32. Ronny Kohavi, Thomas Crook, Roger Longbotham, Brian Frasca, Randy Henne, Juan Lavista Ferres, and Tamir Melamed. 2009. Online experimentation at microsoft. (2009), 11--23.
33. Ron Kohavi, Alex Deng, Brian Frasca, Roger Longbotham, Toby Walker, and Ya Xu. 2012. Trustworthy online controlled experiments: Five puzzling outcomes explained. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 786--794.
34. Ron Kohavi, Alex Deng, Brian Frasca, Toby Walker, Ya Xu, and Nils Pohlmann. 2013. Online controlled experiments at large scale. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 1168--1176.
35. Ron Kohavi, Alex Deng, Roger Longbotham, and Ya Xu. 2014. Seven rules of thumb for web site experimenters. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 1857--1866.
36. Ron Kohavi, Randal M. Henne, and Dan Sommerfield. 2007. Practical guide to controlled experiments on the web: Listen to your customers not to the hippo. In Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 959--967.
37. Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M. Henne. 2009. Controlled experiments on the web: Survey and practical guide. Data Min. Knowl. Discov. 18, 1 (2009), 140--181.
38. Ron Kohavi, David Messner, Seth Eliot, Juan Lavista Ferres, Randy Henne, Vignesh Kannappan, and Justin Wang. 2010. Tracking Users Clicks and Submits: Tradeoffs between User Experience and Data Loss. (2010).
39. Tomáš Kramár and Mária Bieliková. 2014. Context of seasonality in web search. In Advances in Information Retrieval. Springer, 644--649.
40. Janette Lehmann, Mounia Lalmas, Georges Dupret, and Ricardo Baeza-Yates. 2013. Online multitasking and user engagement. In Proceedings of the 22nd ACM International Conference on Conference on Information 8 Knowledge Management. ACM, 519--528.
41. Janette Lehmann, Mounia Lalmas, Elad Yom-Tov, and Georges Dupret. 2012. Models of user engagement. In User Modeling, Adaptation, and Personalization. Springer, 164--175.
42. Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando, and Gabriele Tolomei. 2016. Learning to rank user queries to detect search tasks. In Proceedings of the 2016 ACM on International Conference on the Theory of Information Retrieval. ACM, 157--166.
43. Claudio Lucchese, Salvatore Orlando, Raffaele Perego, Fabrizio Silvestri, and Gabriele Tolomei. 2011. Identifying task-based sessions in search engine query logs. In Proceedings of the 4th ACM International Conference on Web Search and Data Mining. ACM, 277--286.
44. Claudio Lucchese, Salvatore Orlando, Raffaele Perego, Fabrizio Silvestri, and Gabriele Tolomei. 2013. Discovering tasks from search engine query logs. ACM Trans. Inf. Syst. 31, 3 (2013), 14.
45. Stephen L. Morgan and Christopher Winship. 2014. Counterfactuals and Causal Inference. Cambridge University Press.
46. Kirill Nikolaev, Alexey Drutsa, Ekaterina Gladkikh, Alexander Ulianov, Gleb Gusev, and Pavel Serdyukov. 2015. Extreme states distribution decomposition method for search engine online evaluation. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 845--854.
47. Eric T. Peterson. 2004. Web Analytics Demystified: A Marketer’s Guide to Understanding How Your Web Site Affects Your Business. Ingram.
48. Alexey Poyarkov, Alexey Drutsa, Andrey Khalyavin, Gleb Gusev, and Pavel Serdyukov. 2016. Boosted decision tree regression adjustment for variance reduction in online controlled experiments. In Proceedings of the 22th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 235--244.
49. Kira Radinsky, Krysta Svore, Susan Dumais, Jaime Teevan, Alex Bocharov, and Eric Horvitz. 2012. Modeling and predicting behavioral dynamics on the web. In Proceedings of the 21st International Conference on World Wide Web. ACM, 599--608.
50. Kerry Rodden, Hilary Hutchinson, and Xin Fu. 2010. Measuring the user experience on a large scale: User-centered metrics for web applications. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2395--2398.
51. Peter J. Rousseeuw. 1987. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math. 20 (1987), 53--65.
52. Tetsuya Sakai. 2006. Evaluating evaluation metrics based on the bootstrap. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 525--532.
53. Denis Savenkov, Dmitry Lagun, and Qiaoling Liu. 2013. Search engine switching detection based on user personal preferences and behavior patterns. In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 33--42.
54. Milad Shokouhi. 2011. Detecting seasonal queries by time-series analysis. In Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 1171--1172.
55. Yang Song, Xiaolin Shi, and Xin Fu. 2013. Evaluating and predicting user engagement change with degraded search relevance. In Proceedings of the 22nd International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 1213--1224.
56. Diane Tang, Ashish Agarwal, Deirdre O’Brien, and Mike Meyer. 2010. Overlapping experiment infrastructure: More, better, faster experimentation. In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 17--26.
57. Jaime Teevan, Susan T. Dumais, Daniel J. Liebling, and Richard L. Hughes. 2009. Changing how people view changes on the web. In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology. ACM, 237--246.
58. Michail Vlachos, Christopher Meek, Zografoula Vagena, and Dimitrios Gunopulos. 2004. Identifying similarities, periodicities and bursts for online search queries. In Proceedings of the 2004 ACM SIGMOD International Conference on Management of Data. ACM, 131--142.
59. Hongning Wang, Yang Song, Ming-Wei Chang, Xiaodong He, Ryen W. White, and Wei Chu. 2013. Learning to extract cross-session search tasks. In Proceedings of the 22nd International Conference on World Wide Web. ACM, 1353--1364.
60. William Wu-Shyong Wei. 1994. Time Series Analysis. Addison-Wesley Redwood City, CA.
61. Robert West, Ryen W. White, and Eric Horvitz. 2013. From cookies to cooks: Insights on dietary patterns via analysis of web usage logs. In Proceedings of the 22nd International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 1399--1410.
62. Ryen W. White and Susan T. Dumais. 2009. Characterizing and predicting search engine switching behavior. In Proceedings of the 18th ACM Conference on Information and Knowledge Management. ACM, 87--96.
63. Ryen W. White, Ashish Kapoor, and Susan T. Dumais. 2010. Modeling long-term search engine usage. In UMAP. 28--39.
64. Ying Zhang, Bernard J. Jansen, and Amanda Spink. 2009. Time series analysis of a Web search engine transaction log. Inf. Process. Manag. 45, 2 (2009), 230--245.

Index Terms

• General and reference

  • Cross-computing tools and techniques

    • Evaluation

    • Metrics

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI design and evaluation methods

      • User models

      • User studies

• Information systems

  • World Wide Web

    • Web mining

      • Traffic analysis

      • Web log analysis