Abstract
Aggregated search is the task of combining results from multiple independent search systems in a single Search Engine Results Page (SERP). Aggregated search coherence refers to the extent to which different sources on the SERP focus on similar senses of an ambiguous or underspecified query. In previous studies, we found that the query senses in a set of vertical results can influence user engagement with the web results (the so-called “spillover” effect). In this work, we investigate five research questions (RQ1--RQ5) that extend our prior work. First, we investigate the extent to which results from different sources focus on different senses of an ambiguous query (RQ1). Second, we investigate how the vertical-to-web spillover effect varies across different verticals (RQ2). Then, we examine whether the level of spillover depends on the vertical position (RQ3) and on whether the vertical results are displayed with a border and different-colored background to distinguish them from the web results (RQ4). Finally, we propose a new method for displaying results from a particular vertical that are more consistent with the query senses in the web results (RQ5). We evaluate this new method based on how it influences users to make more correct decisions with respect to the web results—to engage with the web results when at least one of them is relevant and to avoid engaging with the web results otherwise. Our results show the following trends. In terms of RQ1, our analysis suggests that the top results from the web search engine are more diversified than the top results from our four different verticals considered (images, news, shopping, and video). In terms of RQ2, we found a stronger spillover effect for the images vertical than the news, shopping, and video verticals. In terms of RQ3, we found a stronger level of spillover when the vertical was positioned at the top of the SERP versus to the right side of the web results. In terms of RQ4, we found an interesting additive effect between the vertical’s position and displaying the vertical results enclosed in a border and with a different-colored background—the image vertical had no spillover when presented to the right side of the web results and with a border and background. Finally, in terms of RQ5, we found that our proposed vertical results selection approach can influence users to make more correct predictions about their level of engagement with the web results.
- Rakesh Agrawal, Sreenivas Gollapudi, Alan Halverson, and Samuel Ieong. 2009. Diversifying search results. In WSDM. ACM, 5--14. Google ScholarDigital Library
- Jaime Arguello. 2015. Improving aggregated search coherence. In Advances in Information Retrieval, Allan Hanbury, Gabriella Kazai, Andreas Rauber, and Norbert Fuhr (Eds.). Lecture Notes in Computer Science, Vol. 9022. Springer International Publishing, 25--36.Google Scholar
- Jaime Arguello and Robert Capra. 2012. The effect of aggregated search coherence on search behavior. In Proceedings of the 21st ACM International Conference on Information and Knowledge Management (CIKM’12). ACM, New York, NY, 1293--1302. Google ScholarDigital Library
- Jaime Arguello and Robert Capra. 2014. The effects of vertical rank and border on aggregated search coherence and search behavior. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management (CIKM’14). ACM, New York, NY, 539--548. Google ScholarDigital Library
- Jaime Arguello, Robert Capra, and Wan-Ching Wu. 2013. Factors affecting aggregated search coherence and search behavior. In Proceedings of the 22nd ACM International Conference on Conference on Information & Knowledge Management (CIKM’13). ACM, New York, NY, 1989--1998. Google ScholarDigital Library
- Jaime Arguello, Fernando Diaz, and Jamie Callan. 2011. Learning to aggregate vertical results into web search results. In Proceedings of the 20th ACM International Conference on Information and Knowledge Management (CIKM’11). ACM, New York, NY, 201--210. Google ScholarDigital Library
- Jaime Arguello, Fernando Diaz, Jamie Callan, and Jean-Francois Crespo. 2009. Sources of evidence for vertical selection. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’09). ACM, New York, NY, 315--322. Google ScholarDigital Library
- Jaime Arguello, Fernando Diaz, and Jean-François Paiement. 2010. Vertical selection in the presence of unlabeled verticals. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’10). ACM, New York, NY, 691--698. Google ScholarDigital Library
- Jaime Arguello, Wan-Ching Wu, Diane Kelly, and Ashlee Edwards. 2012. Task complexity, vertical display, and user interaction in aggregated search. In Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’12). ACM, New York, NY, 435--444. Google ScholarDigital Library
- Peter Bailey, Nick Craswell, Ryen W. White, Liwei Chen, Ashwin Satyanarayana, and S. M. M. Tahaghoghi. 2010. Evaluating search systems using result page context. In Proceedings of the 3rd Symposium on Information Interaction in Context (IIiX’10). ACM, New York, NY, 105--114. Google ScholarDigital Library
- Horaţiu Bota, Ke Zhou, and Joemon J. Jose. 2015. Advances in Information Retrieval. In Proceedings of the 37th European Conference on IR Research (ECIR 2015). Springer International Publishing, 13--24.Google Scholar
- Horatiu Bota, Ke Zhou, and Joemon M. Jose. 2016. Playing your cards right: The effect of entity cards on search behaviour and workload. In Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval (CHIIR’16). ACM, New York, NY, 131--140. Google ScholarDigital Library
- Georg Buscher, Susan T. Dumais, and Edward Cutrell. 2010. The good, the bad, and the random: An eye-tracking study of ad quality in web search. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’10). ACM, New York, NY, 42--49. Google ScholarDigital Library
- Jaime Carbonell and Jade Goldstein. 1998. The use of MMR, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’98). ACM, New York, NY, 335--336. Google ScholarDigital Library
- Ben Carterette and Praveen Chandar. 2009. Probabilistic models of ranking novel documents for faceted topic retrieval. In Proceedings of the 18th ACM Conference on Information and Knowledge Management (CIKM’09). ACM, New York, NY, 1287--1296. Google ScholarDigital Library
- Aleksandr Chuklin, Anne Schuth, Katja Hofmann, Pavel Serdyukov, and Maarten de Rijke. 2013. Evaluating aggregated search using interleaving. In Proceedings of the 22nd ACM International Conference on Conference on Information & Knowledge Management (CIKM’13). ACM, New York, NY, 669--678. Google ScholarDigital Library
- Aleksandr Chuklin, Anne Schuth, Ke Zhou, and Maarten De Rijke. 2015. A comparative analysis of interleaving methods for aggregated search. ACM Transactions on Information Systems 33, 2, Article 5 (Feb. 2015). Google ScholarDigital Library
- Cyril W. Cleverdon. 1960. The Aslib Cranfield research project on the comparative efficiency of indexing systems. Aslib Proceedings 12, 12 (1960), 421--431.Google ScholarCross Ref
- Edward Cutrell and Zhiwei Guan. 2007. What are you looking for? An eye-tracking study of information usage in web search. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’07). ACM, New York, NY, 407--416. Google ScholarDigital Library
- Fernando Diaz. 2009. Integration of news content into web results. In Proceedings of the 2nd ACM International Conference on Web Search and Data Mining (WSDM’09). ACM, New York, NY, 182--191. Google ScholarDigital Library
- Fernando Diaz and Jaime Arguello. 2009. Adaptation of offline vertical selection predictions in the presence of user feedback. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’09). ACM, New York, NY, 323--330. Google ScholarDigital Library
- Fernando Diaz, Ryen White, Georg Buscher, and Dan Liebling. 2013. Robust models of mouse movement on dynamic web search results pages. In Proceedings of the 22nd ACM International Conference on Information & Knowledge Management (CIKM’13). ACM, New York, NY, 1451--1460. Google ScholarDigital Library
- Sergio Duarte Torres, Djoerd Hiemstra, and Theo Huibers. 2013. Vertical selection in the information domain of children. In Proceedings of the 13th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL’13). ACM, New York, NY, 57--66. Google ScholarDigital Library
- Susan T. Dumais, Georg Buscher, and Edward Cutrell. 2010. Individual differences in gaze patterns for web search. In Proceedings of the 3rd Symposium on Information Interaction in Context (IIiX’10). ACM, New York, NY, 185--194. Google ScholarDigital Library
- Luo Jie, Sudarshan Lamkhede, Rochit Sapra, Evans Hsu, Helen Song, and Yi Chang. 2013. A unified search federation system based on online user feedback. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’13). ACM, New York, NY, 1195--1203. Google ScholarDigital Library
- Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. 2005. Accurately interpreting clickthrough data as implicit feedback. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’05). ACM, New York, NY, 154--161. Google ScholarDigital Library
- Kurt Koffka. 1935. Principles of Gestalt Psychology. Harcourt, New York.Google Scholar
- Arnd Christian König, Michael Gamon, and Qiang Wu. 2009. Click-through prediction for news queries. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’09). ACM, New York, NY, 347--354. Google ScholarDigital Library
- J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics 33, 1 (1977), 159--174.Google ScholarCross Ref
- Xiao Li, Ye-Yi Wang, and Alex Acero. 2008. Learning query intent from regularized click graphs. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’08). ACM, New York, NY, 339--346. Google ScholarDigital Library
- Zeyang Liu, Yiqun Liu, Ke Zhou, Min Zhang, and Shaoping Ma. 2015. Influence of vertical result in web search examination. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’15). ACM, New York, NY, 193--202. Google ScholarDigital Library
- Stephen E. Palmer. 1992. Common region: A new principle of perceptual grouping. Cognitive Psychology 24, 3 (1992), 436--447.Google ScholarCross Ref
- Bing Pan, Helene Hembrooke, Thorsten Joachims, Lori Lorigo, Geri Gay, and Laura Granka. 2007. In Google we trust: Users’ decisions on rank, position, and relevance. Journal of Computer-Mediated Communication 12, 3 (2007), 801--823.Google ScholarCross Ref
- Ashok Kumar Ponnuswami, Kumaresh Pattabiraman, Desmond Brand, and Tapas Kanungo. 2011a. Model characterization curves for federated search using click-logs: Predicting user engagement metrics for the span of feasible operating points. In Proceedings of the 20th International Conference on World Wide Web (WWW’11). ACM, New York, NY, 67--76. Google ScholarDigital Library
- Ashok Kumar Ponnuswami, Kumaresh Pattabiraman, Qiang Wu, Ran Gilad-Bachrach, and Tapas Kanungo. 2011b. On composition of a federated web search result page: Using online users to provide pairwise preference for heterogeneous verticals. In Proceedings of the 4th ACM International Conference on Web Search and Data Mining (WSDM’11). ACM, New York, NY, 715--724. Google ScholarDigital Library
- Filip Radlinski and Susan Dumais. 2006. Improving personalized web search using result diversification. In SIGIR. ACM, 691--692. Google ScholarDigital Library
- Mark Sanderson. 2008. Ambiguous queries: Test collections need more sense. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’08). ACM, New York, NY, 499--506. Google ScholarDigital Library
- Rodrygo L. T. Santos, Craig Macdonald, and Iadh Ounis. 2010. Exploiting query reformulations for web search result diversification. In WWW. ACM, 881--890. Google ScholarDigital Library
- Rodrygo L. T. Santos, Craig Macdonald, and Iadh Ounis. 2011a. Aggregated search result diversification. In ITCIR. Springer-Verlag, 250--261. Google ScholarDigital Library
- Rodrygo L. T. Santos, Craig Macdonald, and Iadh Ounis. 2011b. Aggregated search result diversification. In Proceedings of the 3rd International Conference on Advances in Information Retrieval Theory (ICTIR’11). Springer-Verlag, Berlin, 250--261. Google ScholarDigital Library
- Rodrygo L. T. Santos, Craig Macdonald, and Iadh Ounis. 2015. Search result diversification. Foundations and Trends in Information Retrieval 9, 1 (2015), 1--90. http://dx.doi.org/10.1561/1500000040 Google ScholarDigital Library
- Mark D. Smucker, James Allan, and Ben Carterette. 2007. A comparison of statistical significance tests for information retrieval evaluation. In Proceedings of the 16th ACM Conference on Conference on Information and Knowledge Management (CIKM’07). ACM, New York, NY, 623--632. Google ScholarDigital Library
- Cheng Xiang Zhai, William W. Cohen, and John Lafferty. 2003. Beyond independent relevance: Methods and evaluation metrics for subtopic retrieval. In SIGIR. ACM, 10--17. Google ScholarDigital Library
- Ke Zhou, Ronan Cummins, Martin Halvey, Mounia Lalmas, and Joemon M. Jose. 2012a. Assessing and predicting vertical intent for web queries. In Proceedings of the 34th European Conference on Advances in Information Retrieval (ECIR’12). Springer-Verlag, Berlin, 499--502. Google ScholarDigital Library
- Ke Zhou, Ronan Cummins, Mounia Lalmas, and Joemon M. Jose. 2012b. Evaluating aggregated search pages. In Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’12). ACM, New York, NY, 115--124. Google ScholarDigital Library
- Ke Zhou, Ronan Cummins, Mounia Lalmas, and Joemon M. Jose. 2012c. Evaluating reward and risk for vertical selection. In Proceedings of the 21st ACM International Conference on Information and Knowledge Management (CIKM’12). ACM, New York, NY, 2631--2634. Google ScholarDigital Library
Index Terms
The Effects of Aggregated Search Coherence on Search Behavior
Recommendations
The Effects of Vertical Rank and Border on Aggregated Search Coherence and Search Behavior
CIKM '14: Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge ManagementAggregated search is the task of blending results from different search services, or verticals, into a set of web search results. Aggregated search coherence is the extent to which results from different sources focus on similar senses of an ambiguous ...
The effect of aggregated search coherence on search behavior
CIKM '12: Proceedings of the 21st ACM international conference on Information and knowledge managementAggregated search is the task of blending results from different specialized search services, or verticals, into the web search results. Aggregated search coherence refers to the degree to which results from different systems focus on similar senses of ...
Interest and Evaluation of Aggregated Search
WI-IAT '11: Proceedings of the 2011 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology - Volume 01Major search engines perform what is known as Aggregated Search (AS). They integrate results coming from different vertical search engines (images, videos, news, etc.) with typical Web search results. Aggregated search is relatively new and its ...
Comments