skip to main content
research-article

The Effects of Aggregated Search Coherence on Search Behavior

Authors Info & Claims
Published:22 September 2016Publication History
Skip Abstract Section

Abstract

Aggregated search is the task of combining results from multiple independent search systems in a single Search Engine Results Page (SERP). Aggregated search coherence refers to the extent to which different sources on the SERP focus on similar senses of an ambiguous or underspecified query. In previous studies, we found that the query senses in a set of vertical results can influence user engagement with the web results (the so-called “spillover” effect). In this work, we investigate five research questions (RQ1--RQ5) that extend our prior work. First, we investigate the extent to which results from different sources focus on different senses of an ambiguous query (RQ1). Second, we investigate how the vertical-to-web spillover effect varies across different verticals (RQ2). Then, we examine whether the level of spillover depends on the vertical position (RQ3) and on whether the vertical results are displayed with a border and different-colored background to distinguish them from the web results (RQ4). Finally, we propose a new method for displaying results from a particular vertical that are more consistent with the query senses in the web results (RQ5). We evaluate this new method based on how it influences users to make more correct decisions with respect to the web results—to engage with the web results when at least one of them is relevant and to avoid engaging with the web results otherwise. Our results show the following trends. In terms of RQ1, our analysis suggests that the top results from the web search engine are more diversified than the top results from our four different verticals considered (images, news, shopping, and video). In terms of RQ2, we found a stronger spillover effect for the images vertical than the news, shopping, and video verticals. In terms of RQ3, we found a stronger level of spillover when the vertical was positioned at the top of the SERP versus to the right side of the web results. In terms of RQ4, we found an interesting additive effect between the vertical’s position and displaying the vertical results enclosed in a border and with a different-colored background—the image vertical had no spillover when presented to the right side of the web results and with a border and background. Finally, in terms of RQ5, we found that our proposed vertical results selection approach can influence users to make more correct predictions about their level of engagement with the web results.

References

  1. Rakesh Agrawal, Sreenivas Gollapudi, Alan Halverson, and Samuel Ieong. 2009. Diversifying search results. In WSDM. ACM, 5--14. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Jaime Arguello. 2015. Improving aggregated search coherence. In Advances in Information Retrieval, Allan Hanbury, Gabriella Kazai, Andreas Rauber, and Norbert Fuhr (Eds.). Lecture Notes in Computer Science, Vol. 9022. Springer International Publishing, 25--36.Google ScholarGoogle Scholar
  3. Jaime Arguello and Robert Capra. 2012. The effect of aggregated search coherence on search behavior. In Proceedings of the 21st ACM International Conference on Information and Knowledge Management (CIKM’12). ACM, New York, NY, 1293--1302. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Jaime Arguello and Robert Capra. 2014. The effects of vertical rank and border on aggregated search coherence and search behavior. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management (CIKM’14). ACM, New York, NY, 539--548. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Jaime Arguello, Robert Capra, and Wan-Ching Wu. 2013. Factors affecting aggregated search coherence and search behavior. In Proceedings of the 22nd ACM International Conference on Conference on Information & Knowledge Management (CIKM’13). ACM, New York, NY, 1989--1998. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Jaime Arguello, Fernando Diaz, and Jamie Callan. 2011. Learning to aggregate vertical results into web search results. In Proceedings of the 20th ACM International Conference on Information and Knowledge Management (CIKM’11). ACM, New York, NY, 201--210. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Jaime Arguello, Fernando Diaz, Jamie Callan, and Jean-Francois Crespo. 2009. Sources of evidence for vertical selection. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’09). ACM, New York, NY, 315--322. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Jaime Arguello, Fernando Diaz, and Jean-François Paiement. 2010. Vertical selection in the presence of unlabeled verticals. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’10). ACM, New York, NY, 691--698. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Jaime Arguello, Wan-Ching Wu, Diane Kelly, and Ashlee Edwards. 2012. Task complexity, vertical display, and user interaction in aggregated search. In Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’12). ACM, New York, NY, 435--444. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Peter Bailey, Nick Craswell, Ryen W. White, Liwei Chen, Ashwin Satyanarayana, and S. M. M. Tahaghoghi. 2010. Evaluating search systems using result page context. In Proceedings of the 3rd Symposium on Information Interaction in Context (IIiX’10). ACM, New York, NY, 105--114. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Horaţiu Bota, Ke Zhou, and Joemon J. Jose. 2015. Advances in Information Retrieval. In Proceedings of the 37th European Conference on IR Research (ECIR 2015). Springer International Publishing, 13--24.Google ScholarGoogle Scholar
  12. Horatiu Bota, Ke Zhou, and Joemon M. Jose. 2016. Playing your cards right: The effect of entity cards on search behaviour and workload. In Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval (CHIIR’16). ACM, New York, NY, 131--140. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Georg Buscher, Susan T. Dumais, and Edward Cutrell. 2010. The good, the bad, and the random: An eye-tracking study of ad quality in web search. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’10). ACM, New York, NY, 42--49. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Jaime Carbonell and Jade Goldstein. 1998. The use of MMR, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’98). ACM, New York, NY, 335--336. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Ben Carterette and Praveen Chandar. 2009. Probabilistic models of ranking novel documents for faceted topic retrieval. In Proceedings of the 18th ACM Conference on Information and Knowledge Management (CIKM’09). ACM, New York, NY, 1287--1296. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Aleksandr Chuklin, Anne Schuth, Katja Hofmann, Pavel Serdyukov, and Maarten de Rijke. 2013. Evaluating aggregated search using interleaving. In Proceedings of the 22nd ACM International Conference on Conference on Information & Knowledge Management (CIKM’13). ACM, New York, NY, 669--678. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Aleksandr Chuklin, Anne Schuth, Ke Zhou, and Maarten De Rijke. 2015. A comparative analysis of interleaving methods for aggregated search. ACM Transactions on Information Systems 33, 2, Article 5 (Feb. 2015). Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Cyril W. Cleverdon. 1960. The Aslib Cranfield research project on the comparative efficiency of indexing systems. Aslib Proceedings 12, 12 (1960), 421--431.Google ScholarGoogle ScholarCross RefCross Ref
  19. Edward Cutrell and Zhiwei Guan. 2007. What are you looking for? An eye-tracking study of information usage in web search. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’07). ACM, New York, NY, 407--416. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Fernando Diaz. 2009. Integration of news content into web results. In Proceedings of the 2nd ACM International Conference on Web Search and Data Mining (WSDM’09). ACM, New York, NY, 182--191. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Fernando Diaz and Jaime Arguello. 2009. Adaptation of offline vertical selection predictions in the presence of user feedback. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’09). ACM, New York, NY, 323--330. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Fernando Diaz, Ryen White, Georg Buscher, and Dan Liebling. 2013. Robust models of mouse movement on dynamic web search results pages. In Proceedings of the 22nd ACM International Conference on Information & Knowledge Management (CIKM’13). ACM, New York, NY, 1451--1460. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Sergio Duarte Torres, Djoerd Hiemstra, and Theo Huibers. 2013. Vertical selection in the information domain of children. In Proceedings of the 13th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL’13). ACM, New York, NY, 57--66. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Susan T. Dumais, Georg Buscher, and Edward Cutrell. 2010. Individual differences in gaze patterns for web search. In Proceedings of the 3rd Symposium on Information Interaction in Context (IIiX’10). ACM, New York, NY, 185--194. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Luo Jie, Sudarshan Lamkhede, Rochit Sapra, Evans Hsu, Helen Song, and Yi Chang. 2013. A unified search federation system based on online user feedback. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’13). ACM, New York, NY, 1195--1203. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. 2005. Accurately interpreting clickthrough data as implicit feedback. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’05). ACM, New York, NY, 154--161. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Kurt Koffka. 1935. Principles of Gestalt Psychology. Harcourt, New York.Google ScholarGoogle Scholar
  28. Arnd Christian König, Michael Gamon, and Qiang Wu. 2009. Click-through prediction for news queries. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’09). ACM, New York, NY, 347--354. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics 33, 1 (1977), 159--174.Google ScholarGoogle ScholarCross RefCross Ref
  30. Xiao Li, Ye-Yi Wang, and Alex Acero. 2008. Learning query intent from regularized click graphs. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’08). ACM, New York, NY, 339--346. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Zeyang Liu, Yiqun Liu, Ke Zhou, Min Zhang, and Shaoping Ma. 2015. Influence of vertical result in web search examination. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’15). ACM, New York, NY, 193--202. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Stephen E. Palmer. 1992. Common region: A new principle of perceptual grouping. Cognitive Psychology 24, 3 (1992), 436--447.Google ScholarGoogle ScholarCross RefCross Ref
  33. Bing Pan, Helene Hembrooke, Thorsten Joachims, Lori Lorigo, Geri Gay, and Laura Granka. 2007. In Google we trust: Users’ decisions on rank, position, and relevance. Journal of Computer-Mediated Communication 12, 3 (2007), 801--823.Google ScholarGoogle ScholarCross RefCross Ref
  34. Ashok Kumar Ponnuswami, Kumaresh Pattabiraman, Desmond Brand, and Tapas Kanungo. 2011a. Model characterization curves for federated search using click-logs: Predicting user engagement metrics for the span of feasible operating points. In Proceedings of the 20th International Conference on World Wide Web (WWW’11). ACM, New York, NY, 67--76. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Ashok Kumar Ponnuswami, Kumaresh Pattabiraman, Qiang Wu, Ran Gilad-Bachrach, and Tapas Kanungo. 2011b. On composition of a federated web search result page: Using online users to provide pairwise preference for heterogeneous verticals. In Proceedings of the 4th ACM International Conference on Web Search and Data Mining (WSDM’11). ACM, New York, NY, 715--724. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Filip Radlinski and Susan Dumais. 2006. Improving personalized web search using result diversification. In SIGIR. ACM, 691--692. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Mark Sanderson. 2008. Ambiguous queries: Test collections need more sense. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’08). ACM, New York, NY, 499--506. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Rodrygo L. T. Santos, Craig Macdonald, and Iadh Ounis. 2010. Exploiting query reformulations for web search result diversification. In WWW. ACM, 881--890. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Rodrygo L. T. Santos, Craig Macdonald, and Iadh Ounis. 2011a. Aggregated search result diversification. In ITCIR. Springer-Verlag, 250--261. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Rodrygo L. T. Santos, Craig Macdonald, and Iadh Ounis. 2011b. Aggregated search result diversification. In Proceedings of the 3rd International Conference on Advances in Information Retrieval Theory (ICTIR’11). Springer-Verlag, Berlin, 250--261. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Rodrygo L. T. Santos, Craig Macdonald, and Iadh Ounis. 2015. Search result diversification. Foundations and Trends in Information Retrieval 9, 1 (2015), 1--90. http://dx.doi.org/10.1561/1500000040 Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Mark D. Smucker, James Allan, and Ben Carterette. 2007. A comparison of statistical significance tests for information retrieval evaluation. In Proceedings of the 16th ACM Conference on Conference on Information and Knowledge Management (CIKM’07). ACM, New York, NY, 623--632. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Cheng Xiang Zhai, William W. Cohen, and John Lafferty. 2003. Beyond independent relevance: Methods and evaluation metrics for subtopic retrieval. In SIGIR. ACM, 10--17. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Ke Zhou, Ronan Cummins, Martin Halvey, Mounia Lalmas, and Joemon M. Jose. 2012a. Assessing and predicting vertical intent for web queries. In Proceedings of the 34th European Conference on Advances in Information Retrieval (ECIR’12). Springer-Verlag, Berlin, 499--502. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Ke Zhou, Ronan Cummins, Mounia Lalmas, and Joemon M. Jose. 2012b. Evaluating aggregated search pages. In Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’12). ACM, New York, NY, 115--124. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Ke Zhou, Ronan Cummins, Mounia Lalmas, and Joemon M. Jose. 2012c. Evaluating reward and risk for vertical selection. In Proceedings of the 21st ACM International Conference on Information and Knowledge Management (CIKM’12). ACM, New York, NY, 2631--2634. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. The Effects of Aggregated Search Coherence on Search Behavior

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Information Systems
      ACM Transactions on Information Systems  Volume 35, Issue 1
      January 2017
      233 pages
      ISSN:1046-8188
      EISSN:1558-2868
      DOI:10.1145/2986034
      Issue’s Table of Contents

      Copyright © 2016 ACM

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 22 September 2016
      • Revised: 1 May 2016
      • Accepted: 1 May 2016
      • Received: 1 January 2016
      Published in tois Volume 35, Issue 1

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
      • Research
      • Refereed

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader