skip to main content
10.1145/3176349.3176381acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
research-article

Analyzing Knowledge Gain of Users in Informational Search Sessions on the Web

Published: 01 March 2018 Publication History

Abstract

Web search is frequently used by people to acquire new knowledge and to satisfy learning-related objectives, but little is known about how a user»s knowledge evolves through the course of a search session. We present a study addressing the knowledge gain of users in informational search sessions. Using crowdsourcing, we recruited 500 distinct users and orchestrated real-world search sessions spanning 10 different topics and information needs. By using scientifically formulated knowledge tests we calibrated the knowledge of users before and after their search sessions, quantifying their knowledge gain. We investigated the impact of information needs on the search behavior and knowledge gain of users, revealing a significant effect of information need on user queries and navigational patterns, but no direct effect on the knowledge gain. Users on average exhibited a higher knowledge gain through search sessions pertaining to topics they were less familiar with.
Our findings in this paper contribute important ground work towards advancing current research in understanding user knowledge gain through web search sessions.

References

[1]
E. Agichtein, E. Brill, S. Dumais, and R. Ragno. Learning user interaction models for predicting web search result preferences. In Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pages 3--10. ACM, 2006.
[2]
L. W. Anderson, D. R. Krathwohl, P. Airasian, K. Cruikshank, R. Mayer, P. Pintrich, J. Raths, and M. Wittrock. A taxonomy for learning, teaching and assessing: A revision of bloom's taxonomy. New York. Longman Publishing. Artz, AF, & Armour-Thomas, E.(1992). Development of a cognitive-metacognitive framework for protocol analysis of mathematical problem solving in small groups. Cognition and Instruction, 9(2):137--175, 2001.
[3]
J. Arguello. Predicting search task difficulty. In ECIR, volume 14, pages 88--99, 2014.
[4]
A. Broder. A taxonomy of web search. In ACM Sigir forum, volume 36, pages 3--10. ACM, 2002.
[5]
M. P. Carey, D. Morrison-Beedy, and B. T. Johnson. The hiv-knowledge questionnaire: Development and evaluation of a reliable, valid, and practical self-administered questionnaire. AIDS and Behavior, 1(1):61--74, 1997.
[6]
M. J. Cole, J. Gwizdka, C. Liu, N. J. Belkin, and X. Zhang. Inferring user knowledge level from eye movement patterns. Information Processing & Management, 49(5):1075--1091, 2013.
[7]
K. Collins-Thompson, S. Y. Rieh, C. C. Haynes, and R. Syed. Assessing learning outcomes in web search: A comparison of tasks and query strategies. In Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval, pages 163--172. ACM, 2016.
[8]
C. Eickhoff, J. Teevan, R. White, and S. Dumais. Lessons from the journey: a query log analysis of within-session learning. In Proceedings of the 7th ACM International Conference on Web Search & Data Mining. ACM, 2014.
[9]
U. Gadiraju, R. Kawase, S. Dietze, and G. Demartini. Understanding malicious behavior in crowdsourcing platforms: The case of online surveys. In Proceedings of the 33rd Annual ACM CHI Conference, pages 1631--1640. ACM, 2015.
[10]
U. Gadiraju, J. Yang, and A. Bozzon. Clarity is a worthwhile quality--on the role of task clarity in microtask crowdsourcing. In Proceedings of the 28th ACM Conference on Hypertext and Social Media, pages 5--14. ACM, 2017.
[11]
L. A. Granka, T. Joachims, and G. Gay. Eye-tracking analysis of user behavior in www search. In Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, pages 478--479. ACM, 2004.
[12]
J. Gwizdka and X. Chen. Towards observable indicators of learning on search. In SAL@ SIGIR, 2016.
[13]
J. Gwizdka and I. Spence. What can searching behavior tell us about the difficulty of information tasks? a study of web navigation. Proceedings of the Association for Information Science and Technology, 43(1):1--22, 2006.
[14]
M. Hagen, M. Potthast, M. Völske, J. Gomoll, and B. Stein. How writers search: Analyzing the search and writing logs of non-fictional essays. In Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval, pages 193--202. ACM, 2016.
[15]
S. Holm. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics, pages 65--70, 1979.
[16]
B. J. Jansen, D. Booth, and B. Smith. Using the taxonomy of cognitive learning to model online searching. Information Processing & Management, 45(6):643--663, 2009.
[17]
V. Kuperman, H. Stadthagen-Gonzalez, and M. Brysbaert. Age-of-acquisition ratings for 30,000 english words. Behavior Research Methods, 44(4):978--990, 2012.
[18]
Y. Li and N. J. Belkin. A faceted approach to conceptualizing tasks in information seeking. Information Processing & Management, 44(6):1822--1837, 2008.
[19]
R. Syed and K. Collins-Thompson. Retrieval algorithms optimized for human learning. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 555--564. ACM, 2017.
[20]
P. Vakkari. Searching as learning: A systematization based on literature. Journal of Information Science, 42(1):7--18, 2016.
[21]
R. W. White, S. T. Dumais, and J. Teevan. Characterizing the influence of domain expertise on web search behavior. In Proceedings of the second ACM international conference on web search and data mining, pages 132--141. ACM, 2009.
[22]
W.-C. Wu, D. Kelly, A. Edwards, and J. Arguello. Grannies, tanning beds, tattoos and nascar: Evaluation of search tasks with varying levels of cognitive complexity. In Proceedings of the 4th Information Interaction in Context Symposium, pages 254--257. ACM, 2012.
[23]
X. Zhang, M. Cole, and N. Belkin. Predicting users' domain knowledge from search behaviors. In Proceedings of the 34th international ACM SIGIR Conference, pages 1225--1226. ACM, 2011.

Cited By

View all
  • (2024)Everything We Hear: Towards Tackling Misinformation in PodcastsProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3678959(596-601)Online publication date: 4-Nov-2024
  • (2024)The State of Pilot Study Reporting in Crowdsourcing: A Reflection on Best Practices and GuidelinesProceedings of the ACM on Human-Computer Interaction10.1145/36410238:CSCW1(1-45)Online publication date: 26-Apr-2024
  • (2024)The Effects of Goal-setting on Learning Outcomes and Self-Regulated Learning ProcessesProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638348(278-290)Online publication date: 10-Mar-2024
  • Show More Cited By

Index Terms

  1. Analyzing Knowledge Gain of Users in Informational Search Sessions on the Web

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Conferences
        CHIIR '18: Proceedings of the 2018 Conference on Human Information Interaction & Retrieval
        March 2018
        402 pages
        ISBN:9781450349253
        DOI:10.1145/3176349
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

        Sponsors

        In-Cooperation

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 01 March 2018

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. crowdsourcing
        2. information need
        3. knowledge gain
        4. queries
        5. search sessions
        6. user behavior
        7. web search

        Qualifiers

        • Research-article

        Funding Sources

        • European Commission within the H2020-ICT-2015 Programme (AFEL project)

        Conference

        CHIIR '18
        Sponsor:

        Acceptance Rates

        CHIIR '18 Paper Acceptance Rate 22 of 57 submissions, 39%;
        Overall Acceptance Rate 55 of 163 submissions, 34%

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)64
        • Downloads (Last 6 weeks)6
        Reflects downloads up to 19 Feb 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2024)Everything We Hear: Towards Tackling Misinformation in PodcastsProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3678959(596-601)Online publication date: 4-Nov-2024
        • (2024)The State of Pilot Study Reporting in Crowdsourcing: A Reflection on Best Practices and GuidelinesProceedings of the ACM on Human-Computer Interaction10.1145/36410238:CSCW1(1-45)Online publication date: 26-Apr-2024
        • (2024)The Effects of Goal-setting on Learning Outcomes and Self-Regulated Learning ProcessesProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638348(278-290)Online publication date: 10-Mar-2024
        • (2024)On the Effects of Automatically Generated Adjunct Questions for Search as LearningProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638332(266-277)Online publication date: 10-Mar-2024
        • (2024)Towards Self-Contained Answers: Entity-Based Answer Rewriting in Conversational SearchProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638300(209-218)Online publication date: 10-Mar-2024
        • (2024)Disentangling Web Search on Debated Topics: A User-Centered ExplorationProceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3627043.3659559(24-35)Online publication date: 22-Jun-2024
        • (2024)On the Influence of Reading Sequences on Knowledge Gain During Web SearchAdvances in Information Retrieval10.1007/978-3-031-56063-7_28(364-373)Online publication date: 24-Mar-2024
        • (2023)Nudges to Mitigate Confirmation Bias during Web Search on Debated Topics: Support vs. ManipulationACM Transactions on the Web10.1145/363503418:2(1-27)Online publication date: 30-Nov-2023
        • (2023)How search engine marketing influences user knowledge gainProceedings of the 2023 Conference on Human Information Interaction and Retrieval10.1145/3576840.3578297(475-478)Online publication date: 19-Mar-2023
        • (2023)Using Data-Prompted Interviews in Interactive Information Retrieval ResearchProceedings of the 2023 Conference on Human Information Interaction and Retrieval10.1145/3576840.3578285(406-411)Online publication date: 19-Mar-2023
        • Show More Cited By

        View Options

        Login options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Figures

        Tables

        Media

        Share

        Share

        Share this Publication link

        Share on social media