skip to main content
10.1145/1835449.1835494acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
research-article

Using statistical decision theory and relevance models for query-performance prediction

Published: 19 July 2010 Publication History

Abstract

We present a novel framework for the query-performance prediction task. That is, estimating the effectiveness of a search performed in response to a query in lack of relevance judgments. Our approach is based on using statistical decision theory for estimating the utility that a document ranking provides with respect to an information need expressed by the query. To address the uncertainty in inferring the information need, we estimate utility by the expected similarity between the given ranking and those induced by relevance models; the impact of a relevance model is based on its presumed representativeness of the information need. Specific query-performance predictors instantiated from the framework substantially outperform state-of-the-art predictors over five TREC corpora.

References

[1]
N. Abdul-Jaleel, J. Allan, W. B. Croft, F. Diaz, L. Larkey, X. Li, M. D. Smucker, and C. Wade. UMASS at TREC 2004 - novelty and hard. In Proceedings of TREC-13, pages 715--725,2004.
[2]
G. Amati, C. Carpineto, and G. Romano. Query difficulty, robustness and selective application of query expansion. In Proceedings of ECIR, pages 127--137, 2004.
[3]
J. A. Aslam and V. Pavlu. Query hardness estimation using Jensen-Shannon divergence among multiple scoring functions. In Proceeding of ECIR, pages 198--209, 2007.
[4]
D. Carmel, E. Yom-Tov, A. Darlow, and D. Pelleg. What makes a query difficult? In Proceedings of SIGIR, pages 390--397, 2006.
[5]
K. Collins-Thompson and J. Callan. Estimation and use of uncertainty in pseudo-relevance feedback. In Proceedings of SIGIR, pages 303--310, 2007.
[6]
S. Cronen-Townsend, Y. Zhou, and W. B. Croft. Predicting query performance. In Proceedings of SIGIR, pages 299--306, 2002.
[7]
S. Cronen-Townsend, Y. Zhou, and W. B. Croft. A language modeling framework for selective query expansion. Technical Report IR-338, Center for Intelligent Information Retrieval, University of Massachusetts, 2004.
[8]
F. Diaz. Performance prediction using spatial autocorrelation. In Proceedings of SIGIR, pages 583--590, 2007.
[9]
D. Harman and C. Buckley. The NRRC reliable information access (RIA) workshop. In Proceedings of SIGIR, pages 528--529, 2004.
[10]
C. Hauff, L. Azzopardi, and D. Hiemstra. The combination and evaluation of query performance prediction methods. In Proceedings of ECIR, pages 301--312, 2009.
[11]
C. Hauff, D. Hiemstra, and F. de Jong. A survey of pre-retrieval query performance predictors. In Proceedings of CIKM, pages 1419--1420, 2008.
[12]
C. Hauff, V. Murdock, and R. Baeza-Yates. Improved query difficulty prediction for the web. In Proceedings of CIKM, pages 439--448, 2008.
[13]
B. He and I. Ounis. Inferring query performance using pre-retrieval predictors. In Proceedings of SPIRE, pages 43--54, 2004.
[14]
O. Kurland. The opposite of smoothing: A language model approach to ranking query-specific document clusters. In Proceedings of SIGIR, pages 171--178, 2008.
[15]
J. D. Lafferty and C. Zhai. Document language models, query models, and risk minimization for information retrieval. In Proceedings of SIGIR, pages 111--119, 2001.
[16]
V. Lavrenko and W. B. Croft. Relevance-based language models. In Proceedings of SIGIR, pages 120--127, 2001.
[17]
K.-S. Lee, W. B. Croft, and J. Allan. A cluster-based resampling method for pseudo-relevance feedback. In Proceedings of SIGIR, pages 235--242, 2008.
[18]
M. Mitra, A. Singhal, and C. Buckley. Improving automatic query expansion. In Proceedings of SIGIR, pages 206--214, 1998.
[19]
J. Mothe and L. Tanguy. Linguistic features to predict query difficulty. In ACM SIGIR 2005 Workshop on Predicting Query Difficulty - Methods and Applications, 2005.
[20]
S. E. Robertson. The probability ranking principle in IR. Journal of Documentation, pages 294--304, 1977.
[21]
F. Scholer, H. E. Williams, and A. Turpin. Query association surrogates for web search. Journal of the American Society for Information Science and Technology (JASIST), 55(7):637--650, 2004.
[22]
A. Shtok, O. Kurland, and D. Carmel. Predicting query performance by query-drift estimation. In Proceedings of ICTIR, pages 305--312, 2009.
[23]
F. Song and W. B. Croft. A general language model for information retrieval (poster abstract). In Proceedings of SIGIR, pages 279--280, 1999.
[24]
N. Soskin, O. Kurland, and C. Domshlak. Navigating in the dark: Modeling uncertainty in ad hoc retrieval using multiple relevance models. In Proceedings of ICTIR, pages 79--91, 2009.
[25]
S. Tomlinson. Robust, Web and Terabyte Retrieval with Hummingbird Search Server at TREC 2004. In Proceedings of TREC-13, 2004.
[26]
V. Vinay, I. J. Cox, N. Milic-Frayling, and K. R. Wood. On ranking the effectiveness of searches. In Proceedings of SIGIR, pages 398--404, 2006.
[27]
E. M. Voorhees. Overview of the TREC 2004 Robust Retrieval Track. In Proceedings of TREC-13, 2004.
[28]
M. Winaver, O. Kurland, and C. Domshlak. Towards robust query expansion: Model selection in the language model framework to retrieval. In Proceedings of SIGIR, pages 729--730, 2007.
[29]
E. Yom-Tov, S. Fine, D. Carmel, and A. Darlow. Learning to estimate query difficulty: including applications to missing content detection and distributed information retrieval. In Proceedings of SIGIR, pages 512--519, 2005.
[30]
C. Zhai and J. D. Lafferty. A study of smoothing methods for language models applied to ad hoc information retrieval. In Proceedings of SIGIR, pages 334--342, 2001.
[31]
Y. Zhao, F. Scholer, and Y. Tsegay. Effective pre-retrieval query performance prediction using similarity and variability evidence. In ECIR, pages 52--64, 2008.
[32]
Y. Zhou. Retrieval Performance Prediction and Document Quality. PhD thesis, University of Massachusetts, September 2007.
[33]
Y. Zhou and W. B. Croft. Ranking robustness: a novel framework to predict query performance. In Proceedgins of CIKM, pages 567--574, 2006.
[34]
Y. Zhou and W. B. Croft. Query performance prediction in web search environments. In Proceedings of SIGIR, pages 543--550, 2007.

Cited By

View all
  • (2025)Robust query performance prediction for dense retrievers via adaptive disturbance generationMachine Learning10.1007/s10994-024-06659-z114:3Online publication date: 6-Feb-2025
  • (2024)Coherence-based Query Performance Measures for Dense RetrievalProceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3664190.3672518(15-24)Online publication date: 2-Aug-2024
  • (2024)Estimating Query Performance Through Rich Contextualized Query RepresentationsAdvances in Information Retrieval10.1007/978-3-031-56066-8_6(49-58)Online publication date: 15-Mar-2024
  • Show More Cited By

Index Terms

  1. Using statistical decision theory and relevance models for query-performance prediction

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    SIGIR '10: Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval
    July 2010
    944 pages
    ISBN:9781450301534
    DOI:10.1145/1835449
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 19 July 2010

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. query-performance prediction
    2. rank correlation
    3. relevance models
    4. statistical decision theory

    Qualifiers

    • Research-article

    Conference

    SIGIR '10
    Sponsor:

    Acceptance Rates

    SIGIR '10 Paper Acceptance Rate 87 of 520 submissions, 17%;
    Overall Acceptance Rate 792 of 3,983 submissions, 20%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)40
    • Downloads (Last 6 weeks)4
    Reflects downloads up to 15 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)Robust query performance prediction for dense retrievers via adaptive disturbance generationMachine Learning10.1007/s10994-024-06659-z114:3Online publication date: 6-Feb-2025
    • (2024)Coherence-based Query Performance Measures for Dense RetrievalProceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3664190.3672518(15-24)Online publication date: 2-Aug-2024
    • (2024)Estimating Query Performance Through Rich Contextualized Query RepresentationsAdvances in Information Retrieval10.1007/978-3-031-56066-8_6(49-58)Online publication date: 15-Mar-2024
    • (2024)A Deep Learning Approach for Selective Relevance FeedbackAdvances in Information Retrieval10.1007/978-3-031-56060-6_13(189-204)Online publication date: 16-Mar-2024
    • (2023)Noisy Perturbations for Estimating Query Difficulty in Dense RetrieversProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615270(3722-3727)Online publication date: 21-Oct-2023
    • (2023)Towards Query Performance Prediction for Neural Information Retrieval: Challenges and OpportunitiesProceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3578337.3605142(51-63)Online publication date: 9-Aug-2023
    • (2023)Unsupervised Query Performance Prediction for Neural Models with Pairwise Rank PreferencesProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3592082(2486-2490)Online publication date: 19-Jul-2023
    • (2023)iQPP: A Benchmark for Image Query Performance PredictionProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591901(2953-2963)Online publication date: 19-Jul-2023
    • (2023)A Geometric Framework for Query Performance Prediction in Conversational SearchProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591625(1355-1365)Online publication date: 19-Jul-2023
    • (2023)Query Performance Prediction for Neural IR: Are We There Yet?Advances in Information Retrieval10.1007/978-3-031-28244-7_15(232-248)Online publication date: 17-Mar-2023
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media