Statistical precision of information retrieval evaluation

ABSTRACT

We introduce and validate bootstrap techniques to compute confidence intervals that quantify the effect of test-collection variability on average precision (AP) and mean average precision (MAP) IR effectiveness measures. We consider the test collection in IR evaluation to be a representative of a population of materially similar collections, whose documents are drawn from an infinite pool with similar characteristics. Our model accurately predicts the degree of concordance between system results on randomly selected halves of the TREC-6 ad hoc corpus. We advance a framework for statistical evaluation that uses the same general framework to model other sources of chance variation as a source of input for meta-analysis techniques.

References

1. Buckley, C., and Voorhees, E. M. Evaluating evaluation measure stability. In SIGIR Conference 2000 (Athens, Greece, 2000).
2. Cormack, G. V., Palmer, C. R., and Clarke, C. L. A. Efficient construction of large test collections. In SIGIR Conference 1998 (Melbourne, Australia, 1998).
3. Efron, B., and Tsibirani, R. J. An Introduction to the Bootstrap. Chapman and Hall, New York, 1994.
4. Fisher, R. A. Theory of statistical estimation. Proceedings of the Cambridge Philosophical Society 22 (1925), 700--725.
5. Glass, G. V. Meta-analysis at 25. http://glass.ed.asu.edu/gene/papers/meta25.html, 2000.
6. Hull, D. A. Using statistical testing in the evaluation of retrieval experiments. In Research and Development in Information Retrieval (1993), pp. 329--338.
7. Lenhard, J. Models and statistical inference: The controversy between Fisher and Neyman-Pearson. British Journal for the Philosophy of Science (2006).
8. Rothman, K. J., and Greenland, S. Modern Epidemiology. Lippincott Williams & Wilkins, 1998.
9. Sanderson, M., and Johno, H. Test collections with no system pooling. In SIGIR Conference 2004 (Sheffield, UK, 2004).
10. Sanderson, M., and Zobel, J. Information retrieval evaluation: Effort, sensitivity, and reliability. In SIGIR Conference 2005 (Salvador, Brazil, 2005).
11. Savoy, J. Statistical inference in retrieval effectiveness evaluation. Information Processing and Management 33, 4 (1997), 495--512.
12. Tague-Sutcliffe, J. The pragmatics of information retrieval experimentation, revisited. Information Processing and Management 28, 4 (1992), 467--490.
13. Tague-Sutcliffe, J., and Blustein, J. A statistical analysis of the TREC-3 data. In Proceedings of TREC-3, The Third Information Retrieval Conference (1994), pp. 385--398.
14. Voorhees, E., and Harman, D. Overview of the Sixth Text REtrieval Conference (TREC-6). In 6th Text REtrieval Conference (Gaithersburg, MD, 1997).
15. Voorhees, E. M. Variations in relevance judgements and the measurement of retrieval effectiveness. In SIGIR Conference 1998 (Melbourne, Australia, 1998).
16. Voorhees, E. M. Overview of the TREC-2004 robust track. In 13th Text REtrieval Conference (Gaithersburg, MD, 2004).
17. Voorhees, E. M., and Buckley, C. The effect of topic set size on retrieval experiment error. In SIGIR Conference 2002 (Tampere, Finland, 2002).
18. Voorhees, E. M., and Buckley, C. Retrieval evaluation with incomplete information. In SIGIR Conference 2004 (Sheffield, UK, 2004).
19. Voorhees, E. M., and Harman, D. K., Eds. TREC - Experiment and Evaluation in Information Retrieval. MIT Press, 2005.
20. Zobel, J. How reliable are the results of large-scale information retrieval experiments? In SIGIR Conference 1998 (Melbourne, Australia, 1998).

Index Terms

• Information systems

  • Information retrieval

    • Evaluation of retrieval results