A geometric interpretation and analysis of R-precision

ABSTRACT

Average precision and R-precision are two of the most commonly cited measures of overall retrieval performance, but their correlation, though well-known, has defied explanation. We recently devised a geometric interpretation of R-precision which suggests that under a reasonable set of assumptions, R-precision approximates the area under the precision-recall curve, as does average precision, thus explaining their correlation. In this paper, we consider these assumptions and our geometric interpretation of R-precision in order to further understand, and make reasonable use of, the information that R-precision provides. Given our geometric interpretation of R-precision, we show that R-precision is highly informative by demonstrating that it can be used to (1) accurately infer precision-recall curves, (2) accurately infer other measures of retrieval performance, and (3) devise new measures of retrieval performance. Through our analysis, we also state the conditions under which R-precision is informative.

REFERENCES

• 1: J. Allan. HARD track overview in TREC 2003: High accuracy retrieval from documents. In Proceedings of the Twelfth Text REtrieval Conference (TREC 2003), pages 24--37, 2003.
• 2: J. A. Aslam, V. Pavlu, and E. Yilmaz. A sampling technique for efficiently estimating measures of query retrieval performance using incomplete judgments. In Proceedings of the 22nd ICML Workshop on Learning with Partially Classified Training Data, pages 57--66, August 2005.
• 3: Javed A. Aslam , Emine Yilmaz , Virgiliu Pavlu, A geometric interpretation of r-precision and its correlation with average precision, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076134]
• 4: Ricardo A. Baeza-Yates , Berthier Ribeiro-Neto, Modern Information Retrieval, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1999
• 5: Chris Buckley , Ellen M. Voorhees, Evaluating evaluation measure stability, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.33-40, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345543]
• 6: William S. Cooper, On selecting a measure of retrieval effectiveness. Part I., Readings in information retrieval, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1997
• 7: B. Dervin and M. S. Nilan. Information needs and use. In Annual Review of Information Science and Technology, volume 21, pages 3--33, 1986.
• 8: Yuri Kagolovsky , Jochen R. Moehr, Current Status of the Evaluation of Information Retrieval, Journal of Medical Systems, v.27 n.5, p.409-424, October 2003 [doi>10.1023/A:1025603704680]
• 9: Robert M. Losee, Text retrieval and filtering: analytic models of performance, Kluwer Academic Publishers, Norwell, MA, 1998
• 10: Robert M. Losee, When information retrieval measures agree about the relative quality of document rankings, Journal of the American Society for Information Science, v.51 n.9, p.834-840, July 2000 [doi>10.1002/(SICI)1097-4571(2000)51:9<834::AID-ASI60>3.3.CO;2-T]
• 11: Vijay Raghavan , Peter Bollmann , Gwang S. Jung, A critical investigation of recall and precision as measures of retrieval system performance, ACM Transactions on Information Systems (TOIS), v.7 n.3, p.205-229, July 1989 [doi>10.1145/65943.65945]
• 12: J. Tague-Sutcliffe and J. Blustein. A statistical analysis of the TREC-3 data. In Proceedings of the Third Text REtrieval Conference (TREC-3), pages 385--398, 1995.
• 13: E. M. Voorhees and D. Harman. Overview of the seventh text retrieval conference (TREC-7). In Proceedings of the Seventh Text REtrieval Conference (TREC-7), pages 1--24, 1999.
• 14: E. M. Voorhees and D. Harman. Overview of the eighth text retrieval conference (TREC-8). In Proceedings of the Eighth Text REtrieval Conference (TREC-8), pages 1--24, 2000.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

General Terms:
Experimentation, Measurement, Theory

Keywords:
R-precision, average precision, evaluation