Ellen M. Voorhees
ABSTRACT
Given the size of the web, the search engine industry has argued that engines should be evaluated by their ability to retrieve highly relevant pages rather than all possible relevant pages. To explore the role highly relevant documents play in retrieval system evaluation, assessors for the \mbox{TREC-9} web track used a three-point relevance scale and also selected best pages for each topic. The relative effectiveness of runs evaluated by different relevant document sets differed, confirming the hypothesis that different retrieval techniques work better for retrieving highly relevant documents. Yet evaluating by highly relevant documents can be unstable since there are relatively few highly relevant documents. TREC assessors frequently disagreed in their selection of the best page, and subsequent evaluation by best page across different assessors varied widely. The discounted cumulative gain measure introduced by J\"{a}rvelin and Kek\"{a}l\"{a}inen increases evaluation stability by incorporating all relevance judgments while still giving precedence to highly relevant documents.
- 1.Internet Archive. The Internet Archive: Building an 'Internet Library'. http://www.archive.org.Google Scholar
- 2.Pia Borlund and Peter Ingwersen. Measures of relative relevance and ranked half-life: Performance indicators for interactive IR. In W. Bruce Croft, Alistair Moffat, C.J. van Rijsbergen, Ross Wilkinson, and Justin Zobel, editors, Proceedings of the 21st Annual International ACM SIGIR Conference onResearch and Development in Information Retrieval, pages 324-331, Melbourne, Australia, August 1998. ACM Press, New York. Google ScholarDigital Library
- 3.Sergey Brin and Lawrence Page. The anatomy ofa large-scale hypretextual web search engine. In Proceedings of the Seventh International World Wide Web Conference, http://www7.scu.edu.au/ programme/fullpapers/1921/com1921.htm, 1998. Google ScholarDigital Library
- 4.Chris Buckley and Ellen M. Voorhees. Evaluating evaluation measure stability. In N. Belkin, P. Ingwersen, and M.K. Leong, editors, Proceedings of the 23rd Annual International ACM SIGIR Conference onResearch and Development in Information Rertrieval, pages 33-40, 2000. Google ScholarDigital Library
- 5.W.S. Cooper. Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systems. American Documentation, 19:30-41, January 1968.Google ScholarCross Ref
- 6.Michael Gordon and Praveen Pathak. Finding information on the world wide web: the retrieval effectiveness of search engines. Information Processing and Management, 35:141-180, 1999. Google ScholarDigital Library
- 7.David Hawking, Nick Craswell, and Paul Thistlewaite. Overview of TREC-7 very large collection track. In E.M. Voorhees and D.K. Harman, editors, Proceedings of the Seventh Text REtrieval Conference (TREC-7), pages 91-103, August 1999. NIST Special Publication 500-242. Electronic version available at http://trec.nist.gov/pubs.html.Google Scholar
- 8.David Hawking, Nick Craswell, Paul Thistlewaite, and Donna Harman. Results and challenges in web search evaluation. In Proceedings of the Eighth International World Wide Web Conference, http://www8.org/w8-papers/2c-search-discover/ results/results.html, 1999. Google ScholarDigital Library
- 9.David Hawking, Ellen Voorhees, Nick Craswell, and Peter Bailey. Overview of the TREC-8 web track. In E.M. Voorhees and D.K. Harman, editors, Proceedings of the Eighth Text REtrieval Conference (TREC-8), pages 131-150, 2000. NIST Special Publication 500-246. Electronic version available at http://trec.nist.gov/pubs.html.Google Scholar
- 10.Kalervo J. arvelin and Jaana Kek.al. ainen. IR evaluation methods for retrieving highly relevant documents. In Nicholas J. Belkin, Peter Ingwersen, and Mun-Kew Leong, editors, Proceedings of the 23rd Annual International ACM SIGIR Conference onResearch and Development in Information Retrieval, pages 41-48, July 2000. Google ScholarDigital Library
- 11.Chris Sherman. Special report|the 5th annual search engines meeting. http://websearch.about.com/ internet/websearch/library/blsem.htm. Section "The Fireworks Fly".Google Scholar
- 12.Alan Stuart. Kendall's tau. In Samuel Kotz and Norman L. Johnson, editors, Encyclopedia of Statistical Sciences, volume 4, pages 367-369. John Wiley & Sons, 1983.Google Scholar
- 13.Bob Travis and Andrei Broder. The need behind the query: Web search vs classic information retrieval. http://www.infonortics.com/searchengines/sh01/ slides-01/sh01pro.html.Google Scholar
- 14.Ellen M. Voorhees. Variations in relevance judgments and the measurement ofretrieval effectiveness. Information Processing and Management, 36:697-716, 2000. Google ScholarDigital Library
- 15.Ellen M. Voorhees and Donna Harman. Overview of the sixth Text REtrieval Conference (TREC-6). Information Processing and Management, 36(1):3-35, January 2000. Google ScholarDigital Library
Index Terms
- Evaluation by highly relevant documents
Recommendations
IR evaluation methods for retrieving highly relevant documents
SIGIR Test-of-Time Awardees 1978-2001This paper proposes evaluation methods based on the use of non-dichotomous relevance judgements in IR experiments. It is argued that evaluation methods should credit IR methods for their ability to retrieve highly relevant documents. This is desirable ...
IR evaluation methods for retrieving highly relevant documents
SIGIR '00: Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrievalThis paper proposes evaluation methods based on the use of non-dichotomous relevance judgements in IR experiments. It is argued that evaluation methods should credit IR methods for their ability to retrieve highly relevant documents. This is desirable ...
A WEBIR Crawling Framework for Retrieving Highly Relevant Web Documents: Evaluation Based on Rank Aggregation and Result Merging Algorithms
CICN '11: Proceedings of the 2011 International Conference on Computational Intelligence and Communication NetworksFinding relevant information on the web is an ongoing problem. Commercial search engines like Google rely on sophisticated algorithms to index huge collection of web pages to make them accessible to user queries. Users, however, are still frequently ...
Comments