The TREC 2005 robust track

ABSTRACT

The TREC robust retrieval track explores methods for improving the consistency of retrieval technology by focusing on poorly performing topics. The retrieval task in the track is a traditional ad hoc retrieval task where the evaluation methodology emphasizes a system's least effective topics. The 2005 edition of the track used 50 topics that had been demonstrated to be difficult on one document collection, and ran those topics on a different document collection. Relevance information from the first collection could be exploited in producing a query for the second collection, if desired. As in previous years, the most effective retrieval strategy was to expand queries using terms derived from additional corpora. The relative difficulty of topics differed across the two document sets.

REFERENCES

• 1: Chris Buckley. Looking at limits and tradeoffs: Sabir Research at TREC 2005. In Proceedings of the Fourteenth Text REtrieval Conference (TREC 2005), 2006. http://trec.nist.gov/pubs/trec14/papers/sabir.tera.robust.qa.pdf.
• 2: David Hawking , Stephen Robertson, On Collection Size and Retrieval Effectiveness, Information Retrieval, v.6 n.1, p.99-105, January 2003 [doi>10.1023/A:1022904715765 ]
• 3: Ellen M. Voorhees. Overview of the TREC 2004 robust retrieval track. In Proceedings of the Thirteenth Text REtrieval Conference, TREC 2004, pages 70--79, 2005.
• 4: Elad Yom-Tov, David Carmel, Adam Darlow, Dan Pelleg, Shai Errera-Yaakov, and Shai Fine. Juru at TREC 2005: Query prediction in the terabyte and the robust tracks. In Proceedings of the Fourteenth Text REtrieval Conference (TREC 2005), 2006. http://trec.nist.gov/pubs/trec14/papers/ibm-haifa.tera.robust.pdf.
• 5: Justin Zobel, How reliable are the results of large-scale information retrieval experiments?, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.307-314, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291014]

INDEX TERMS

Primary Classification:
  A. General Literature
  A.0 GENERAL

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)