A multi-system analysis of document and term selection for blind feedback

ABSTRACT

Experiments were conducted to explore the impact of combining various components of eight leading information retrieval systems. Each system demonstrated improved effectiveness with the use of <i>blind feedback</i>, in which the results of a preliminary retrieval step were used to augment the efficacy of a secondary retrieval step. The hybrid combination of primary and secondary retrieval steps from different systems in a number of cases yielded better effectiveness than either of the constituent systems alone. This positive combining effect was observed when entire documents were passed between the two retrieval steps, but not when only the expansion terms were passed. Several combinations of primary and secondary retrieval steps were fused using the CombMNZ algorithm; all yielded significant effectiveness improvement over the individual systems, with the best yielding a an improvement of 13% (<i>p</i> = 10^-6) over the best individual system and an improvement of 4% (<i>p</i> = 10^-5) over a simple fusion of the eight systems.

References

1. Gianni Amati and C. J. van Rijsbergen. Probabilistic models of information retrieval based on measuring the divergence from randomness. ACM Tranactions on Information Systems, 20(4):357--389, October 2002.]]
2. Chris Buckley. Implementation of the SMART information retrieval system. Technical Report 85--686, Computer Science Department, Cornell University, Ithaca, New York, May 1985.]]
3. Chris Buckley. Topic prediction based on comparative retrieval rankings. In Proceedings of the 27th annual international conference on Research and development in information retrieval, She.eld, United Kingdom, 2004.]]
4. Chris Buckley. Why current IR engines fail. In Proceedings of the 27th annual international conference on Research and development in information retrieval, She.eld, United Kingdom, 2004.]]
5. Charles L. A. Clarke, Gordon V. Cormack, and Thomas R. Lynam. Exploiting redundancy in question answering. In SIGIR 2001, New Oreans, Louisiana, 2001.]]
6. Kevyn Collins-Thompson, Jamie Callan, Egidio Terra, and Charles L.A. Clarke. The effect of document retrieval quality on factoid question answering performance. In Proceedings of the 27th annual international conference on Research and development in information retrieval, She.eld, United Kingdom, 2004.]]
7. W. Bruce Croft. Combining approaches to information retrieval. In Advances in Information Retrieval: Recent Research from the Center for Intelligent Information Retrieval. Kluwer Academic Publishers, 2000.]]
8. David A. Evans and Robert G.Lefferts. Design and evaluation of the CLARIT-TREC-2 system. In Second Text REtrieval Conference, Gaithersburg, MD, 1994.]]
9. David A. Evans and Robert G.Lefferts. CLARIT-TREC experiments. In Third Text REtrieval Conference, Gaithersburg, MD, 1995.]]
10. E. A. Fox and J. A. Shaw. Combintation of multiple searches. In Second Text REtrieval Conference, 1994.]]
11. Zhenmei Gu and Ming Luo. Comparison of using passages and documents for blind relevance feedback in information retrieval. In Proceedings of the 27th annual international conference on Research and development in information retrieval, Sheffield, United Kingdom, 2004.]]
12. Donna Harman and Chris Buckley. The NRRC reliable information access (RIA) workshop. In Proceedings of the 27th annual international conference on Research and development in information retrieval, Sheffield, United Kingdom, 2004.]]
13. Victor Lavrenko and W. Bruce Croft. Relevance-based language models. In SIGIR 2001, pages 120--127, New Orleans, 2001.]]
14. Lemur. The Lemur toolkit for language modeling and information retrieval. Available at http://www-2.cs.cmu.edu/ lemur/.]]
15. Mark Montague and Javed A Aslam. Condorcet fusion for improved retrieval. In Eleventh International Conference on Information and Knowledge Management (CIKM 2002), pages 538--548, McLean, Virginia, 2002.]]
16. Jesse Montgomery, Luo Si, Jamie Callan, and David A. Evans. E.ect of varying number of documents in blind feedback: analysis of the 2003 NRRC RIA workshop "bf numdocs" experiment suite. In Proceedings of the 27th annual international conference on Research and development in information retrieval, Sheffield, United Kingdom, 2004.]]
17. Okapi. Details of okapi and its functionality. http://www.soi.city.ac.uk/ andym/OKAPI-PACK/.]]
18. Jay M. Ponte and W. Bruce Croft. A language modeling approach to information retrieval. In SIGIR 1998, pages 275--281, Melbourne, Australia, 1998.]]
19. S.E. Robertson. On term selection for query expansion. Journal of Documentation, Vol 46:p359--364, 1990.]]
20. S.E. Robertson and K. Sparck Jones. Relevance weighting of search terms. Journal of the American Society for Information Science, Vol 27(p129--146), May-June 1976.]]
21. S.E. Robertson, S. Walker, S. Jones, M.M. Hancock-Beaulieu, and M. Gatford. Okapi at TREC-3. In Third Text REtrieval Conference, Gaithersburg, MD, 1995.]]
22. Small et al. A Data Driven Approach to Interactive Question Answering (in preparation). AAAI/MIT Press, 2004.]]
23. Strzalkowski et al. Advances in Open-Domain Question Answering (in preparation). Kluwer Academic Publishers, 2004.]]
24. Ellen Voorhees. Overview of the Seventh Text REtrieval Conference (TREC-7). In 7th Text REtrieval Conference, Gaithersburg, MD, 1998.]]
25. Ellen Voorhees and Donna Harman. Overview of the Sixth Text REtrieval Conference (TREC-6). In 6th Text REtrieval Conference, Gaithersburg, MD, 1997.]]
26. Ellen Voorhees and Donna Harman. Overview of the Eighth Text REtrieval Conference (TREC-8). In 8th Text REtrieval Conference, Gaithersburg, MD, 1999.]]
27. Robert H. Warren and Ting Liu. A review of relevance feedback experiments at the 2003 reliable information access (RIA) workshop. In Proceedings of the 27th annual international conference on Research and development in information retrieval, Sheffield, United Kingdom, 2004.]]
28. D. Williamson, R. Williamson, and M. Lesk. The Cornell implementation of the SMART system. In G. Salton, editor, The SMART Retrieval System: Experiments in Automatic Document Processing, chapter 2, pages 43--44. Prentice-Hall, Inc. Englewood Cliffs, New Jersey, 1971.]]
29. David L. Yeung, Charles L. A. Clarke, Gordon V. Cormack, Thomas R. Lynam, and Egidio L. Terra. Task-specific query expansion. Text REtrieval Conference 2003, 2003.]]
30. ChengXiang Zhai and John Lafferty. Model-based-feedback in the language modeling approach to information retrieval. Tenth International Conference on Information and Knowledge Management (CIKM 2001), 2001.]]
31. ChengXiang Zhai and John Lafferty. Two-stage language model for information retrieval. In SIGIR 2002, pages 49--56, Tampere, Finland, 2002.]]
32. ChengXiang Zhai, Xiang Tong, Peter Jansen, and David A. Evans. The CLARIT system TREC-6 report. In Sixth Text REtrieval Conference, Gaithersburg, MD, 1998.]]

Index Terms

• Information systems

  • Information retrieval