Word document density and relevance scoring (poster session)

ABSTRACT

Previous work addressing the issue of word distribution in documents has shown the importance of Word repetitiveness as an indicator of the word content-bearing characteristics. In this paper we propose a simple method using a measure of the tendency of words to repeat within a document to separate the words with similar document frequencies, but different topic discriminating characteristics. We describe the application of the new measure in query-document relevance scoring. Experiments on the TREC Ad Hoc and Spoken Document Retrieval tasks [7] show useful performance improvements.

REFERENCES

• 1: Doug Beeferman , Adam Berger , John Lafferty, A model of lexical attraction and repulsion, Proceedings of the eighth conference on European chapter of the Association for Computational Linguistics, p.373-380, July 07-12, 1997, Madrid, Spain
• 2: M. Franz, J. S. McCarley, S. Roukos, Ad hoe and MUltilingual Information Retrieval at IBM, in Proceedings of the Seventh Text REtrieval Conference (TREC-7) ed. by E. M. Vorhees and D.K. Harman. NIST Special Publication 500-242: 157-168, 1999.
• 3: M. Franz, J. S. McCarley, R. T. Ward, Ad hoc, Crosslanguage and Spoken Document Information Retrieval at IBM, to apear in Proceedings of the Eighth Text RE- tmeval Conference (TREC-8} ed. by E. M. Vorhees and D.K. Harman.
• 4: Slava M. Katz, Distribution of content words and phrases in text and language modelling, Natural Language Engineering, v.2 n.1, p.15-59, March 1996 [doi>10.1017/S1351324996001246]
• 5: S. E. Robertson, S. Walker, S. Jones, M. M. Hancock- Beaulieu, M. Gatford, Okapi at TREC-3 in Proceedings of the Third Text REtrieval Conference (TREC-3) ed. by D.K. Harman. NIST Special Publication 500-225, 1995.
• 6: R. Rosenfeld, A Maximum Entropy Approach to Adaptive Statistical Language Modeling, in Computer Speech and Language, 10: 187-228, 996.
• 7: E. M. Voorhees, D. Harman, Overview of the Seventh Text Retrieval Conference (TREC-7), in Proceedings of the Seventh Text REtrieval Conference (TREC-7) ed. by E. M. Voorhees and D.K. Harman. NIST Special Publication 500-242: 1-23, 1999.
• 8: Jinxi Xu , W. Bruce Croft, Query expansion using local and global document analysis, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.4-11, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243202]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL

General Terms:
Experimentation, Human Factors, Languages, Measurement, Performance, Theory