The role of knowledge in conceptual retrieval: a study in the domain of clinical medicine

ABSTRACT

Despite its intuitive appeal, the hypothesis that retrieval at the level of "concepts" should outperform purely term-based approaches remains unverified empirically. In addition, the use of "knowledge" has not consistently resulted in performance gains. After identifying possible reasons for previous negative results, we present a novel framework for "conceptual retrieval" that articulates the types of knowledge that are important for information seeking. We instantiate this general framework in the domain of clinical medicine based on the principles of evidence-based medicine (EBM). Experiments show that an EBM-based scoring algorithm dramatically outperforms a state-of-the-art baseline that employs only term statistics. Ablation studies further yield a better understanding of the performance contributions of different components. Finally, we discuss how other domains can benefit from knowledge-based approaches.

REFERENCES

• 1: Gianni Amati , Cornelis Joost Van Rijsbergen, Probabilistic models of information retrieval based on measuring the divergence from randomness, ACM Transactions on Information Systems (TOIS), v.20 n.4, p.357-389, October 2002 [doi>10.1145/582415.582416]
• 2: A. Aronson. Effective mapping of biomedical text to the UMLS Metathesaurus: The MetaMap program. In AMIA 2001.
• 3: N. Belkin. Anomalous states of knowledge as a basis for information retrieval. Canadian Journal of Information Science, 5:133--143, 1980.
• 4: C. Buckley and D. Harman. Reliable information access final workshop report, 2004.
• 5: M. Chambliss and J. Conley. Answering clinical questions. The Journal of Family Practice, 43:140--144, 1996.
• 6: J. Chu-Carroll, J. Prager, C. Welty, K. Czuba, and D. Ferrucci. A multi-strategy and multi-source approach to question answering. In TREC 2002.
• 7: K. Cogdill and M. Moore. First-year medical students' information needs and resource selection: Responses to a clinical scenario. Bulletin of the Medical Library Association, 85(1):51--54, 1997.
• 8: D. Covell, G. Uman, and P. Manning. Information needs in office practice: Are they being met? Annals of Internal Medicine, 103(4):596--599, 1985.
• 9: Hang Cui , Renxu Sun , Keya Li , Min-Yen Kan , Tat-Seng Chua, Question answering passage retrieval using dependency relations, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076103]
• 10: S. De Groote and J. Dorsch. Measuring use patterns of online journals and databases. Journal of the Medical Library Association, 91(2):231--241, 2003.
• 11: D. Demner-Fushman and J. Lin. Knowledge extraction for clinical question answering: Preliminary results. In Proc. of the AAAI 2005 Workshop on Question Answering in Restricted Domains.
• 12: D. Demner-Fushman and J. Lin. Answering clinical questions with knowledge-based and statistical techniques. Computational Linguistics, 2006, in press.
• 13: J. Ely, J. Osheroff, M. Ebell, G. Bergus, B. Levy, M. Chambliss, and E. Evans. Analysis of questions asked by family doctors regarding patient care. BMJ, 319:358--361, 1999.
• 14: J. Fagan. Experiments in Automatic Phrase Indexing for Document Retrieval: A Comparisons of Syntactic and Non-Syntactic Methods. Ph.D., Cornell, 1987.
• 15: Luanne Freund , Elaine G. Toms , Charles L.A. Clarke, Modeling task-genre relationships for IR in the workplace, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076110]
• 16: Jianfeng Gao , Jian-Yun Nie , Guangyuan Wu , Guihong Cao, Dependence language model for information retrieval, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009024]
• 17: P. Gorman, J. Ash, and L. Wykoff. Can primary care physicians' questions be answered using the medical journal literature? Bulletin of the Medical Library Association, 82(2):140--146, 1994.
• 18: W. Hersh, A. Cohen, J. Yang, R. Bhupatiraju, P. Roberts, and M. Hearst. TREC 2005 genomics track overview. In TREC 2005.
• 19: L. Hirschman , R. Gaizauskas, Natural language question answering: the view from here, Natural Language Engineering, v.7 n.4, p.275-300, December 2001 [doi>10.1017/S1351324901002807]
• 20: P. Ingwersen. Cognitive information retrieval. ARIST, 34:3--52, 1999.
• 21: Noriko Kando , Mun-Kew Leong, Workshop on patent retrieval SIGIR 2000 workshop report, ACM SIGIR Forum, v.34 n.1, p.28-30, April 2000 [doi>10.1145/373593.373621]
• 22: Sang-Bum Kim , Hee-Cheol Seo , Hae-Chang Rim, Information retrieval using word senses: root sense tagging approach, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009038]
• 23: Douglas B. Lenat, CYC: a large-scale investment in knowledge infrastructure, Communications of the ACM, v.38 n.11, p.33-38, Nov. 1995 [doi>10.1145/219717.219745]
• 24: D. Lindberg, B. Humphreys, and A. McCray. The Unified Medical Language System. Methods of Information in Medicine, 32(4):281--291, 1993.
• 25: Donald Metzler , W. Bruce Croft, A Markov random field model for term dependencies, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076115]
• 26: Donald Metzler , W. Bruce Croft, Combining the language model and inference network approaches to retrieval, Information Processing and Management: an International Journal, v.40 n.5, p.735-750, September 2004 [doi>10.1016/j.ipm.2004.05.001]
• 27: Rada Mihalcea , Dan Moldovan, Semantic indexing using WordNet senses, Proceedings of the ACL-2000 workshop on Recent advances in natural language processing and information retrieval: held in conjunction with the 38th Annual Meeting of the Association for Computational Linguistics, October 08-08, 2000, Hong Kong [doi>10.3115/1117755.1117760]
• 28: Dan Moldovan , Marius Paşca , Sanda Harabagiu , Mihai Surdeanu, Performance issues and error analysis in an open-domain Question Answering system, Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, July 07-12, 2002, Philadelphia, Pennsylvania [doi>10.3115/1073083.1073091]
• 29: S. Narayanan and S. Harabagiu. Question answering based on semantic structures. In COLING 2004.
• 30: Jay M. Ponte , W. Bruce Croft, A language modeling approach to information retrieval, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.275-281, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291008]
• 31: W. Richardson, M. Wilson, J. Nishikawa, and R. Hayward. The well-built clinical question: A key to evidence-based decisions American College of Physicians Journal Club, 123(3):A12--A13, 1995.
• 32: Thomas C. Rindflesh , Marcelo Fiszman, The interaction of domain knowledge and linguistic structure in natural language processing: interpreting hypernymic propositions in biomedical text, Journal of Biomedical Informatics, v.36 n.6, p.462-477, December 2003 [doi>10.1016/j.jbi.2003.11.003]
• 33: S. Robertson, S. Walker, S. Jones, M. Hancock-Beaulieu, and M. Gatford. Okapi at TREC-3. In TREC-3, 1994.
• 34: D. Sackett, S. Straus, W. Richardson, W. Rosenberg, and R. Haynes. Evidence-Based Medicine: How to Practice and Teach EBM. Churchill Livingstone, second edition, 2000.
• 35: G. Salton , A. Wong , C. S. Yang, A vector space model for automatic indexing, Communications of the ACM, v.18 n.11, p.613-620, Nov. 1975 [doi>10.1145/361219.361220]
• 36: Mark Sanderson, Word sense disambiguation and information retrieval, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.142-151, July 03-06, 1994, Dublin, Ireland
• 37: A. Smeaton, R. O'Donnell, and F. Kelledy. Indexing structures derived from syntax in TREC-3: System description. In TREC-3, 1994.
• 38: Ellen M. Voorhees, Query expansion using lexical-semantic relations, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.61-69, July 03-06, 1994, Dublin, Ireland
• 39: Ellen M. Voorhees, Using WordNet to disambiguate word senses for text retrieval, Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval, p.171-180, June 27-July 01, 1993, Pittsburgh, Pennsylvania, United States [doi>10.1145/160688.160715]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

General Terms:
Experimentation, Measurement

Keywords:
question answering, reranking, semantic models