Constructing informative prior distributions from domain knowledge in text classification

ABSTRACT

Supervised learning approaches to text classification are in practice often required to work with small and unsystematically collected training sets. The alternative to supervised learning is usually viewed to be building classifiers by hand, using a domain expert's understanding of which features of the text are related to the class of interest. This is expensive, requires a degree of sophistication about linguistics and classification, and makes it difficult to use combinations of weak predictors. We propose instead combining domain knowledge with training examples in a Bayesian framework. Domain knowledge is used to specify a prior distribution for the parameters of a logistic regression model, and labeled training data is used to produce a posterior distribution, whose mode we take as the final classifier. We show on three text categorization data sets that this approach can rescue what would otherwise be disastrously bad training situations, producing much more effective classifiers.

REFERENCES

• 1: Medical Subject Headings -- Home Page, 2005. http://www.nlm.nih.gov/mesh.
• 2: Chris Buckley , Gerard Salton , James Allan, The effect of adding relevance information in a relevance feedback environment, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.292-300, July 03-06, 1994, Dublin, Ireland
• 3: B. Carlin and T. Louis. Bayes and Empirical Bayes Methods for Data Analysis. Chapman & Hall, London, 1996.
• 4: Kian Ming Adam Chai , Hai Leong Chieu , Hwee Tou Ng, Bayesian online classifiers for text classification and filtering, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564395]
• 5: C. Chelba and A. Acero. Adaptation of maximum entropy capitalizer: Little data can help a lot. In EMNLP '04, 2004.
• 6: W. Cohen and D. Kudenko. Transferring and retraining learned information filters. In AAAI/IAAI '97, pages 583--590, 1997.
• 7: A. Dayanik, D. Fradkin, A. Genkin, P. Kantor, D. Lewis, D. Madigan, and V. Menkov. DIMACS at the TREC 2004 genomics track. In TREC '04, 2005.
• 8: Norbert Fuhr , Ulrich Pfeifer, Combining model-oriented and description-oriented approaches for probabilistic indexing, Proceedings of the 14th annual international ACM SIGIR conference on Research and development in information retrieval, p.46-56, October 13-16, 1991, Chicago, Illinois, United States [doi>10.1145/122860.122865]
• 9: E. Gabrilovich and S. Markovitch. Feature generation for text categorization using world knowledge. In IJCAI '05, pages 1048--1053, 2005.
• 10: A. Genkin, D. Lewis, and D. Madigan. Large-scale Bayesian logistic regression for text categorization. Technometrics, 2006. To appear.
• 11: W. Hersh, R. Bhuptiraju, L. Ross, A. Cohen, D. Kraemer, and P. Johnson. TREC 2004 genomics track overview. In TREC '04, 2004.
• 12: Thorsten Joachims, Text Categorization with Suport Vector Machines: Learning with Many Relevant Features, Proceedings of the 10th European Conference on Machine Learning, p.137-142, April 21-23, 1998
• 13: Thorsten Joachims, Learning to Classify Text Using Support Vector Machines: Methods, Theory and Algorithms, Kluwer Academic Publishers, Norwell, MA, 2002
• 14: R. Jones, A. McCallum, K. Nigam, and E. Riloff. Bootstrapping for text learning tasks. In IJCAI '99 Workshop on Text Mining, 1999.
• 15: David D. Lewis , Yiming Yang , Tony G. Rose , Fan Li, RCV1: A New Benchmark Collection for Text Categorization Research, The Journal of Machine Learning Research, 5, p.361-397, 12/1/2004
• 16: F. Li and Y. Yang. A loss function analysis for classification methods in text categorization. In ICML '03, pages 472--479, 2003.
• 17: B. Liu, X. Li, W. Lee, and P. Yu. Text classification by labeling words. In AAAI '04, 2004.
• 18: D. Madigan, J. Gavrin, and A. Raftery. Eliciting prior information to enhance the predictive performance of bayesian graphical models. Communications in Statistics - Theory and Methods, pages 2271--2292, 1995.
• 19: T. Meyer and B. Whateley. SpamBayes: Effective open-source, Bayesian based, email classification system. In CEAS '04, 2004.
• 20: K. Nigam, J. Lafferty, and A. McCallum. Using maximum entropy for text classification. In IJCAI'99 Workshop on Information Filtering, 1999.
• 21: M. Porter. An algorithm for suffix stripping. Program, 14(3):130--137, July 1980.
• 22: H. Raghavan, O. Madani, and R. Jones. Interactive feature selection. In IJCAI '05, pages 841--846, 2005.
• 23: G. Salton, The SMART Retrieval System—Experiments in Automatic Document Processing, Prentice-Hall, Inc., Upper Saddle River, NJ, 1971
• 24: Gerard Salton , Christopher Buckley, Term-weighting approaches in automatic text retrieval, Information Processing and Management: an International Journal, v.24 n.5, p.513-523, 1988 [doi>10.1016/0306-4573(88)90021-0 ]
• 25: Gerard Salton , Michael J. McGill, Introduction to Modern Information Retrieval, McGraw-Hill, Inc., New York, NY, 1986
• 26: Robert E. Schapire , Marie Rochery , Mazin G. Rahim , Narendra Gupta, Incorporating Prior Knowledge into Boosting, Proceedings of the Nineteenth International Conference on Machine Learning, p.538-545, July 08-12, 2002
• 27: Hinrich Schütze , David A. Hull , Jan O. Pedersen, A comparison of classifiers and document representations for the routing problem, Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, p.229-237, July 09-13, 1995, Seattle, Washington, United States [doi>10.1145/215206.215365]
• 28: Fabrizio Sebastiani, Machine learning in automated text categorization, ACM Computing Surveys (CSUR), v.34 n.1, p.1-47, March 2002 [doi>10.1145/505282.505283]
• 29: R. Smith. Bayesian and frequentist approaches to parametric predictive inference (with discussion). In Bayesian Statistics 6. Oxford Univ. Press, 1999.
• 30: R. Tibshirani. Regression shrinkage and selection via the lasso. J. Royal Statistical Soc. B., 58:267--288, 1996.
• 31: C. J. Van Rijsbergen, Information Retrieval, Butterworth-Heinemann, Newton, MA, 1979
• 32: Xiaoyun Wu , Rohini Srihari, Incorporating prior knowledge with weighted margin support vector machines, Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, August 22-25, 2004, Seattle, WA, USA [doi>10.1145/1014052.1014089]
• 33: Jian Zhang , Yiming Yang, Robustness of regularized linear classification methods in text categorization, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860471]
• 34: Tong Zhang , Frank J. Oles, Text Categorization Based on Regularized Linear Classification Methods, Information Retrieval, v.4 n.1, p.5-31, April 2001 [doi>10.1023/A:1011441423217 ]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing

General Terms:
Algorithms, Experimentation

Keywords:
MAP estimation, knowledge-based, maximum entropy