Text categorization for multiple users based on semantic features from a machine-readable dictionary

ABSTRACT

The text categorization module described here provides a front-end filtering function for the larger DR-LINK text retrieval system [Liddy and Myaeing 1993]. The model evaluates a large incoming stream of documents to determine which documents are sufficiently similar to a profile at the broad subject level to warrant more refined representation and matching. To accomplish this task, each substantive word in a text is first categorized using a feature set based on the semantic Subject Field Codes (SFCs) assigned to individual word senses in a machine-readable dictionary. When tested on 50 user profiles and 550 megabytes of documents, results indicate that the feature set that is the basis of the text categorization module and the algorithm that establishes the boundary of categories of potentially relevant documents accomplish their tasks with a high level of performance. This means that the category of potentially relevant documents for most profiles would contain at least 80% of all documents later determined to be relevant to the profile. The number of documents in this set would be uniquely determined by the system's category-boundary predictor, and this set is likely to contain less than 5% of the incoming stream of documents.

REFERENCES

• 1: M. J. Blosseville , G. Hébrail , M. G. Monteil , N. Pénot, Automatic document classification: natural language processing, statistical analysis, and expert system techniques used together, Proceedings of the 15th annual international ACM SIGIR conference on Research and development in information retrieval, p.51-58, June 21-24, 1992, Copenhagen, Denmark [doi>10.1145/133160.133175]
• 2: Bran Boguraev , Ted Briscoe, Computational lexicography for natural language processing, Longman Publishing Group, White Plains, NY, 1989
• 3: CHOUE~(A, T. AND LUSIGNAN, S. 1985. Disambiguation by short contexts. Comput. Hum. 19, 3, 147 157.
• 4: HALE, R. L. 1990. MYSTAT Stattstzcal Apphcat*ons. Course Technology, Inc, Cambridge, Mass.
• 5: Donna Harman, Overview of the second text retrieval conference (TREC-2), Proceedings of the second conference on Text retrieval conference, p.271-289, May 1995, Washington, D.C., United States
• 6: Philip J. Hayes , Steven P. Weinstein, CONSTRUE/TIS: A System for Content-Based Indexing of a Database of News Stories, Proceedings of the The Second Conference on Innovative Applications of Artificial Intelligence, p.49-64, May 01-03, 1990
• 7: KELLY, E. F. AND STONE, P.J. 1975. Computer Recognition of English Word Senses. North Holland, Amsterdam.
• 8: KROVETZ, R. 1991. Lexical acquisition and information retrieval. In Lexical Acquisition: Exploiting On-Line Resources to Build a Lexicon, U. Zernik, Ed., Lawrence Earlbaum, Hillsdale, N.J..
• 9: David Dolan Lewis, Representation and learning in information retrieval, University of Massachusetts, Amherst, MA, 1992
• 10: David D. Lewis, Text representation for intelligent text retrieval: a classification-oriented view, Text-based intelligent systems: current research and practice in information extraction and retrieval, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 1992
• 11: LIDDY, E.n. 1994. Development and implementation of a discourse model for newspaper texts. In Proceedings of the Dagstuhl on Summarizing Text for Intelligent Communication (Saarbrilken, Germany). International Conference and Research Center for Computer Science in Schloss Dagstuhl. To be published.
• 12: Elizabeth D. Liddy , Woojin Paik , Edmund S. Yu , Kenneth A. McVearry, An overview of DR-LINK and its approach to document filtering, Proceedings of the workshop on Human Language Technology, March 21-24, 1993, Princeton, New Jersey [doi>10.3115/1075671.1075755]
• 13: LintY, E. D. AND PAIK, W. (1992). Statistically-guided word sense disambiguation. In Proceed- ~ngs of AAAI Fall '92 Symposium on Probabilistic Approaches to Natural Language (Boston, Mass.). AAAI, Menlo Park, Calif.
• 14: Brij Masand , Gordon Linoff , David Waltz, Classifying news stories using memory based reasoning, Proceedings of the 15th annual international ACM SIGIR conference on Research and development in information retrieval, p.59-65, June 21-24, 1992, Copenhagen, Denmark [doi>10.1145/133160.133177]
• 15: McGmL, M., KOLL, M., AND NOREAVLT, T. 1979. An evaluation of factors affecting document ranking by information retrieval systems. Final Report to National Science Foundation. Syracuse Univ., Syracuse, N.Y.
• 16: METEER, M., SCHWARTZ, R., AND WEISCHEDEL, R. 1991. POST: Using probabilities in language processing. In Proceedings of the 12th Internahonal Jotnt Conference on Artificial Intelligence (Sydney, Australia). Morgan Kaufmann, San Mateo, Calif.
• 17: PAIK, W., LmDY, E. D., Yu, E. S., AND MCKENNA, M. 1993. Extracting and classifying proper nouns in documents. In Proceedings of the Human Language Technology Workshop (Princeton, N.J.). ARPA, Washington, D.C.
• 18: SAGER, W. K. H. ~D LOCk--N, P.C. 1976. Classification of ranking algorithms. Int. Forum Inf. Doc. 1, 4, 2-25.
• 19: SLATOR, B. 1991. Using context for sense preference. In Lexical Acquisition: Exploiting On- Line Resources to Build a Lexicon, Zernik, U. Ed, Lawrence Earlbaum, Hillsdale, N.J.
• 20: Steven Lawrence Small , Garrison Weeks Cottrell , Michael Tanennaus, Lexical Ambiguity Resolution: Perspectives from Psycholinguistics, Neuropsychology, and Artificial Intelligence, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1988
• 21: TANIMOTO, T. 1958. An elementary mathematical theory of classification and prediction. Int. Rep., IBM Corp., Watson Research Center, Kingston, N.Y.
• 22: WALKER, D. E. AND AMSLER, R.A. 1986. The use of machine-readable dictionaries in sublang-uage analysis. In Analyzing Language in Restricted Domains: Sublanguage Descriptwn and Processing, R. Grishman and R. Kittredge, Eds., Lawrence Earlbaum, Hillsdale, N.J.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Indexing methods

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Abstracting methods; Linguistic processing
      H.3.3 Information Search and Retrieval
          Subjects: Clustering; Search process

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Text analysis
  I.7 DOCUMENT AND TEXT PROCESSING

General Terms:
Experimentation, Theory

Keywords:
semantic vectors, subject field coding