Understanding temporal aspects in document classification

ABSTRACT

Due to the increasing amount of information present on the Web, Automatic Document Classification (ADC) has become an important research topic. ADC usually follows a standard supervised learning strategy, where we first build a model using preclassified documents and then use it to classify new unseen documents. One major challenge for ADC in many scenarios is that the characteristics of the documents and the classes to which they belong may change over time. However, most of the current techniques for ADC are applied without taking into account the temporal evolution of the collection of documents

In this work, we perform a detailed study of the temporal evolution in the ADC, introducing an analysis methodology. We discuss that temporal evolution may be explained by three factors: 1) class distribution; 2) term distribution; and 3) class similarity. We employ metrics and experimental strategies capable of isolating each of these factors in order to analyze them separately, using two very different document collections: the ACM Digital Library and the Medline medical collections. Moreover, we present some preliminary results of potential gains that could be obtained by varying the training set to find the ideal size that minimizes the time effects. We show that by using just 69% of the ACM database, we are able to have an accuracy of 89.76%, and with only 25% of the Medline, an accuracy of 87.57%, which means gains of up to 20% in accuracy with much smaller training sets

REFERENCES

• 1: James Allan, Incremental relevance feedback for information filtering, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.270-278, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243274]
• 2: J. Allan, J. Carbonell, G. Doddington, J. Yamron, and Y. Yang. Topic detection and tracking pilot study final report. In Proceedings of the DARPA Broadcast News Transcription and Understanding Workshop, 1998.
• 3: Nicholas J. Belkin , W. Bruce Croft, Information filtering and information retrieval: two sides of the same coin?, Communications of the ACM, v.35 n.12, p.29-38, Dec. 1992 [doi>10.1145/138859.138861]
• 4: Harold Borko , Myrna Bernick, Automatic Document Classification, Journal of the ACM (JACM), v.10 n.2, p.151-162, April 1963 [doi>10.1145/321160.321165]
• 5: L. Brieman and P. Spector. Submodel selection and evaluation in regression: The x-random case. International Statistical Review, 60:291--319, 1992.
• 6: Nicholas H. M. Caldwell , P. John Clarkson , Paul A. Rodgers , Avon P. Huxor, Web-Based Knowledge Management for Distributed Design, IEEE Intelligent Systems, v.15 n.3, p.40-47, May 2000 [doi>10.1109/5254.846284 ]
• 7: William W. Cohen , Yoram Singer, Context-sensitive learning methods for text categorization, ACM Transactions on Information Systems (TOIS), v.17 n.2, p.141-173, April 1999 [doi>10.1145/306686.306688]
• 8: Susan Dumais , John Platt , David Heckerman , Mehran Sahami, Inductive learning algorithms and representations for text categorization, Proceedings of the seventh international conference on Information and knowledge management, p.148-155, November 02-07, 1998, Bethesda, Maryland, United States [doi>10.1145/288627.288651]
• 9: George Forman, An extensive empirical study of feature selection metrics for text classification, The Journal of Machine Learning Research, 3, 3/1/2003
• 10: George Forman, Tackling concept drift by temporal inductive transfer, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148216]
• 11: S. Haykin. Adaptive filters. In Signal Processing Magazine. IEEE Computer Society, 1999.
• 12: Thorsten Joachims, Making large-scale support vector machine learning practical, Advances in kernel methods: support vector learning, MIT Press, Cambridge, MA, 1999
• 13: Thorsten Joachims, Training linear SVMs in linear time, Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, August 20-23, 2006, Philadelphia, PA, USA [doi>10.1145/1150402.1150429]
• 14: Rosie Jones , Fernando Diaz, Temporal profiles of queries, ACM Transactions on Information Systems (TOIS), v.25 n.3, p.14-es, July 2007 [doi>10.1145/1247715.1247720]
• 15: R. Klinkenberg and I. Renz. Adaptive information filtering: Learning in the presence of concept drifts. In Learning for Text Categorization, pages 33--40, Menlo Park, California, USA, 1998. AAAI Press.
• 16: Carsten Lanquillon , Ingrid Renz, Adaptive information filtering: detecting changes in text streams, Proceedings of the eighth international conference on Information and knowledge management, p.538-544, November 02-06, 1999, Kansas City, Missouri, United States [doi>10.1145/319950.320061]
• 17: Steve Lawrence , C. Lee Giles, Context and Page Analysis for Improved Web Search, IEEE Internet Computing, v.2 n.4, p.38-46, July 1998 [doi>10.1109/4236.707689 ]
• 18: David D. Lewis, Evaluating and optimizing autonomous text classification systems, Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, p.246-254, July 09-13, 1995, Seattle, Washington, United States [doi>10.1145/215206.215366]
• 19: Rey-Long Liu , Yun-Ling Lu, Incremental context mining for adaptive document classification, Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, July 23-26, 2002, Edmonton, Alberta, Canada [doi>10.1145/775047.775136]
• 20: Gerard Salton , Michael J. McGill, Introduction to Modern Information Retrieval, McGraw-Hill, Inc., New York, NY, 1986
• 21: Martin Scholz , Ralf Klinkenberg, Boosting classifiers for drifting concepts, Intelligent Data Analysis, v.11 n.1, p.3-28, January 2007
• 22: Fabrizio Sebastiani, Machine learning in automated text categorization, ACM Computing Surveys (CSUR), v.34 n.1, p.1-47, March 2002 [doi>10.1145/505282.505283]
• 23: Yiming Yang , Bryan Kisiel, Margin-based local regression for adaptive filtering, Proceedings of the twelfth international conference on Information and knowledge management, November 03-08, 2003, New Orleans, LA, USA [doi>10.1145/956863.956902]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

Additional Classification:
  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology
          Subjects: Pattern analysis

General Terms:
Algorithms, Experimentation

Keywords:
digital libraries, temporal analysis, text classification