New event detection based on indexing-tree and named entity

ABSTRACT

New Event Detection (NED) aims at detecting from one or multiple streams of news stories that which one is reported on a new event (i.e. not reported previously). With the overwhelming volume of news available today, there is an increasing need for a NED system which is able to detect new events more efficiently and accurately. In this paper we propose a new NED model to speed up the NED task by using news indexing-tree dynamically. Moreover, based on the observation that terms of different types have different effects for NED task, two term reweighting approaches are proposed to improve NED accuracy. In the first approach, we propose to adjust term weights dynamically based on previous story clusters and in the second approach, we propose to employ statistics on training data to learn the named entity reweighting model for each class of stories. Experimental results on two Linguistic Data Consortium (LDC) datasets TDT2 and TDT3 show that the proposed model can improve both efficiency and accuracy of NED task significantly, compared to the baseline system and other existing systems.

REFERENCES

• 1: http://www.nist.gov/speech/tests/tdt/index.htm
• 2: James Allan, Topic Detection and Tracking: Event-Based Information Organization, Kluwer Academic Publishers, Norwell, MA, 2002
• 3: Yiming Yang , Jaime G. Carbonell , Ralf D. Brown , Thomas Pierce , Brian T. Archibald , Xin Liu, Learning Approaches for Detecting and Tracking News Events, IEEE Intelligent Systems, v.14 n.4, p.32-43, July 1999 [doi>10.1109/5254.784083 ]
• 4: Yiming Yang , Tom Pierce , Jaime Carbonell, A study of retrospective and on-line event detection, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.28-36, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.290953]
• 5: J. Allan, V. Lavrenko, D. Malin, and R. Swan. Detections, Bounds, and Timelines: Umass and tdt-3. In Proceedings of Topic Detection and Tracking Workshop (TDT-3), Vienna, VA, 2000, 167--174.
• 6: R. Papka , J. Allan, On-Line New Event Detection using Single Pass Clustering TITLE2:, University of Massachusetts, Amherst, MA, 1998
• 7: W. Lam, H. Meng, K. Wong, and J. Yen. Using Contextual Analysis for News Event Detection. International Journal on Intelligent Systems, 2001, 525--546.
• 8: Thorsten Brants , Francine Chen , Ayman Farahat, A System for new event detection, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860495]
• 9: Nicola Stokes , Joe Carthy, Combining semantic and syntactic document classifiers to improve first story detection, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.424-425, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.384068]
• 10: Yiming Yang , Jian Zhang , Jaime Carbonell , Chun Jin, Topic-conditioned novelty detection, Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, July 23-26, 2002, Edmonton, Alberta, Canada [doi>10.1145/775047.775150]
• 11: M. Juha, A. M. Helena, and S. Marko. Applying Semantic Classes in Event Detection and Tracking. In Proceedings of International Conference on Natural Language Processing (ICON 2002), 2002, pages 175--183.
• 12: Juha Makkonen , Helena Ahonen-Myka , Marko Salmenkivi, Simple Semantics in Topic Detection and Tracking, Information Retrieval, v.7 n.3-4, p.347-368, September-December 2004
• 13: Giridhar Kumaran , James Allan, Text classification and named entities for new event detection, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009044]
• 14: J. P. Callan, W. B. Croft, and S. M. Harding. The INQUERY Retrieval System. In Proceedings of DEXA-92, 3rd International Conference on Database and Expert Systems Applications, 1992, 78--83.
• 15: Robert Krovetz, Viewing morphology as an inference process, Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval, p.191-202, June 27-July 01, 1993, Pittsburgh, Pennsylvania, United States [doi>10.1145/160688.160718]
• 16: Yiming Yang , Jan O. Pedersen, A Comparative Study on Feature Selection in Text Categorization, Proceedings of the Fourteenth International Conference on Machine Learning, p.412-420, July 08-12, 1997
• 17: Thomas M. Cover , Joy A. Thomas, Elements of information theory, Wiley-Interscience, New York, NY, 1991
• 18: The linguistic data consortium, http://www.ldc,upenn.edu/.
• 19: The 2001 TDT task definition and evaluation plan, http://www.nist.gov/speech/tests/tdt/tdt2001/evalplan.htm.
• 20: Robert E. Schapire , Yoram Singer, BoosTexter: A Boosting-based Systemfor Text Categorization, Machine Learning, v.39 n.2-3, p.135-168, May-June 2000
• 21: Giridhar Kumaran , James Allan, Using names and topics for new event detection, Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, p.121-128, October 06-08, 2005, Vancouver, British Columbia, Canada [doi>10.3115/1220575.1220591]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

Additional Classification:
  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS
      H.4.2 Types of Systems
          Subjects: Decision support (e.g., MIS)

General Terms:
Algorithms, Experimentation, Performance

Keywords:
named entity, new event detection, real-time indexing, topic detection and tracking information systems