A compositional context sensitive multi-document summarizer: exploring the factors that influence summarization

ABSTRACT

The usual approach for automatic summarization is sentence extraction, where key sentences from the input documents are selected based on a suite of features. While word frequency often is used as a feature in summarization, its impact on system performance has not been isolated. In this paper, we study the contribution to summarization of three factors related to frequency: content word frequency, composition functions for estimating sentence importance from word frequency, and adjustment of frequency weights based on context. We carry out our analysis using datasets from the Document Understanding Conferences, studying not only the impact of these features on automatic summarizers, but also their role in human summarization. Our research shows that a frequency based summarizer can achieve performance comparable to that of state-of-the-art systems, but only with a good composition function; context sensitivity improves performance and significantly reduces repetition.

REFERENCES

• 1: M. Banko and L. Vanderwende. Using n-grams to understand the nature of summaries. In Proceedings of HLT/NAACL'04, 2004.
• 2: Regina Barzilay , Kathleen R. McKeown, Sentence Fusion for Multidocument News Summarization, Computational Linguistics, v.31 n.3, p.297-328, September 2005 [doi>10.1162/089120105774321091]
• 3: Jaime Carbonell , Jade Goldstein, The use of MMR, diversity-based reranking for reordering documents and producing summaries, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.335-336, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291025]
• 4: J. Conroy, J. Schlesinger, J. Goldstein, and D. O'Leary. Left-brain/right-brain multi-document summarization. In Proceedings of the 4th Document Undersatnding Conference (DUC'04), 2004.
• 5: T. Copeck and S. Szpakowicz. Vocabulary agreement among model summaries and source documents. In Proceedings of the Document Understanding Conference DUC'04, 2004.
• 6: H. Daumé III and D. Marcu. Bayesian multi-document summarization at mse. In Proceedings of the Workshop on Multilingual Summarization Evaluation (MSE), Ann Arbor, MI, June 29 2005.
• 7: D. K. Elson. Project logline: Rhetorical categorization for multidocument news summarization. Master's thesis, Columbia University, 2005.
• 8: D. K. Evans and K. McKeown. Identifying similarities and differences across english and arabic news. In Proceedings of the International Conference on Intelligence Analysis, 2005.
• 9: Jade Goldstein , Vibhu Mittal , Jaime Carbonell , Jamie Callan, Creating and evaluating multi-document sentence extract summaries, Proceedings of the ninth international conference on Information and knowledge management, p.165-172, November 06-11, 2000, McLean, Virginia, United States [doi>10.1145/354756.354815]
• 10: Hongyan Jing , Kathleen R. McKeown, Cut and paste based text summarization, Proceedings of the first conference on North American chapter of the Association for Computational Linguistics, p.178-185, April 29-May 04, 2000, Seattle, Washington
• 11: Kevin Knight , Daniel Marcu, Summarization beyond sentence extraction: a probabilistic approach to sentence compression, Artificial Intelligence, v.139 n.1, p.91-107, July 2002 [doi>10.1016/S0004-3702(02)00222-9 ]
• 12: Julian Kupiec , Jan Pedersen , Francine Chen, A trainable document summarizer, Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, p.68-73, July 09-13, 1995, Seattle, Washington, United States [doi>10.1145/215206.215333]
• 13: C.-Y. Lin. Rouge: a package for automatic evaluation of summaries. In Proceedings of the Workshop in Text Summarization, ACL'04, 2004.
• 14: C.-Y. Lin and E. Hovy. Automated multi-document summarization in neats. In Proceedings of the Human Language Technology Conference (HLT2002 ), 2002.
• 15: Chin-Yew Lin , Eduard Hovy, Automatic evaluation of summaries using N-gram co-occurrence statistics, Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, p.71-78, May 27-June 01, 2003, Edmonton, Canada [doi>10.3115/1073445.1073465]
• 16: C.-Y. Lin and F. J. Och. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL 2004), 2004.
• 17: H. P. Luhn. The automatic creation of literature abstracts. IBM Journal of Research and Development, 2(2):159--165, 1958.
• 18: D. Marcu and L. Gerber. An inquiry into the nature of multidocument abstracts, extracts, and their evaluation. In Proceedings of the NAACL-2001 Workshop on Automatic Summarization, 2001.
• 19: A. Nenkova and R. Passonneau. Evaluating content selection in summarization: The pyramid method. In Proceedings of HLT/NAACL 2004, 2004.
• 20: E. Newman, W. Doran, N. Stokes, J. Carthy, and J. Dunnion. Comparing redundancy removal techniques for multi-document summarisation. In Proceedings of STAIRS, pages 223--228, 2004.
• 21: P. Over and J. Yen. An introduction to duc 2004 intrinsic evaluation of generic news text summarization systems. In Proceedings of DUC 2004, 2004.
• 22: R. Passonneau, A. Nenkova, K. McKeown, and S. Sigleman. Pyramid evaluation ot duc 2005. In Proceedings of the Document Understanding Conference (DUC'05), 2005.
• 23: Dragomir R. Radev , Simone Teufel , Horacio Saggion , Wai Lam , John Blitzer , Hong Qi , Arda Çelebi , Danyu Liu , Elliott Drabek, Evaluation challenges in large-scale document summarization, Proceedings of the 41st Annual Meeting on Association for Computational Linguistics, p.375-382, July 07-12, 2003, Sapporo, Japan [doi>10.3115/1075096.1075144]
• 24: G. J. Rath, A. Resnick, and R. Savage. The formation of abstracts by the selection of sentences: Part 1: sentence selection by man and machines. American Documentation, 2(12):139--208, 1961.
• 25: N. Schenker and J. Gentleman. On judging the significance of differences by examining the overlap between confidence intervals. The American Statistician, 55(3):182--186, 2001.
• 26: B. Schiffman, A. Nenkova, and K. McKeown. Experiments in multidocument summarization. In Proceedings of the Human Language Technology Conference, 2002.
• 27: Hans van Halteren , Simone Teufel, Examining the consensus between human summaries: initial experiments with factoid analysis, Proceedings of the HLT-NAACL 03 on Text summarization workshop, p.57-64, May 31, 2003 [doi>10.3115/1119467.1119475]
• 28: L. Vanderwende, M. Banko, and A. Menezes. Event-centric summary generation. In Proceedings of the Document Understanding Conference (DUC'04), 2004.
• 29: L. Vanderwende and H. Suzuki. Frequency-based summarizer and a language modeling extention. In MSE 2005 common data task evaluation, 2005.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing

General Terms:
Experimentation, Human Factors, Measurement

Keywords:
compositionality, context-sensitivity, frequency, multi-document summarization