Similarity-based Distant Supervision for Definition Retrieval

ABSTRACT

Recognizing definition sentences from free text corpora often requires hand-crafted patterns or explicitly labeled training instances. We present a distant supervision approach addressing this challenge without using explicitly labeled data. We use plausibly good but imperfect definition sentences from Wikipedia as references to annotate sentences in a target corpus based on text similarity measures such as ROUGE. Experimental results show our approach is highly effective, generating noisy but large, useful, and localized training instances. Definition sentence retrieval models trained using the synthesized training examples are more effective than those learned from manual judgments of a few thousand sentences. We also examine different text similarity measures for annotation, including both unsupervised and supervised ones. We show that our method can significantly benefit from supervised text similarity measures learned from either external training data (from the SemEval Semantic Text Similarity task) or local ones (a few hundred judged sentences on the target corpus). Our method offers a cheap, effective, and flexible solution to this task and can benefit a broad range of applications such as web search engines and QA systems.

References

1. E. Agirre, C. Banea, D. Cer, M. Diab, A. Gonzalez-Agirre, R. Mihalcea, G. Rigau, and J. Wiebe. SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In Proceedings of SemEval-2016, pages 497--511, 2016.
2. M. S. Bernstein, J. Teevan, S. Dumais, D. Liebling, and E. Horvitz. Direct answers for search queries in the long tail. In CHI '12, pages 237--246, 2012.
3. G. Boella and L. Di Caro. Extracting definitions and hypernym relations relying on syntactic dependencies and support vector machines. In ACL '13, pages 532--537, 2013.
4. T. Brychcın and L. Svoboda. UWB at SemEval-2016 task 1: Semantic textual similarity using lexical, syntactic, and semantic information. In Proceedings of SemEval-2016, pages 588--594, 2016.
5. C. J. Burges. From RankNet to LambdaRank to LambdaMART: An overview. Technical Report MSR-TR-2010-82, Microsoft Research, 2010.
6. B. Carterette and J. Allan. Semiautomatic evaluation of retrieval systems using document similarities. In CIKM '07, pages 873--876, 2007.
7. L. B. Chilton and J. Teevan. Addressing people's information needs directly in a web search result page. In WWW '11, pages 27--36, 2011.
8. H. Cui, M.-Y. Kan, and T.-S. Chua. Generic soft pattern models for definitional question answering. In SIGIR '05, pages 384--391, 2005.
9. H. Cui, M.-Y. Kan, and T.-S. Chua. Soft pattern matching models for definitional question answering. ACM Transactions on Information Systems, 25(2), 2007.
10. L. Espinosa-Anke, F. Ronzano, and H. Saggion. Weakly supervised definition extraction. In Proceedings of Recent Advances in Natural Language Processing, pages 176--185, 2015.
11. L. Espinosa-Anke and H. Saggion. Applying dependency relations to definition extraction. In Proceedings of the 19th International Conference on Applications of Natural Language to Information Systems, pages 63--74, 2014.
12. M. A. Hearst. Automatic acquisition of hyponyms from large text corpora. In COLING '92, pages 539--545, 1992.
13. E. Hovy, C.-Y. Lin, L. Zhou, and J. Fukumoto. Automated summarization evaluation with basic elements. In LREC '06, pages 899--902, 2006.
14. A. Intxaurrondo, E. Agirre, O. L. de Lacalle, and M. Surdeanu. Diamonds in the rough: Event extraction from imperfect microblog data. In NAACL '15, pages 641--650, 2015.
15. K. Järvelin and J. Kekäläinen. IR evaluation methods for retrieving highly relevant documents. In SIGIR '00, pages 41--48, 2000.
16. Y. Jin, M.-Y. Kan, J.-P. Ng, and X. He. Mining scientific terms and their definitions: A study of the ACL anthology. In EMNLP '13, pages 780--790, 2013.
17. K.-W. Kor and T.-S. Chua. Interesting nuggets and their impact on definitional question answering. In SIGIR '07, pages 335--342, 2007.
18. R. Krovetz. Viewing morphology as an inference process. In SIGIR '93, pages 191--202, 1993.
19. Q. Le and T. Mikolov. Distributed representations of sentences and documents. In ICML '14, pages 1188--1196, 2014.
20. C.-Y. Lin. ROUGE: A package for automatic evaluation of summaries. In ACL '04 Workshop on Text Summarization Branches Out, 2004.
21. C.-Y. Lin and E. Hovy. Automatic evaluation of summaries using n-gram co-occurrence statistics. In NAACL '03, pages 71--78, 2003.
22. C. D. Manning, M. Surdeanu, J. Bauer, J. Finkel, S. J. Bethard, and D. McClosky. The Stanford CoreNLP natural language processing toolkit. In ACL '14, pages 55--60, 2014.
23. T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. In NIPS '13, pages 3111--3119, 2013.
24. M. Mintz, S. Bills, R. Snow, and D. Jurafsky. Distant supervision for relation extraction without labeled data. In ACL '09, pages 1003--1011, 2009.
25. S. Muresan and J. Klavans. A method for automatically building and evaluating dictionary resources. In LREC '02, pages 231--234, 2002.
26. R. Navigli and P. Velardi. Learning word-class lattices for definition and hypernym extraction. In ACL '10, pages 1318--1327, 2010.
27. A. Nenkova and R. Passonneau. Evaluating content selection in summarization: The pyramid method. In NAACL '04, 2004.
28. J.-P. Ng and V. Abrecht. Better summarization evaluation with word embeddings for ROUGE. In EMNLP '15, pages 1925--1930, 2015.
29. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. BLEU: A method for automatic evaluation of machine translation. In ACL '02, pages 311--318, 2002.
30. R. J. Passonneau, E. Chen, W. Guo, and D. Perin. Automated pyramid scoring of summaries using distributional semantics. In ACL '13, pages 143--147, 2013.
31. J. Pennington, R. Socher, and C. D. Manning. Glove: Global vectors for word representation. In EMNLP '14, pages 1532--1543, 2014.
32. M. Purver and S. Battersby. Experimenting with distant supervision for emotion classification. In EACL '12, pages 482--491, 2012.
33. M. Reiplinger, U. Schäfer, and M. Wolska. Extracting glossary sentences from scholarly articles: A comparative evaluation of pattern bootstrapping and deep analysis. In Proceedings of the ACL-2012 Special Workshop on Rediscovering 50 Years of Discoveries, pages 55--65, 2012.
34. X. Ren, A. El-Kishky, C. Wang, F. Tao, C. R. Voss, and J. Han. ClusType: Effective entity recognition and typing by relation phrase-based clustering. In KDD '15, pages 995--1004, 2015.
35. R. Snow, D. Jurafsky, and A. Y. Ng. Learning syntactic patterns for automatic hypernym discovery. In NIPS '04, pages 1297--1304, 2004.
36. M. A. Sultan, S. Bethard, and T. Sumner. DLS@CU: Sentence similarity from word alignment and semantic vector composition. In Proceedings of SemEval-2015, pages 148--153, 2015.
37. E. Westerhout and P. Monachesi. Extraction of Dutch definitory contexts for eLearning purposes. In Proceedings of the 17th Meeting of Computational Linguistics in the Netherlands, pages 219--234, 2007.
38. F. Wu and D. S. Weld. Open information extraction using Wikipedia. In ACL '10, pages 118--127, 2010.
39. J. Xu, R. Weischedel, and A. Licuanan. Evaluation of an extraction-based approach to answering definitional questions. In SIGIR '04, pages 418--424, 2004.

Index Terms

• Information systems

  • Information retrieval

    • Evaluation of retrieval results

      • Relevance assessment

    • Retrieval models and ranking

      • Similarity measures