skip to main content
10.1145/1298406.1298420acmconferencesArticle/Chapter ViewAbstractPublication Pagesk-capConference Proceedingsconference-collections
Article

Capturing a taxonomy of failures during automatic interpretation of questions posed in natural language

Published: 28 October 2007 Publication History

Abstract

An important problem in artificial intelligence is capturing, from natural language, formal representationsallthat can be used by a reasoner to compute an answer. Many researchers have studied this problem by developing algorithms addressing specific phenomena in natural language interpretation, but few have studied (or cataloged) the types of failures associated with this problem. Knowledgeallof these failures can help researchers by providing a roadallmap of open research problems and help practitioners by providing a checklist of issues to address in order to build systems that can achieve good performance on this problem.allIn this paper, we present a study -- conducted in the context of the Halo Project -- cataloging the types of failures that occur when capturing knowledge from naturallanguage. We identified the categories of failures by examining a corpus of questions posed byallnaive usersallto a knowledge based question answering system and empirically demonstrated the generality of ourallcategorizations. We also describe available technologies that can address some of the failures we have identified.

References

[1]
K. Barker, B. Agashe, S. Chaw, J. Fan, M. Glass, J. Hobbs, E. Hovy, D. Israel, D.S. Kim, R. Mulkar, S. Patwardhan, B. Porter, D. Tecuci, and P. Yeh. Learning by Reading: A Prototype System, Performance Baseline and Lessons Learned. In Proceedings of Twenty-Second National Conference on Artificial Intelligence, 2007.
[2]
S. Chaw, K. Barker, B. Porter, and P. Yeh. An Ontology-Independent Problem Solver. Technical report, University of Texas at Austin, May 2007.
[3]
P. Clark, K. Barker, B. Porter, V. Chaudhri, S. Mishra, and J. Thomere. Enabling domain experts to convey questions to a machine: A modified, template-based approach. In Proceedings of Second International Conference on Knowledge Capture, 2003.
[4]
P. Clark, S. Chaw, K. Barker, V. Chaudhri, P. Harrison, J. Fan, B. John, B. Porter, A. Spaulding, J. Thompson, and P.Z. Yeh. Capturing and Answering Questions Posed to a Knowledge-Based System. In Proceedings of Fourth International Conference on Knowledge Capture, 2007.
[5]
P.R. Cohen, R. Schrag, E.K. Jones, A. Pease, A. Lin, B. Starr, D. Gunning, and M. Burke. The DARPA High-Performance Knowledge Bases project. AI Magazine, 19(4):25--49, 1998.
[6]
J. Fan, K. Barker, and B. Porter. The knowledge required to interpret noun compounds. In Proceedings of Eighteenth International Joint Conference on Artificial Intelligence, 2003.
[7]
J. Fan, K. Barker, and B. Porter. Indirect anaphora resolution as semantic path search. In Proceedings of Third International Conference on Knowledge Capture, 2005.
[8]
J. Fan and B. Porter. Interpret loosely encoded questions. In Proceedings of Nineteenth National Conference on Artificial Intelligence. AAAI Press, 2004.
[9]
N. Friedland, P. Allen, P. Matthews, M. Witbrock, D. Baxter, J. Curtis, B. Shepard, P. Miraglia, J. Angele, S. Staab, E. Moench, H. Oppermann, D. Wenke, D. Israel, V. Chaudhri, B. Porter, K. Barker, J. Fan, S. Chaw, P. Yeh, D. Tecuci, and P. Clark. Project Halo: Towards a Digital Aristotle. AI Magazine, 2004.
[10]
N. Friedland, P. Allen, M. Witbrock, J. Angele, S. Staab, D. Israel, V. Chaudhri, B. Porter, K. Barker, and P. Clark. Towards a quantitative, platform-independent analysis of knowledge systems. In Proceedings of Ninth International Conference on the Principles of Knowledge Representation and Reasoning, 2004.
[11]
D. Gildea and D. Jurafsky. Automatic Labeling of Semantic Roles. Computational Linguistics, 28(3), 2002.
[12]
J.R. Landis and G.G. Koch. The measurement of observer agreement for categorical data. Biometrics, 33:159--174, 1977.
[13]
R. Mihalcea, P. Tarau, and E. Figa. PageRank on Semantic Networks, with Application to Word Sense Disambiguation. In COLING, 2004.
[14]
R. Mitkov. Anaphora Resolution. Longman, London, 2002.
[15]
V. Nastase, J. Sayyad-Shirabad, M. Sokolova, and S. Szpakowicz. Learning noun-modifer semantic relations with corpus--based and wordnet-based features. In Proceedings of the Twenty-first National Conference on Artificial Intelligence (AAAI '07), 2006.
[16]
R. Schrag, M. Pool, V. Chaudhri, R.C. Kahlert, J. Powers, P. Cohen, JFitzgerald, and M.S. Experimental evaluation of subject matter expert-oriented. Technical report, Information Extraction and Transport, Inc., August 2002.
[17]
Proceedings of the 2002 PerMIS Workshop, August 13--15, 2002, NIST Special Publication 990, pp. 272--279.
[18]
J.F. Sowa. Conceptual Structures: Information Processing in Mind and Machine. Addison--Wesley, 1984.
[19]
D. Tecuci and B. Porter. A Generic Memory Module for Events. In Proceedings to the FLAIRS20 Conference, Key West, FL, 2007.
[20]
Teknowledge Corporation. Rapid Knowledge Formation project, 2002. http://reliant.teknowledge.com/RKF.
[21]
Vulcan Inc. Project Halo, 2003. http://projecthalo.com.
[22]
P. Yeh, B. Porter, and K. Barker. Using Transformations to Improve Semantic Matching. In KCAP, 2003.
[23]
P. Yeh, B. Porter, and K. Barker. Matching Utterances to Rich Knowledge Structures to Acquire a Model of the Speaker's Goal. In KCAP, 2005.
[24]
P. Yeh, B. Porter, and K. Barker. A unified knowledge based approach for sense disambiguation and semantic role labeling. In Proceedings of Twenty-First National Conference on Artificial Intelligence. AAAI Press, 2006.

Index Terms

  1. Capturing a taxonomy of failures during automatic interpretation of questions posed in natural language

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      K-CAP '07: Proceedings of the 4th international conference on Knowledge capture
      October 2007
      216 pages
      ISBN:9781595936431
      DOI:10.1145/1298406
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 28 October 2007

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. NLP
      2. controlled languages
      3. knowledge based systems
      4. ontology
      5. question answering

      Qualifiers

      • Article

      Conference

      K-CAP07
      Sponsor:
      K-CAP07: International Conference on Knowledge Capture 2007
      October 28 - 31, 2007
      BC, Whistler, Canada

      Acceptance Rates

      Overall Acceptance Rate 55 of 198 submissions, 28%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • 0
        Total Citations
      • 219
        Total Downloads
      • Downloads (Last 12 months)2
      • Downloads (Last 6 weeks)0
      Reflects downloads up to 12 Feb 2025

      Other Metrics

      Citations

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media