Confidence estimation for NLP applications

ABSTRACT

Confidence measures are a practical solution for improving the usefulness of Natural Language Processing applications. Confidence estimation is a generic machine learning approach for deriving confidence measures. We give an overview of the application of confidence estimation in various fields of Natural Language Processing, and present experimental results for speech recognition, spoken language understanding, and statistical machine translation.

REFERENCES

• 1: Christopher M. Bishop, Neural Networks for Pattern Recognition, Oxford University Press, Inc., New York, NY, 1995
• 2: Blatz, J., Fitzgerald, E., Foster, G., Gandrabur, S., Goutte, C., Kulesza, A., Sanchis, A., and Ueffing, N. 2004a. Confidence estimation for machine translation. In Proceedings of COLING 2004 (Geneva). 315--321.
• 3: Blatz, J., Fitzgerald, E., Foster, G., Gandrabur, S., Goutte, C., Kulesza, A., Sanchis, A., and Ueffing, N. 2004b. Confidence estimation for machine translation. Tech. Rep., Final report JHU /CLSP 2003 Summer Workshop, Baltimore, MD.
• 4: Caminero, J., De La Torre, D., Villarubia, L., Martin, C., and Hernandez, L. 1996. On-line garbage modeling with discriminant analysis for utterance verification. In Proceedings of the 4th ICSLP 1996. Vol. 4. Philadelphia, PA, 2115--2118.
• 5: Carpenter, P., Jin, C., Wilson, D., Zhang, R., Bohus, D., and Rudnicky, A. 2001. Is this conversation on track? In Eurospeech 2001 (Aalborg, Denmark). 2121--2124.
• 6: Michael Collins, Ranking algorithms for named-entity extraction: boosting and the voted perceptron, Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, July 07-12, 2002, Philadelphia, Pennsylvania [doi>10.3115/1073083.1073165]
• 7: Michael Collins , Terry Koo, Discriminative Reranking for Natural Language Parsing, Computational Linguistics, v.31 n.1, p.25-70, March 2005 [doi>10.1162/0891201053630273]
• 8: Collins, M., Roark, B., and Saraclar, M. 2005. Discriminative syntactic language modeling for speech recognition. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05). Association for Computational Linguistics, Ann Arbor, MI. 507--514.
• 9: Collobert, R., Bengio, S., and Mariéthoz, J. 2002. Torch: A modular machine learning software library. Tech. Rep. IDIAP-RR 02-46, IDIAP.
• 10: Culotta, A. and McCallum, A. 2004. Confidence estimation for information extraction. In Proceedings of HLT-NAACL.
• 11: Culotta, A. and McCallum, A. 2005. Reducing labeling effort for structured prediction tasks. In Proceedings of AAAI.
• 12: Delany, S. J., Cunningham, P., and Doyle, D. 2005. Generating estimates of classification confidence for a case-based spam filter. Tech. Rep. TCD-CS-2005-20, Trinity College Dublin Computer Science Department.
• 13: George Foster, A maximum entropy/minimum divergence translation model, Proceedings of the 38th Annual Meeting on Association for Computational Linguistics, p.45-52, October 03-06, 2000, Hong Kong [doi>10.3115/1075218.1075224]
• 14: George Foster , Pierre Isabelle , Pierre Plamondon, Target-Text Mediated Interactive Machine Translation, Machine Translation, v.12 n.1-2, p.175-194, 1997
• 15: George Foster , Philippe Langlais , Guy Lapalme, User-friendly text prediction for translators, Proceedings of the ACL-02 conference on Empirical methods in natural language processing, p.148-155, July 06, 2002 [doi>10.3115/1118693.1118713]
• 16: Simona Gandrabur , George Foster, Confidence estimation for translation prediction, Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003, p.95-102, May 31, 2003, Edmonton, Canada [doi>10.3115/1119176.1119189]
• 17: L. Gillick , Y. Ito , J. Young, A Probabilistic Approach to Confidence Estimation and Evaluation, Proceedings of the 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '97)-Volume 2, p.879, April 21-24, 1997
• 18: Dirk Grunwald , Artur Klauser , Srilatha Manne , Andrew Pleszkun, Confidence estimation for speculation control, Proceedings of the 25th annual international symposium on Computer architecture, p.122-131, June 27-July 02, 1998, Barcelona, Spain
• 19: Guillevic, D., Gandrabur, S., and Normandin, Y. 2002. Robust semantic confidence scoring. In Proceedings of the 7th International Conference on Spoken Language Processing (ICSLP) 2002 (Denver, CO).
• 20: Hasegawa-Johnson, M., Baker, J., Borys, S., Chen, K., Coogan, E., Greenberg, S., Juneja, A., Kirchhoff, K., Livescu, K., Mohan, S., Muller, J., Sonmez, K., and Wang, T. 2005. Landmark-based speech recognition: Report of the 2004 Johns Hopkins Summer Workshop. In ICASSP.
• 21: Hazen, T., Burianek, T., Polifroni, J., and Seneff, S. 2000. Recognition confidence scoring for use in speech understanding systems. In Proc. Automatic Speech Recognition Workshop (Paris, France). 213--220.
• 22: Hazen, T., Seneff, S., and Polifroni, J. 2002. Recognition confidence scoring and its use in speech understanding systems. Compu. Speech Lang., 49--67.
• 23: Thorsten Joachims, Text Categorization with Suport Vector Machines: Learning with Many Relevant Features, Proceedings of the 10th European Conference on Machine Learning, p.137-142, April 21-23, 1998
• 24: Kemp, T. and Schaaf, T. 1997. Estimating confidence using word lattices. In Eurospeech. 827--830.
• 25: Philippe Langlais , Guy Lapalme, Trans Type: Development-Evaluation Cycles to Boost Translator's Productivity, Machine Translation, v.17 n.2, p.77-98, 2002 [doi>10.1023/B:COAT.0000010117.98933.a0]
• 26: Ma, C., Randolph, M., and Drish, J. 2001. A support vector machines-based rejection technique for speech recognition. In ICASSP 2001.
• 27: Macklovitch, E. 2006. TransType2: The last word. In Proceedings of LREC 2006. Genoa, Italy.
• 28: Maison, B. and Gopinath, R. 2001. Robust confidence annotation and rejection for continuous speech recognition. In ICASSP 2001.
• 29: Mangu, L., Brill, E., and Stolcke, A. 2000. Finding consensus in speech recognition: Word error minimization and other applications of confusion networks. Compu. Speech Lang. 14, 4, 373--400.
• 30: Manmatha, R. and Sever, H. 2002. A formal approach to score normalization for meta-search. In Proceedings of HLT 2002, 2nd International Conference on Human Language Technology Research, M. Marcus, Ed. Morgan Kaufmann, San Francisco, CA. 98--103.
• 31: Moreno, P., Logan, B., and Raj, B. 2001. A boosting approach for confidence scoring. In Proceedings of Eurospeech 2001 (Aalborg, Denmark). 2109--2112.
• 32: Nepveu, L., Lapalme, G., Langlais, P., and Foster, G. 2004. Adaptive language and translation models for interactive machine translation. In EMLNP'04 (Barcelona, Spain). 190--197.
• 33: C. Neti , S. Roukos , E. Eide, Word-Based Confidence Measures As a Guide for Stack Search in Speech Recognition, Proceedings of the 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '97)-Volume 2, p.883, April 21-24, 1997
• 34: NIST. 2001. Automatic evaluation of machine translation quality using n-gram co-occurence statistics. www.nist.gov/speech/tests/mt/doc/ngram-study.pdf.
• 35: Franz Josef Och , Hermann Ney, Discriminative training and maximum entropy models for statistical machine translation, Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, July 07-12, 2002, Philadelphia, Pennsylvania [doi>10.3115/1073083.1073133]
• 36: Papineni, K., Roukos, S., Ward, T., and Zhu, W. 2001. Bleu: A method for automatic evaluation of machine translation. Technical Report RC22176 (W0109-022), IBM Research Division, Thomas J. Watson Research Center (2001).
• 37: Quirk, C. B. 2004. Training a sentence-level machine translation confidence measure. In Proceedings of LREC (Lisbon, Portual). 825--828.
• 38: San-Secundo, R., Pellom, B., Hacioglu, K., Ward, W., and Pardo, J. 2001. Confidence measures for spoken dialogue systems. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP-2001) (Salt Lake City, UT).
• 39: Sanchis, A., Juan, A., and Vidal, E. 2003. Improving utterance verification using a smoothed naive Bayes model. In ICASSP 2003, (Hong Kong). 29--35.
• 40: Schapire, R. 2001. The boosting approach to machine learning: An overview. In Proceedings of the MSRI Workshop on Nonlinear Estimation and Classification (Berkeley, CA, Mar.).
• 41: Shen, L., Sarkar, A., and Och, F. J. 2004. Discriminative reranking for machine translation. In HLT-NAACL. 177--184.
• 42: Siu, M. and Gish, H. 1999. Evaluation of word confidence for speech recognition systems. Comput. Speech Lang. 13, 4, 299--318.
• 43: Stolcke, A., Koenig, Y., and Weintraub, M. 1997. Explicit word error minimization in N-best list rescoring. In Proceedings of the 5th European Conference on Speech Communication and Technology. Vol. 1. (Rhodes, Greece). 163--166.
• 44: Kai Ming Ting , Ian H. Witten, Stacking Bagged and Dagged Models, Proceedings of the Fourteenth International Conference on Machine Learning, p.367-375, July 08-12, 1997
• 45: Ueffing, N., Macherey, K., and Ney, H. 2003. Confidence measures for statistical machine translation. In Proceedings of the 6th Conference of the Association for Machine Translation in the Americas (New-Orleans, LA). Springer-Verlag, New York, 394--401.
• 46: Ueffing, N. and Ney, H. 2005. Word-level confidence estimation for machine translation using phrase-based translation models. In Proceedings of HLT/EMNLP. Vancouver, BC, Canada.
• 47: M. Weintraub , F. Beaufays , Z. Rivlin , Y. Konig , A. Stolcke, Neural - Network Based Measures of Confidence for Word Recognition, Proceedings of the 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '97)-Volume 2, p.887, April 21-24, 1997
• 48: Wessel, F., Schluter, R., Macherey, K., and Ney, H. 2001. Confidence measures for large vocabulary continuous speech recognition. IEEE Trans. Speech Audio Process. 9, 3, 288--298.
• 49: Xu, J., Licuanan, A., May, J., Miller, S., and Weischedel, R. 2002. Trec2002qa at bbn: Answer selection and confidence estimation. In Proceedings of the 10th Text REtrieval Conference (TREC 2002).
• 50: Zhang, R. and Rudnicky, A. 2001. Word level confidence annotation using combinations of features. In Eurospeech. Aalborg, Denmark, 2105--2108.
• 51: Zou, K. H. 2004. Receiver operating characteristic (ROC) literature research. http://splweb.bwh.harvard.edu:8000/pages/ppl/zou/roc.html.

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS
      Subjects: Fault tolerance

Additional Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS
      Subjects: Reliability, availability, and serviceability; Modeling techniques

  F. Theory of Computation
  F.1 COMPUTATION BY ABSTRACT DEVICES
      F.1.1 Models of Computation
          Subjects: Self-modifying machines (e.g., neural networks)

  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Reliability and life testing; Statistical computing

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning
          Subjects: Connectionism and neural nets
      I.2.7 Natural Language Processing
          Subjects: Machine translation; Language parsing and understanding; Speech recognition and synthesis

  J. Computer Applications
  J.5 ARTS AND HUMANITIES
      Subjects: Language translation; Linguistics

General Terms:
Algorithms, Design, Experimentation, Reliability, Theory

Keywords:
Confidence measures, computer-aided machine translation, confidence estimation, speech recognition, spoken language understanding, statistical machine learning