skip to main content
10.1145/2070481.2070532acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
research-article

American sign language recognition with the kinect

Published:14 November 2011Publication History

ABSTRACT

We investigate the potential of the Kinect depth-mapping camera for sign language recognition and verification for educational games for deaf children. We compare a prototype Kinect-based system to our current CopyCat system which uses colored gloves and embedded accelerometers to track children's hand movements. If successful, a Kinect-based approach could improve interactivity, user comfort, system robustness, system sustainability, cost, and ease of deployment. We collected a total of 1000 American Sign Language (ASL) phrases across both systems. On adult data, the Kinect system resulted in 51.5% and 76.12% sentence verification rates when the users were seated and standing respectively. These rates are comparable to the 74.82% verification rate when using the current(seated) CopyCat system. While the Kinect computer vision system requires more tuning for seated use, the results suggest that the Kinect may be a viable option for sign verification.

References

  1. V. Athitsos and S. Sclaroff. 3d hand pose estimation by finding appearance-based matches in a large database of training views. In In IEEE Workshop on Cues in Communication, 2001.Google ScholarGoogle Scholar
  2. B. Bauer, H. Hienz, and K. Kraiss. Video-based continuous sign language recognition using statistical methods. In Proceedings of the 15th International Conference on Pattern Recognition, volume 2, pages 463--466, September 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. H. Brashear. Improving the Efficacy of Automated Sign Language Practice Tools. PhD thesis, Georgia Institute of Technology, College of Computing, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. H. Brashear, K.-H. Park, S. Lee, V. Henderson, H. Hamilton, and T. Starner. American Sign Language Recognition in Game Development for Deaf Children. In Assets'06: Proceedings of the 8th International ACM SIGACCESS Conference on Computers and Accessibility, Portland, Oregon, 2006. ACM Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. H. Brashear, T. Starner, P. Lukowicz, and H. Junker. Using Multiple Sensors for Mobile Sign Language Recognition. In Proceedings of the Seventh IEEE International Symposium on Wearable Computers, pages 45--52, 2003. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. H. Brashear, Z. Zafrulla, T. Starner, H. Hamilton, P. Presti, and S. Lee. CopyCat: A Corpus for Verifying American Sign Language During Game Play by Deaf Children. In 4th Workshop on the Representation and Processing of Sign Languages: Corpora and Sign Language Technologies, Proceedings of the 7th Annual International Language Resources and Evaluation Conference, Valetta, Malta, 2010.Google ScholarGoogle Scholar
  7. Cooper and Bowden. Sign Language Recognition: Working with Limited Corpora. In Universal Access in Human-Computer Interaction. Applications and Services 5th International Conference, UAHCI 2009, Held as Part of HCI International 2009, pages 472--481, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. O. Eng-Jon and R. Bowden. A boosted classifier tree for hand shape detection. In Proceedings of the Sixth IEEE International Conference on Automatic Face and Gesture Recognition, pages 889--894, May 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. G. Fang, W. Gao, and D. Zhao. Large Vocabulary Sign Language Recognition Based on Hierarchical Decision Trees. In International Conference on Multimodal Interfaces, pages 125--131, 2003. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. W. Gao, G. Fang, D. Zhao, and Y. Chen. Transition Movement Models for Large Vocabulary Continuous Sign Language Recognition (CSL). In Sixth IEEE International Conference on Automatic Face and Gesture Recognition, pages 553--558, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. J. L. Hernandez-Rebollar, N. Kyriakopoulos, and R. W. Lindeman. A New Instrumented Approach for Translating American Sign Language into Sound and Text. In Proceedings of the Sixth IEEE International Conference on Automatic Face and Gesture Recognition, pages 547--552, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. J. S. Kim, W. Jang, and Z. Bien. A Dynamic Gesture Recognition System for the Korean Sign Language KSL. IEEE Transactions on Systems, Man and Cybernetics, 26(2):354--359, 1996. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. R. Liang and M. Ouhyoung. A Real-Time Continuous Gesture Recognition System for Sign Language. In Third International Conference on Automatic Face and Gesture Recognition, pages 558--565, 1998. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Z. Mo and U. Neumann. Real-time hand pose recognition using low-resolution depth images. In Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2, CVPR'06, pages 1499--1505, Washington, DC, USA, 2006. IEEE Computer Society. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. S. Ong and S. Ranganath. Automatic sign language analysis: a survey and the future beyond lexical meaning. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 27(6):873 -- 891, jun 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Prime Sense. The PrimeSensor#8482; Reference Design 1.08. April 2011.Google ScholarGoogle Scholar
  17. H. Sagawa and M. Takeuchi. A method for recognizing a sequence of sign language words represented in a Japanese Sign Language sentence. In Proceedings of the 4th IEEE International Conf. on Automatic Face and Gesture Recognition, pages 434--439, Grenoble, France, Mar 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and A. Blake. Real-Time Human Pose Recognition in Parts from Single Depth Images. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. M. Sonka, V. Hlavac, and R. Boyle. Image Processing, Analysis, and Machine Vision. Thomspon Learning, Toronto, Ontario, Canada, third edition, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. T. Starner and A. Pentland. Visual Recognition of American Sign Language Using Hidden Markov Models. In Proceedings of the International Workshop on Automatic Face and Gesture Recognition, 1995.Google ScholarGoogle Scholar
  21. W. Stokoe. Sign Language Structure: An Outline of the Visual Communication Systems of the American Deaf. Studies in Linguistics, Occasional Papers 8, 1960.Google ScholarGoogle Scholar
  22. J. Stuckler and S. Behnke. Combining depth and color cues for scale- and viewpoint-invariant object segmentation and recognition using random forests. In Intelligent Robots and Systems (IROS), 2010 IEEE/RSJ International Conference on, pages 4566--4571, October 2010.Google ScholarGoogle ScholarCross RefCross Ref
  23. G. Ten Holt, P. Hendriks, and T. Andringa. Why Don't You See What I Mean? Prospects and Limitations of Current Automatic Sign Recognition Research. Sign Language Studies, 6(4), Summer 2006.Google ScholarGoogle Scholar
  24. C. Vogler and D. Metaxas. Adapting Hidden Markov Models for ASL recognition by using three-dimensional computer vision methods. In Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, pages 156--161, Orlando, FL, October 1997.Google ScholarGoogle ScholarCross RefCross Ref
  25. C. Vogler and D. Metaxas. ASL Recognition Based on a Coupling Between HMMs and 3D Motion Analysis. In Proceedings of the IEEE International Conference on Computer Vision, pages 363--369, 1998. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. C. Vogler and D. Metaxas. Handshapes and Movements: Multiple-Channel American Sign Language Recognition. In Gesture-Based Communication in Human-Computer Interaction, volume 2915, pages 247--258. Springer-Verlag, January 2004. Lecture notes in Artificial Intelligence.Google ScholarGoogle Scholar
  27. C. Vogler, H. Sun, and D. Metaxas. A framework for motion recognition with applications to American sign language and gait recognition. In Proceedings of Workshop on Human Motion, pages 33--38, 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. R. Y. Wang and J. Popović. Real-time hand-tracking with a color glove. ACM Trans. Graph., 28:63:1--63:8, July 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. K. A. Weaver, H. Hamilton, Z. Zafrulla, H. Brashear, T. Starner, P. Presti, , and A. Bruckman. Improving the Language Ability of Deaf Signing Children through an Interactive American Sign Language-Based Video Game. In Proceedings of 9th International Conference of the Learning Sciences, June 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. T. Westeyn, H. Brashear, A. Atrash, and T. Starner. Georgia Tech Gesture Toolkit: Supporting Experiments in Gesture Recognition. In ICMI'03: Proceedings of the 5th International Conference on Multimodal Interfaces, New York, NY, USA, 2003. ACM Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. A. Wilson. Using a Depth Camera as a Touch Sensor. The ACM International Conference on Interactive Tabletops and Surfaces, November 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. R. Yang, S. Sarkar, and B. Loeding. Handling movement epenthesis and hand segmentation ambiguities in continuous sign language recognition using nested dynamic programming. The IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(3):462--477, March 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Z. Zafrulla, H. Brashear, H. Hamilton, and T. Starner. A novel approach to American Sign Language (ASL) Phrase Verification using Reversed Signing. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2010.Google ScholarGoogle Scholar
  34. Z. Zafrulla, H. Brashear, H. Hamilton, and T. Starner. Towards an American Sign Langauge Verifier for Educational Game for Deaf Children. In Proceedings of International Conference on Pattern Recognition, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. American sign language recognition with the kinect

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in
    • Published in

      cover image ACM Conferences
      ICMI '11: Proceedings of the 13th international conference on multimodal interfaces
      November 2011
      432 pages
      ISBN:9781450306416
      DOI:10.1145/2070481

      Copyright © 2011 ACM

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 14 November 2011

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article

      Acceptance Rates

      Overall Acceptance Rate453of1,080submissions,42%

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader