ABSTRACT
We investigate the potential of the Kinect depth-mapping camera for sign language recognition and verification for educational games for deaf children. We compare a prototype Kinect-based system to our current CopyCat system which uses colored gloves and embedded accelerometers to track children's hand movements. If successful, a Kinect-based approach could improve interactivity, user comfort, system robustness, system sustainability, cost, and ease of deployment. We collected a total of 1000 American Sign Language (ASL) phrases across both systems. On adult data, the Kinect system resulted in 51.5% and 76.12% sentence verification rates when the users were seated and standing respectively. These rates are comparable to the 74.82% verification rate when using the current(seated) CopyCat system. While the Kinect computer vision system requires more tuning for seated use, the results suggest that the Kinect may be a viable option for sign verification.
- V. Athitsos and S. Sclaroff. 3d hand pose estimation by finding appearance-based matches in a large database of training views. In In IEEE Workshop on Cues in Communication, 2001.Google Scholar
- B. Bauer, H. Hienz, and K. Kraiss. Video-based continuous sign language recognition using statistical methods. In Proceedings of the 15th International Conference on Pattern Recognition, volume 2, pages 463--466, September 2000. Google ScholarDigital Library
- H. Brashear. Improving the Efficacy of Automated Sign Language Practice Tools. PhD thesis, Georgia Institute of Technology, College of Computing, 2010. Google ScholarDigital Library
- H. Brashear, K.-H. Park, S. Lee, V. Henderson, H. Hamilton, and T. Starner. American Sign Language Recognition in Game Development for Deaf Children. In Assets'06: Proceedings of the 8th International ACM SIGACCESS Conference on Computers and Accessibility, Portland, Oregon, 2006. ACM Press. Google ScholarDigital Library
- H. Brashear, T. Starner, P. Lukowicz, and H. Junker. Using Multiple Sensors for Mobile Sign Language Recognition. In Proceedings of the Seventh IEEE International Symposium on Wearable Computers, pages 45--52, 2003. Google ScholarDigital Library
- H. Brashear, Z. Zafrulla, T. Starner, H. Hamilton, P. Presti, and S. Lee. CopyCat: A Corpus for Verifying American Sign Language During Game Play by Deaf Children. In 4th Workshop on the Representation and Processing of Sign Languages: Corpora and Sign Language Technologies, Proceedings of the 7th Annual International Language Resources and Evaluation Conference, Valetta, Malta, 2010.Google Scholar
- Cooper and Bowden. Sign Language Recognition: Working with Limited Corpora. In Universal Access in Human-Computer Interaction. Applications and Services 5th International Conference, UAHCI 2009, Held as Part of HCI International 2009, pages 472--481, 2009. Google ScholarDigital Library
- O. Eng-Jon and R. Bowden. A boosted classifier tree for hand shape detection. In Proceedings of the Sixth IEEE International Conference on Automatic Face and Gesture Recognition, pages 889--894, May 2004. Google ScholarDigital Library
- G. Fang, W. Gao, and D. Zhao. Large Vocabulary Sign Language Recognition Based on Hierarchical Decision Trees. In International Conference on Multimodal Interfaces, pages 125--131, 2003. Google ScholarDigital Library
- W. Gao, G. Fang, D. Zhao, and Y. Chen. Transition Movement Models for Large Vocabulary Continuous Sign Language Recognition (CSL). In Sixth IEEE International Conference on Automatic Face and Gesture Recognition, pages 553--558, 2004. Google ScholarDigital Library
- J. L. Hernandez-Rebollar, N. Kyriakopoulos, and R. W. Lindeman. A New Instrumented Approach for Translating American Sign Language into Sound and Text. In Proceedings of the Sixth IEEE International Conference on Automatic Face and Gesture Recognition, pages 547--552, 2004. Google ScholarDigital Library
- J. S. Kim, W. Jang, and Z. Bien. A Dynamic Gesture Recognition System for the Korean Sign Language KSL. IEEE Transactions on Systems, Man and Cybernetics, 26(2):354--359, 1996. Google ScholarDigital Library
- R. Liang and M. Ouhyoung. A Real-Time Continuous Gesture Recognition System for Sign Language. In Third International Conference on Automatic Face and Gesture Recognition, pages 558--565, 1998. Google ScholarDigital Library
- Z. Mo and U. Neumann. Real-time hand pose recognition using low-resolution depth images. In Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2, CVPR'06, pages 1499--1505, Washington, DC, USA, 2006. IEEE Computer Society. Google ScholarDigital Library
- S. Ong and S. Ranganath. Automatic sign language analysis: a survey and the future beyond lexical meaning. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 27(6):873 -- 891, jun 2005. Google ScholarDigital Library
- Prime Sense. The PrimeSensor#8482; Reference Design 1.08. April 2011.Google Scholar
- H. Sagawa and M. Takeuchi. A method for recognizing a sequence of sign language words represented in a Japanese Sign Language sentence. In Proceedings of the 4th IEEE International Conf. on Automatic Face and Gesture Recognition, pages 434--439, Grenoble, France, Mar 2000. Google ScholarDigital Library
- J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and A. Blake. Real-Time Human Pose Recognition in Parts from Single Depth Images. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2011. Google ScholarDigital Library
- M. Sonka, V. Hlavac, and R. Boyle. Image Processing, Analysis, and Machine Vision. Thomspon Learning, Toronto, Ontario, Canada, third edition, 2008. Google ScholarDigital Library
- T. Starner and A. Pentland. Visual Recognition of American Sign Language Using Hidden Markov Models. In Proceedings of the International Workshop on Automatic Face and Gesture Recognition, 1995.Google Scholar
- W. Stokoe. Sign Language Structure: An Outline of the Visual Communication Systems of the American Deaf. Studies in Linguistics, Occasional Papers 8, 1960.Google Scholar
- J. Stuckler and S. Behnke. Combining depth and color cues for scale- and viewpoint-invariant object segmentation and recognition using random forests. In Intelligent Robots and Systems (IROS), 2010 IEEE/RSJ International Conference on, pages 4566--4571, October 2010.Google ScholarCross Ref
- G. Ten Holt, P. Hendriks, and T. Andringa. Why Don't You See What I Mean? Prospects and Limitations of Current Automatic Sign Recognition Research. Sign Language Studies, 6(4), Summer 2006.Google Scholar
- C. Vogler and D. Metaxas. Adapting Hidden Markov Models for ASL recognition by using three-dimensional computer vision methods. In Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, pages 156--161, Orlando, FL, October 1997.Google ScholarCross Ref
- C. Vogler and D. Metaxas. ASL Recognition Based on a Coupling Between HMMs and 3D Motion Analysis. In Proceedings of the IEEE International Conference on Computer Vision, pages 363--369, 1998. Google ScholarDigital Library
- C. Vogler and D. Metaxas. Handshapes and Movements: Multiple-Channel American Sign Language Recognition. In Gesture-Based Communication in Human-Computer Interaction, volume 2915, pages 247--258. Springer-Verlag, January 2004. Lecture notes in Artificial Intelligence.Google Scholar
- C. Vogler, H. Sun, and D. Metaxas. A framework for motion recognition with applications to American sign language and gait recognition. In Proceedings of Workshop on Human Motion, pages 33--38, 2000. Google ScholarDigital Library
- R. Y. Wang and J. Popović. Real-time hand-tracking with a color glove. ACM Trans. Graph., 28:63:1--63:8, July 2009. Google ScholarDigital Library
- K. A. Weaver, H. Hamilton, Z. Zafrulla, H. Brashear, T. Starner, P. Presti, , and A. Bruckman. Improving the Language Ability of Deaf Signing Children through an Interactive American Sign Language-Based Video Game. In Proceedings of 9th International Conference of the Learning Sciences, June 2010. Google ScholarDigital Library
- T. Westeyn, H. Brashear, A. Atrash, and T. Starner. Georgia Tech Gesture Toolkit: Supporting Experiments in Gesture Recognition. In ICMI'03: Proceedings of the 5th International Conference on Multimodal Interfaces, New York, NY, USA, 2003. ACM Press. Google ScholarDigital Library
- A. Wilson. Using a Depth Camera as a Touch Sensor. The ACM International Conference on Interactive Tabletops and Surfaces, November 2010. Google ScholarDigital Library
- R. Yang, S. Sarkar, and B. Loeding. Handling movement epenthesis and hand segmentation ambiguities in continuous sign language recognition using nested dynamic programming. The IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(3):462--477, March 2010. Google ScholarDigital Library
- Z. Zafrulla, H. Brashear, H. Hamilton, and T. Starner. A novel approach to American Sign Language (ASL) Phrase Verification using Reversed Signing. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2010.Google Scholar
- Z. Zafrulla, H. Brashear, H. Hamilton, and T. Starner. Towards an American Sign Langauge Verifier for Educational Game for Deaf Children. In Proceedings of International Conference on Pattern Recognition, 2010. Google ScholarDigital Library
Index Terms
American sign language recognition with the kinect
Recommendations
American sign language recognition in game development for deaf children
Assets '06: Proceedings of the 8th international ACM SIGACCESS conference on Computers and accessibilityCopyCat is an American Sign Language (ASL) game, which uses gesture recognition technology to help young deaf children practice ASL skills. We describe a brief history of the game, an overview of recent user studies, and the results of recent work on ...
Kinect-based Taiwanese sign-language recognition system
Gesture-recognition is an important component for many intelligent human---computer interaction applications. For example, a realtime sign-language recognition system would detect and interpret hand gestures. Many vision-based sign-language recognition ...
Gesture-driven American sign language phraselator
ICMI '05: Proceedings of the 7th international conference on Multimodal interfacesThis paper describes a portable American Sign Language (ASL)-to-English phraselator. This wearable device is based on an Acceleglove originally developed for recognizing the hand alphabet, and a two-link arm skeleton that detects hand location and ...
Comments