Improving accuracy in face tracking user interfaces using consumer devices

ABSTRACT

Using face and head movements to control a computer can be especially helpful for users who, for various reasons, cannot effectively use common input devices with their hands. Using vision-based consumer devices makes such a user interface readily available and allows its use to be non-intrusive. However, a characteristic problem with this system is accurate control. Consumer devices capture already small face movements at a resolution that is usually lower than the screen resolution. Computer vision algorithms and technologies that enable such also introduce noise, adversely affecting usability. This paper describes how different components of this perceptual user interface contribute to the problem of accuracy and presents potential solutions. This interface was implemented with different configurations and was statistically evaluated to support the analysis. The different configurations include, among other things, the use of 2D and depth images from consumer devices, different input styles, and the use of the Kalman filter.

References

1. ALDOMA, A. Progress on head detection and pose estimation (II). http://pointclouds.org/blog/hrcs/ (Last Accessed: July 2012)
2. BETKE, M., GIPS, J., et al., 2002. The Camera Mouse: visual tracking of body features to provide computer access for people with severe disabilities. Neural Systems and Rehabilitation Engineering, IEEE Transactions on 10, 1, 1--10.
3. BREIMAN, L., 2001. Random Forests. Mach. Learn. 45, 1, 5--32. DOI= http://dx.doi.org/10.1023/a:1010933404324.
4. CHATHURANGA, S.K., SAMARAWICKRAMA, K.C., et al., 2010. Hands free interface for Human Computer Interaction. In Information and Automation for Sustainability (ICIAFs), 2010 5th International Conference on, 359--364.
5. FANELLI, G., GALL, J., et al., 2012. Real Time 3D Head Pose Estimation: Recent Achievements and Future Challenges. In Communications, Control and Signal Processing, 2012. ISCCSP 2012. 5th International Symposium on.
6. FANELLI, G., WEISE, T., et al., 2011. Real Time Head Pose Estimation from Consumer Depth Cameras. In DAGM'11, Frankfurt, Germany.
7. GORODNICHY, D., 2006. Perceptual Cursor - A Solution to the Broken Loop Problem in Vision-Based Hands-Free Computer Control Devices.
8. GORODNICHY, D.O., 2002. On importance of nose for face tracking. In Automatic Face and Gesture Recognition, 2002. Proceedings. Fifth IEEE International Conference on, 181--186. DOI=http://dx.doi.org/10.1109/afgr.2002.1004153.
9. GORODNICHY, D.O. and ROTH, G., 2004. Nouse 'use your nose as a mouse' perceptual vision technology for hands-free games and interfaces. Image and Vision Computing 22, 12, 931--942. DOI= http://dx.doi.org/10.1016/j.imavis.2004.03.021.
10. HEIKKIL, H., #228, et al., 2012. Simple gaze gestures and the closure of the eyes as an interaction technique. In Proceedings of the Proceedings of the Symposium on Eye Tracking Research and Applications (Santa Barbara, California2012), ACM, 2168579, 147--154. DOI= http://dx.doi.org/10.1145/2168556.2168579.
11. HUNKE, M. and WAIBEL, A., 1994. Face locating and tracking for human-computer interaction. In Signals, Systems and Computers, 1994. 1994 Conference Record of the Twenty-Eighth Asilomar Conference on, 1277--1281 vol.1272. DOI= http://dx.doi.org/10.1109/acssc.1994.471664.
12. ISTANCE, H., BATES, R., et al., 2008. Snap Clutch, a Moded Approach to Solving the Midas Touch Problem. In Proceedings of the Eye Tracking Research and Applications Symposium 2008 ACM, 221--228. DOI= http://dx.doi.org/citeulike-article-id:2603228.
13. KONDORI, F.A., YOUSEFI, S., et al., 2011. 3D head pose estimation using the Kinect. In Wireless Communications and Signal Processing (WCSP), 2011 International Conference on, 1--4. DOI= http://dx.doi.org/10.1109/wcsp.2011.6096866.
14. LUNT, B., EKSTROM, J., et al. IT 2008. Communications of the ACM 53, 12, 133.
15. MICROSOFT Kinect for Windows Human Interface Guidelines v1.5.0. 2012. http://www.microsoft.com/en-us/kinectforwindows/develop/learn.aspx (Last Accessed: May 2012)
16. MING-HSUAN, Y., KRIEGMAN, D.J., et al., 2002. Detecting faces in images: a survey. Pattern Analysis and Machine Intelligence, IEEE Transactions on 24, 1, 34--58. DOI= http://dx.doi.org/10.1109/34.982883.
17. MORRIS, T. and CHAUHAN, V., 2006. Facial feature tracking for cursor control. Journal of Network and Computer Applications 29, 1, 62--80. DOI= http://dx.doi.org/10.1016/j.jnca.2004.07.003.
18. MUCHUN, S., CHINYEN, Y., et al., 2008. An implementation of an eye-blink-based communication aid for people with severe disabilities. In Audio, Language and Image Processing, 2008. ICALIP 2008. International Conference on, 351--356.
19. MURPHY-CHUTORIAN, E. and TRIVEDI, M.M., 2009. Head Pose Estimation in Computer Vision: A Survey. Pattern Analysis and Machine Intelligence, IEEE Transactions on 31, 4, 607--626. DOI= http://dx.doi.org/10.1109/tpami.2008.106.
20. PORTA, M. and TURINA, M., 2008. Eye-S: a full-screen input modality for pure eye-based communication. In Proceedings of the Proceedings of the 2008 Symposium on Eye Tracking Research and Applications (Savannah, Georgia2008), ACM, 1344477, 27--34. DOI= http://dx.doi.org/10.1145/1344471.1344477.
21. PRIMESENSE The PrimeSense 3D Awareness Sensor. PrimeSense Ltd, http://primesense.com/press-room/resources/file/4-primesense-3d-sensor-data-sheet?lang=en (Last Accessed: May 2012)
22. SEEING MACHINES faceAPI. http://www.seeingmachines.com/product/faceapi/ (Last Accessed: May 2012)
23. VARONA, J., MANRESA-YEE, C., et al., 2008. Hands-free vision-based interface for computer accessibility. Journal of Network and Computer Applications 31, 4, 357--374. DOI= http://dx.doi.org/10.1016/j.jnca.2008.03.003.
24. VIOLA, P. and JONES, M., 2001. Rapid object detection using a boosted cascade of simple features. In Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on, I-511-I-518 vol.511.
25. WANG, J.-G. and SUNG, E., 2007. EM enhancement of 3D head pose estimated by point at infinity. Image and Vision Computing 25, 12, 1864--1874. DOI= http://dx.doi.org/10.1016/j.imavis.2005.12.017.
26. WEISE, T., BOUAZIZ, S., et al., 2011. Realtime performance-based facial animation. ACM Trans. Graph. 30, 4, 1--10. DOI= http://dx.doi.org/10.1145/2010324.1964972.
27. WEISE, T., WISMER, T., et al., 2009. In-hand scanning with online loop closure. In Computer Vision Workshops (ICCV Workshops), 2009 IEEE 12th International Conference on, 1630--1637. DOI= http://dx.doi.org/10.1109/iccvw.2009.5457479.
28. YILMAZ, A., JAVED, O., et al., 2006. Object tracking: A survey. ACM Comput. Surv. 38, 4, 13. DOI= http://dx.doi.org/10.1145/1177352.1177355.
29. YUN, F. and HUANG, T.S., 2006. Graph embedded analysis for head pose estimation. In Automatic Face and Gesture Recognition, 2006. FGR 2006. 7th International Conference on, 6 pp.-8. DOI= http://dx.doi.org/10.1109/fgr.2006.60.
30. ZHANG, C. and ZHANG, Z. A Survey of Recent Advances in Face Detection. Microsoft Research, 2010. http://research.microsoft.com/apps/pubs/default.aspx?id=132077 (Last Accessed: June 2012)

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

        • Object recognition

        • Tracking

      • Computer vision tasks

        • Scene understanding

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction devices

      • Touch screens

    • Interactive systems and tools

• Social and professional topics

  • Professional topics

    • Computing profession

      • Assistive technologies

  • User characteristics

    • People with disabilities