Everyday Eye Contact Detection Using Unsupervised Gaze Target Discovery

ABSTRACT

Eye contact is an important non-verbal cue in social signal processing and promising as a measure of overt attention in human-object interactions and attentive user interfaces. However, robust detection of eye contact across different users, gaze targets, camera positions, and illumination conditions is notoriously challenging. We present a novel method for eye contact detection that combines a state-of-the-art appearance-based gaze estimator with a novel approach for unsupervised gaze target discovery, i.e. without the need for tedious and time-consuming manual data annotation. We evaluate our method in two real-world scenarios: detecting eye contact at the workplace, including on the main work display, from cameras mounted to target objects, as well as during everyday social interactions with the wearer of a head-mounted egocentric camera. We empirically evaluate the performance of our method in both scenarios and demonstrate its effectiveness for detecting eye contact independent of target object type and size, camera position, and user and recording environment.

References

1. Michael F Land and Mary Hayhoe. In what ways do eye movements contribute to everyday activities' Vision research, 41(25):3559--3565, 2001.
2. Chris L Kleinke. Gaze and eye contact: a research review. Psychological bulletin, 100(1):78, 1986.
3. Jeffrey S Shell, Ted Selker, and Roel Vertegaal. Interacting with groups of computers. Communications of the ACM, 46(3):40--46, 2003.
4. Zhefan Ye, Yin Li, Alireza Fathi, Yi Han, Agata Rozga, Gregory D Abowd, and James M Rehg. Detecting eye contact using wearable eye-tracking glasses. In Proceedings of the 2012 ACM conference on ubiquitous computing, pages 699--704. ACM, 2012.
5. Connor Dickie, Roel Vertegaal, David Fono, Changuk Sohn, Daniel Chen, Daniel Cheng, Jeffrey S Shell, and Omar Aoudeh. Augmenting and sharing memory with eyeblog. In Proceedings of the the 1st ACM workshop on Continuous archival and retrieval of personal experiences, pages 105--109. ACM, 2004.
6. Michita Imai, Tetsuo Ono, and Hiroshi Ishiguro. Physical relation and expression: Joint attention for human-robot interaction. IEEE Transactions on Industrial Electronics, 50(4):636--643, 2003.
7. Jeffrey S Shell, Roel Vertegaal, Daniel Cheng, Alexander W Skaburskis, Changuk Sohn, A James Stewart, Omar Aoudeh, and Connor Dickie. Ecsglasses and eyepliances: using attention to open sociable windows of interaction. In Proceedings of the 2004 symposium on Eye tracking research & applications, pages 93--100. ACM, 2004.
8. Yanxia Zhang, Ming Ki Chong, Jörg Müller, Andreas Bulling, and Hans Gellersen. Eye tracking for public displays in the wild. Personal and Ubiquitous Computing, 19(5--6):967--981, 2015.
9. Xucong Zhang, Yusuke Sugano, Mario Fritz, and Andreas Bulling. It's written all over your face: Full-face appearance-based gaze estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017.
10. Yusuke Sugano, Xucong Zhang, and Andreas Bulling. Aggregaze: Collective estimation of audience attention on public displays. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology, pages 821--831. ACM, 2016.
11. Jeffrey S Shell, Roel Vertegaal, and Alexander W Skaburskis. Eyepliances: attention-seeking devices that respond to visual attention. In CHI'03 extended abstracts on Human factors in computing systems, pages 770--771. ACM, 2003.
12. Roel Vertegaal, Connor Dickie, Changuk Sohn, and Myron Flickner. Designing attentive cell phone using wearable eyecontact sensors. In CHI'02 extended abstracts on Human factors in computing systems, pages 646--647. ACM, 2002.
13. Sarah R Edmunds, Agata Rozga, Yin Li, Elizabeth A Karp, Lisa V Ibanez, James M Rehg, and Wendy L Stone. Brief report: Using a point-of-view camera to measure eye gaze in young children with autism spectrum disorder during naturalistic social interactions: A pilot study. Journal of Autism and Developmental Disorders, pages 1--7, 2017.
14. Brian A Smith, Qi Yin, Steven K Feiner, and Shree K Nayar. Gaze locking: passive eye contact detection for human-object interaction. In Proceedings of the 26th annual ACM symposium on User interface software and technology, pages 271--280. ACM, 2013.
15. Zhefan Ye, Yin Li, Yun Liu, Chanel Bridges, Agata Rozga, and James M Rehg. Detecting bids for eye contact using a wearable camera. In Automatic Face and Gesture Recognition (FG), 2015 11th IEEE International Conference and Workshops on, volume 1, pages 1--8. IEEE, 2015.
16. Antje Nuthmann and John M Henderson. Object-based attentional selection in scene viewing. Journal of vision, 10(8):20--20, 2010.
17. Xucong Zhang, Yusuke Sugano, Mario Fritz, and Andreas Bulling. Appearance-based gaze estimation in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4511--4520, 2015.
18. Paul P Maglio, Rob Barrett, Christopher S Campbell, and Ted Selker. Suitor: An attentive information system. In Proceedings of the 5th international conference on Intelligent user interfaces, pages 169--176. ACM, 2000.
19. Paul P Maglio, Teenie Matlock, Christopher S Campbell, Shumin Zhai, and Barton A Smith. Gaze and speech in attentive user interfaces. In Advances in Multimodal Interfaces ICMI 2000, pages 1--7. Springer, 2000.
20. Alice Oh, Harold Fox, Max Van Kleek, Aaron Adler, Krzysztof Gajos, Louis-Philippe Morency, and Trevor Darrell. Evaluating look-to-talk: a gaze-aware interface in a collaborative environment. In CHI'02 Extended Abstracts on Human Factors in Computing Systems, pages 650--651. ACM, 2002.
21. Eric Horvitz, Carl Kadie, Tim Paek, and David Hovel. Models of attention in computing and communication: from principles to applications. Communications of the ACM, 46(3):52--59, 2003.
22. Roel Vertegaal and Jeffrey S Shell. Attentive user interfaces: the surveillance and sousveillance of gaze-aware objects. Social Science Information, 47(3):275--298, 2008.
23. Frederik Brudy, David Ledo, Saul Greenberg, and Andreas Butz. Is anyone looking? mitigating shoulder surfing on public displays through awareness and protection. In Proceedings of The International Symposium on Pervasive Displays, page 1. ACM, 2014.
24. Kevin Smith, Sileye O Ba, Jean-Marc Odobez, and Daniel Gatica-Perez. Tracking the visual focus of attention for a varying number of wandering people. IEEE transactions on pattern analysis and machine intelligence, 30(7):1212--1229, 2008.
25. Dan Witzner Hansen and Qiang Ji. In the eye of the beholder: A survey of models for eyes and gaze. IEEE transactions on pattern analysis and machine intelligence, 32(3):478--500, 2010.
26. Carlos Hitoshi Morimoto, Arnon Amir, and Myron Flickner. Detecting eye position and gaze from a single camera and 2 light sources. In Pattern Recognition, 2002. Proceedings. 16th International Conference on, volume 4, pages 314--317. IEEE, 2002.
27. Zhiwei Zhu and Qiang Ji. Eye gaze tracking under natural head movements. In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, volume 1, pages 918--923. IEEE, 2005.
28. Zhiwei Zhu, Qiang Ji, and Kristin P Bennett. Nonlinear eye gaze mapping function estimation via support vector regression. In Pattern Recognition, 2006. ICPR 2006. 18th International Conference on, volume 1, pages 1132--1135. IEEE, 2006.
29. Dan Witzner Hansen and Arthur EC Pece. Eye tracking in the wild. Computer Vision and Image Understanding, 98(1):155--181, 2005.
30. Feng Lu, Yusuke Sugano, Takahiro Okabe, and Yoichi Sato. Inferring human gaze from appearance via adaptive linear regression. In Computer Vision (ICCV), 2011 IEEE International Conference on, pages 153--160. IEEE, 2011.
31. Erroll Wood and Andreas Bulling. Eyetab: Model-based gaze estimation on unmodified tablet computers. In Proceedings of the Symposium on Eye Tracking Research and Applications, pages 207--210. ACM, 2014.
32. Jixu Chen, Qiang Ji, et al. A probabilistic approach to online eye gaze tracking without personal calibration. IEEE Transactions on Image Processing, 2014.
33. Yusuke Sugano, Yasuyuki Matsushita, and Yoichi Sato. Appearance-based gaze estimation using visual saliency. IEEE transactions on pattern analysis and machine intelligence, 35(2):329--341, 2013.
34. Michael Xuelin Huang, Tiffany CK Kwok, Grace Ngai, Stephen CF Chan, and Hong Va Leong. Building a personalized, auto-calibrating eye tracker from user interactions. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pages 5169--5179. ACM, 2016.
35. Yusuke Sugano, Yasuyuki Matsushita, Yoichi Sato, and Hideki Koike. Appearance-based gaze estimation with online calibration from mouse operations. IEEE Transactions on Human-Machine Systems, 45(6):750--760, 2015.
36. Ben Benfold and Ian Reid. Unsupervised learning of a scene-specific coarse gaze estimator. In Computer Vision (ICCV), 2011 IEEE International Conference on, pages 2344--2351. IEEE, 2011.
37. Isarun Chamveha, Yusuke Sugano, Daisuke Sugimura, Teera Siriteerakul, Takahiro Okabe, Yoichi Sato, and Akihiro Sugimoto. Head direction estimation from low resolution images with scene adaptation. Computer Vision and Image Understanding, 117(10):1502--1511, 2013.
38. Stefan Duffner and Christophe Garcia. Visual focus of attention estimation with unsupervised incremental learning. IEEE Transactions on Circuits and Systems for Video Technology, 26(12):2264--2272, 2016.
39. Ted Selker, Andrea Lockerd, and Jorge Martinez. Eye-r, a glasses-mounted eye motion detection interface. In CHI'01 extended abstracts on Human factors in computing systems, pages 179--180. ACM, 2001.
40. Connor Dickie, Roel Vertegaal, Jeffrey S Shell, Changuk Sohn, Daniel Cheng, and Omar Aoudeh. Eye contact sensing glasses for attention-sensitive wearable video blogging. In CHI'04 extended abstracts on Human factors in computing systems, pages 769--770. ACM, 2004.
41. John D Smith, Roel Vertegaal, and Changuk Sohn. Viewpointer: lightweight calibration-free eye tracking for ubiquitous handsfree deixis. In Proceedings of the 18th annual ACM symposium on User interface software and technology, pages 53--61. ACM, 2005.
42. Adria Recasens, Aditya Khosla, Carl Vondrick, and Antonio Torralba. Where are they looking? In Advances in Neural Information Processing Systems, pages 199--207, 2015.
43. Adrià Recasens, Carl Vondrick, Aditya Khosla, and Antonio Torralba. Following gaze across views. arXiv preprint arXiv:1612.03094, 2016.
44. Kenneth Alberto Funes Mora, Florent Monay, and Jean-Marc Odobez. Eyediap: A database for the development and evaluation of gaze estimation algorithms from rgb and rgb-d cameras. In Proceedings of the Symposium on Eye Tracking Research and Applications, pages 255--258. ACM, 2014.
45. Davis E. King. Dlib-ml: A machine learning toolkit. Journal of Machine Learning Research, 10:1755--1758, 2009.
46. Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency. Openface: an open source facial behavior analysis toolkit. In Applications of Computer Vision (WACV), 2016 IEEE Winter Conference on, pages 1--10. IEEE, 2016.
47. Mihael Ankerst, Markus M Breunig, Hans-Peter Kriegel, and Jörg Sander. Optics: ordering points to identify the clustering structure. In ACM Sigmod record, volume 28, pages 49--60. ACM, 1999.
48. Petros Xanthopoulos and Talayeh Razzaghi. A weighted support vector machine method for control chart pattern recognition. Computers & Industrial Engineering, 70:134--149, 2014.
49. Rainer Stiefelhagen, Jie Yang, and Alex Waibel. Modeling focus of attention for meeting indexing based on multiple cues. IEEE Transactions on Neural Networks, 13(4):928--938, 2002.
50. Michael Voit and Rainer Stiefelhagen. Deducing the visual focus of attention from head pose estimation in dynamic multi-view meeting scenarios. In Proceedings of the 10th international conference on Multimodal interfaces, pages 173--180. ACM, 2008.
51. ByungIn Yoo, Jae-Joon Han, Changkyu Choi, Kwonju Yi, Sungjoo Suh, Dusik Park, and Changyeong Kim. 3d user interface combining gaze and hand gestures for large-scale display. In CHI'10 Extended Abstracts on Human Factors in Computing Systems, pages 3709--3714. ACM, 2010.

Index Terms

• Human-centered computing