PupilNet, Measuring Task Evoked Pupillary Response using Commodity RGB Tablet Cameras: Comparison to Mobile, Infrared Gaze Trackers for Inferring Cognitive Load

Abstract

Pupillary diameter monitoring has been proven successful at objectively measuring cognitive load that might otherwise be unobservable. This paper compares three different algorithms for measuring cognitive load using commodity cameras. We compare the performance of modified starburst algorithm (from previous work) and propose two new algorithms: 2 Level Snakuscules and a convolutional neural network which we call PupilNet. In a user study with eleven participants, our comparisons show PupilNet outperforms other algorithms in measuring pupil dilation, is robust to various lighting conditions, and robust to different eye colors. We show that the difference between PupilNet and a gold standard head-mounted gaze tracker varies only from -2.6% to 2.8%. Finally, we also show that PupilNet gives similar conclusions about cognitive load during a longer duration typing task.

References

1. 2015. Tobii Pro Glasses 2 wearable eye tracker. (Jun 2015). https://www.tobiipro.com/product-listing/tobii-pro-glasses-2/
2. Alexandra Branzan Albu, Ben Widsten, Tiange Wang, Julie Lan, and Jordana Mah. 2008. A computer vision-based system for real-time detection of sleep onset in fatigued drivers. In 2008 IEEE Intelligent Vehicles Symposium. IEEE, 25--30. https://doi.org/10.1109/IVS.2008.4621133
3. Gary Aston-Jones and Jonathan D Cohen. 2005. An Integrative Theory of Locus Coeruleus-Norepinephrine Function: Adaptive Gain and Optimal Performance. Annual review of neuroscience 28 (2005), 403--50. https://doi.org/10.1146/
4. Gary Aston-Jones, Janusz Rajkowski, Piotr Kubiak, and Tatiana Alexinsky. 1994. Locus coeruleus neurons in monkey are selectively activated by attended cues in vigilance tasks. Journal of Neuroscience 14 (1994), 4467--4480.
5. Tadas Baltrušaitis, Peter Robinson, and Louis Philippe Morency. 2013. Constrained local neural fields for robust facial landmark detection in the wild. Proceedings of the IEEE International Conference on Computer Vision (2013), 354--361. https://doi.org/10.1109/ICCVW.2013.54
6. Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency. 2016. OpenFace: an open source facial behavior analysis toolkit. In IEEE Winter Conference on Applications of Computer Vision.
7. Jackson Beatty. 1982. Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychological bulletin 91, 2 (1982), 276--292. https://doi.org/10.1037/0033-2909.91.2.276
8. Jackson Beatty. 1982. Task-Evoked Pupillary Responses, Processing Load, and the Structure of Processing Resources. (1982), 276--292 pages.
9. Jackson Beatty and Brennis Lucero-Wagoner. 2000. The pupillary system. 142--162 pages. http://prx.library.gatech.edu/login?url=http://search.ebscohost.com/login.aspx?direct=true&db=psyh&AN=2000-03927-005&site=ehost-live
10. Craig W Berridge and Barry D Waterhouse. 2003. The locus coeruleusâĂŞnoradrenergic system: modulation of behavioral state and state-dependent cognitive processes. Brain Research Reviews 42, 1 (2003), 33--84. https://doi.org/10.1016/S0165-0173(03)00143-7
11. Zhijian Chen and Nelson Cowan. 2005. Chunk limits and length limits in immediate recall: a reconciliation. Journal of experimental psychology. Learning, memory, and cognition 31, 6 (11 2005), 1235--49. https://doi.org/10.1037/0278-7393.31.6.1235
12. John Daugman. 2004. How Iris Recognition Works. In IEEE Transactions on Circuits and Systems for Video Technology. Vol. 14. https://doi.org/10.1109/TCSVT.2003.818350
13. Pan Du, Warren A Kibbe, and Simon M Lin. 2006. Improved peak detection in mass spectrum by incorporating continuous wavelet transform-based pattern matching. Bioinformatics 22, 17 (2006), 2059--2065.
14. Maria K. Eckstein, BelÃľn Guerra-Carrillo, Alison T. Miller Singley, and Silvia A. Bunge. 2017. Beyond eye gaze: What else can eyetracking reveal about cognition and cognitive development? Developmental Cognitive Neuroscience 25 (2017), 69--91. https://doi.org/10.1016/j.dcn.2016.11.001
15. Wolfgang Fuhl, Thomas Kübler, Katrin Sippel, Wolfgang Rosenstiel, and Enkelejda Kasneci. 2015. Excuse: Robust pupil detection in real-world scenarios. In International Conference on Computer Analysis of Images and Patterns. Springer, 39--51.
16. Kunihiko Fukushima and Sei Miyake. 1982. Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition. In Competition and cooperation in neural nets. Springer, 267--285.
17. Sanyam Garg, Abhinav Tripathi, and Edward Cutrell. 2016. Accurate eye center localization using Snakuscule. 2016 IEEE Winter Conference on Applications of Computer Vision, WACV 2016 (2016). https://doi.org/10.1109/WACV.2016.7477673
18. Alaa Hilal, Bassam Daya, and Pierre Beauseroy. [n. d.]. Hough Transform and Active Contour for Enhanced Iris Segmentation. ([n. d.]). https://www.ijcsi.org/papers/IJCSI-9-6-2-1-10.pdf
19. Qiong Huang, Ashok Veeraraghavan, and Ashutosh Sabharwal. 2015. TabletGaze: unconstrained appearance-based gaze estimation in mobile tablets. arXiv preprint arXiv:1508.01244 (2015).
20. Shamsi T Iqbal, Xianjun Sam Zheng, and Brian P Bailey. 2004. Task-evoked pupillary response to mental workload in human-computer interaction. Extended abstracts of the 2004 conference on Human factors and computing systems CHI 04 (2004), 1477. https://doi.org/10.1145/985921.986094
21. Amir-Homayoun Javadi, Zahra Hakimi, Morteza Barati, Vincent Walsh, and Lili Tcheang. 2015. SET: a pupil detection method using sinusoidal approximation. Frontiers in neuroengineering 8 (2015).
22. John S Kafka. 2016. Psychoanalysis and the Temporal Trace. Time and Trace: Multidisciplinary Investigations of Temporality (2016), 197.
23. Daniel Kahneman and Jackson Beatty. 1966. Pupil Diameter and Load on Memory. Source: Science, New Series 154, 3756 (12 1966), 1583--1585. http://www.jstor.org/stable/1720478http://www.jstor.org.proxy.libraries.smu.edu/stable/pdfplus/10.2307/1720478.pdf?acceptTC=truehttp://about.jstor.org/terms
24. Koray Kara, Dursun Karaman, Uzeyir Erdem, Mehmet Ayhan Congologlu, Ibrahim Durukan, and Abdullah Ilhan. 2013. Investigation of Autonomic Nervous System Functions by Pupillometry in Children with Attention-Deficit/ Hyperactivity Disorder Investigation of autonomic nervous system functions by pupillometry in children with Attention-Deficit/Hyperactivity Disorder. Bulletin of Clinical Psychopharmacology 23, 1 (2013). https://doi.org/10.5455/bcp.20121130085850
25. Canan Karatekin, David J Marcus, J W Couperous, and Jane W Couperus. 2007. Regulations of cognitive resources during sustained attention and working memory in 10-year-olds and adults. Psychophysiology 44, 1 (1 2007), 128--144. https://doi.org/10.1111/j.1469-8986.2006.00477.x
26. Michael Kass, Andrew Witkin, and Demetri Terzopoulos. 1988. Snakes: Active contour models. International journal of computer vision 1, 4 (1988), 321--331.
27. Vahid Kazemi and Josephine Sullivan. 2014. One Millisecond Face Alignment with an Ensemble of Regression Trees. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. https://doi.org/10.13140/2.1.1212.2243
28. Diederik P. Kingma and Jimmy Lei Ba. 2015. Adam:. International Conference on Learning Representations (ICLR2015) (12 2015). https://doi.org/10.1145/1830483.1830503
29. Jeff Klingner. 2010. Measuring cognitive load during visual tasks by combining pupillometry and eye tracking. Perspective May (2010), 130.
30. Jeff Klingner, Rakshit Kumar, and Pat Hanrahan. 2008. Measuring the task-evoked pupillary response with a remote eye tracker. Proceedings of the 2008 symposium on Eye tracking research 8 applications - ETRA ‘08 1, 212 (2008), 69. https://doi.org/10.1145/1344471.1344489
31. Jaehan Koh, Venu Govindaraju, and Vipin Chaudhary. [n. d.]. A Robust Iris Localization Method Using an Active Contour Model and Hough Transform. ([n. d.]). https://pdfs.semanticscholar.org/4709/a9e2920f4083264f04e94c71463b528af128.pdf
32. Bruno Laeng, Marte Ørbo, Terje Holmlund, and Michele Miozzo. 2011. Pupillary Stroop effects. Cognitive processing 12, 1 (2 2011), 13--21. https://doi.org/10.1007/s10339-010-0370-z
33. Daniel Lafond, René Proulx, Alexis Morris, William Ross, Alexandre Bergeron-Guyard, and Mihaela Ulieru. 2014. Hci dilemmas for context-aware support in intelligence analysis. In Adapt. 2014, Sixth Int. Conf. Adapt. Self-Adaptive Syst. Appl. 68--72.
34. Yann LeCun et al. 1989. Generalization and network design strategies. Connectionism in perspective (1989), 143--155.
35. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278--2324.
36. Dongheng Li, David Winfield, and Derrick J Parkhurst. 2012. Starburst: A hybrid algorithm for video-based eye tracking combining feature-based and model-based approaches. (2012). https://pdfs.semanticscholar.org/db1d/7f94e91feea0a0e0b2f4563f2d05b0338732.pdf
37. Irene E. Loewenfeld. 1993. The pupil: Anatomy, physiology, and clinical applications. Wayne State University Press. Google Scholar, Detroit, MI.
38. George A. Miller. 1956. The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychological Review 63, 2 (1956), 81--97. https://doi.org/10.1037/h0043158
39. Shwetak Patel. 2008. Infrastructure Mediated Sensing. August (2008), 274. http://hdl.handle.net/1853/24829
40. Ken Pfeuffer, Jason Alexander, and Hans Gellersen. 2016. Partially-indirect Bimanual Input with Gaze, Pen, and Touch for Pan, Zoom, and Ink Interaction. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (2016), 2845--2856. https://doi.org/10.1145/2858036.2858201
41. Jan L Plass, Roxana Moreno, and Roland Brünken. 2010. Cognitive Load Theory. Vol. 55. 286 pages. https://doi.org/10.1016/B978-0-12-387691-1.00002-8 arXiv:arXiv:1011.1669v3
42. Sohail Rafiqi, Chatchai Wangwiwattana, Ephrem Fernandez, Suku Nair, and Eric C. Larson. 2015. Work-in-progress, PupilWare-M: Cognitive load estimation using unmodified smartphone cameras. In Proceedings - 2015 IEEE 12th International Conference on Mobile Ad Hoc and Sensor Systems, MASS 2015. https://doi.org/10.1109/MASS.2015.31
43. Sohail Rafiqi, Chatchai Wangwiwattana, Jasmine Kim, Ephrem Fernandez, Suku Nair, and Eric C. Larson. 2015. PupilWare: Towards pervasive cognitive load measurement using commodity devices. In 8th ACM International Conference on PErvasive Technologies Related to Assistive Environments, PETRA 2015 - Proceedings. https://doi.org/10.1145/2769493.2769506
44. Gerulf Rieger and Ritch C Savin-Williams. 2012. The eyes have it: sex and sexual orientation differences in pupil dilation patterns. PloS one 7, 8 (1 2012), e40256. https://doi.org/10.1371/journal.pone.0040256
45. Kaushik Roy, Prabir Bhattacharya, and Ching Y Suen. 2010. Unideal Iris Segmentation Using Region-Based Active Contour Model. LNCS 6112 (2010), 256--265. https://pdfs.semanticscholar.org/a5da/0a5fbfe89bd678d099c504a7d94bce955019.pdf
46. Wayne J. Ryan, Damon L. Woodard, Andrew T. Duchowski, and Stan T. Birchfield. 2008. Adapting Starburst for Elliptical Iris Segmentation. In 2008 IEEE Second International Conference on Biometrics: Theory, Applications and Systems. IEEE, 1--7. https://doi.org/10.1109/BTAS.2008.4699340
47. Lech Świrski, Andreas Bulling, and Neil Dodgson. 2012. Robust real-time pupil tracking in highly off-axis images. In Proceedings of the Symposium on Eye Tracking Research and Applications. ACM, 173--176.
48. Philippe Thevenaz and Michael Unser. 2006. The Snakuscule. 2006 International Conference on Image Processing (2006), 1633--1636. https://doi.org/10.1109/ICIP.2006.312658
49. Warren Tryon W. 1975. Pupillometry: A Survey of Sources of Variation. Psychophysiology 12 (1975). https://doi.org/10.1111/j.1469-8986.1975.tb03068.x
50. Alex Waibel, Toshiyuki Hanazawa, Geoffrey Hinton, Kiyohiro Shikano, and Kevin J Lang. 1989. Phoneme recognition using time-delay neural networks. IEEE transactions on acoustics, speech, and signal processing 37, 3 (1989), 328--339.
51. Richard P Wildes, Jane C Asmuth, Gilbert L Green, Stephen C Hsu, Raymond J Kolczynski, James R Matey, Sterling E McBride, Richard P Wildes, Jane C Asmuth, Gilbert L Green, Stephen C Hsu, Raymond J Kolczynski, James R Matey, and Sterling E McBride. 1994. A system for automated iris recognition. In Applications of Computer Vision, 1994., Proceedings of the Second IEEE Workshop on. IEEE, IEEE Comput. Soc. Press, 121--128. https://doi.org/10.1109/ACV.1994.341298
52. Erroll Wood and Andreas Bulling. 2014. EyeTab: Model-based gaze estimation on unmodified tablet computers. In Proceedings of the Symposium on Eye Tracking Research and Applications. 207--210.
53. Jie Xu, Yang Wang, Fang Chen, and Eric Choi. 2011. Pupillary Response Based Cognitive Workload Measurement under Luminance Changes. 178--185. https://doi.org/10.1007/978-3-642-23771-3{_}14
54. Beste F Yuksel, Kurt B Oleson, Lane Harrison, Evan M Peck, Daniel Afergan, Remco Chang, and Robert J K Jacob. 2016. Learn Piano with BACh: An Adaptive Learning Interface that Adjusts Task Difficulty based on Brain State. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (2016), 5372--5384. https://doi.org/10.1145/2858036.2858388

Index Terms

• Computer systems organization

  • Embedded and cyber-physical systems

    • Sensors and actuators

• Computing methodologies

  • Machine learning

    • Machine learning approaches

      • Neural networks

• Human-centered computing

  • Ubiquitous and mobile computing