iLid: Low-power Sensing of Fatigue and Drowsiness Measures on a Computational Eyeglass

Abstract

The ability to monitor eye closures and blink patterns has long been known to enable accurate assessment of fatigue and drowsiness in individuals. Many measures of the eye are known to be correlated with fatigue including coarse-grained measures like the rate of blinks as well as fine-grained measures like the duration of blinks and the extent of eye closures. Despite a plethora of research validating these measures, we lack wearable devices that can continually and reliably monitor them in the natural environment. In this work, we present a low-power system, iLid, that can continually sense fine-grained measures such as blink duration and Percentage of Eye Closures (PERCLOS) at high frame rates of 100fps. We present a complete solution including design of the sensing, signal processing, and machine learning pipeline; implementation on a prototype computational eyeglass platform; and extensive evaluation under many conditions including illumination changes, eyeglass shifts, and mobility. Our results are very encouraging, showing that we can detect blinks, blink duration, eyelid location, and fatigue-related metrics such as PERCLOS with less than a few percent error.

References

1. 2007. SensoMotoric Instruments (SMI): Eye and gaze tracking systems. (28 Nov. 2007). http://www.smivision.com/en.html.
2. 2014. Current Centeye Vision Chips. http://www.centeye.com/. (22 Feb. 2014).
3. 2014. OmniVision: CMOS VGA (640x480) Image Sensor. (27 Feb. 2014). http://www.ovt.com/products/sensor.php?id=146.
4. 2015. JINS MEME Eyewear. (14 Oct. 2015). https://jins-meme.com/en/.
5. 2015. STM3232-bitARMCortex MCUs - STMicroelectronics. (24 Mar. 2015). Retrieved September 10, 2016 from http://www.st.com/stm32.
6. 2015. Tobii Glasses: Mobile Eye Tracker for real world research. (15 Oct. 2015). http://www.tobii.com.
7. 2016. Research on Drowsy Driving. (23 Jan. 2016). http://www.nhtsa.gov/Driving+Safety/Drowsy+Driving/scope-of-the-problem.
8. 2016. SMI Eye Tracking Glasses. (2016). http://www.eyetracking-glasses.com/products/eye-tracking-glasses-2-wireless/technology/.
9. 2017. Center of Excellence for Mobile Sensor Data-to-Knowledge. (2017). https://md2k.org.
10. Saeed Abdullah, Elizabeth L Murnane, Mark Matthews, Matthew Kay, Julie A Kientz, Geri Gay, and Tanzeem Choudhury. 2016. Cognitive rhythms: unobtrusive and continuous sensing of alertness using a mobile phone. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 178--189.
11. Christer Ahlstrom, Marcus Nyström, Kenneth Holmqvist, Carina Fors, David Sandberg, Anna Anund, Göran Kecklund, and Torbjörn Åkerstedt. 2013. Fit-for-duty test for estimation of drivers? sleepiness level: eye movements improve the sleep/wake predictor. Transportation research part C: emerging technologies 26 (2013), 20--32.
12. Rizwan Ahmad. 2016. Understanding the Language of the Eye: Detecting and Identifying Eye Events in Real Time via Electrooculography. (2016). http://www.escholarship.org/uc/item/53n1p2z5
13. Tobias Appel, Thiago Santini, and Enkelejda Kasneci. 2016. Brightness-and motion-based blink detection for head-mounted eye trackers. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct. ACM, 1726--1735.
14. Ioana Bacivarov, Mircea Ionita, and Peter Corcoran. 2008. Statistical models of appearance for eye tracking and eye-blink detection and measurement. IEEE transactions on consumer electronics 54, 3 (2008).
15. P. L. Benitez, G. H. Kamimori, T. J. Balkin, A. Greene, and M. L. Johnson. 2009. Modeling fatigue over sleep deprivation, circadian rhythm, and caffeine with a minimal performance inhibitor model. Meth. Enzymol. 454 (2009), 405--421.
16. Luis Miguel Bergasa, Jesús Nuevo, Miguel A Sotelo, Rafael Barea, and María Elena Lopez. 2006. Real-time system for monitoring driver vigilance. IEEE Transactions on Intelligent Transportation Systems 7, 1 (2006), 63--77.
17. Serge Boverie and Alain Giralt. 2008. Driver vigilance diagnostic based on eyelid movement observation. IFAC Proceedings Volumes 41, 2 (2008), 12831--12836.
18. J. E. Bower. 2014. Cancer-related fatigue--mechanisms, risk factors, and treatments. Nat Rev Clin Oncol 11, 10 (Oct 2014), 597--609.
19. Andreas Bulling, Daniel Roggen, and Gerhard Tröster. 2008. It's in your eyes: towards context-awareness and mobile HCI using wearable EOG goggles. In Proceedings of the 10th international conference on Ubiquitous computing. ACM, 84--93.
20. Andreas Bulling, Daniel Roggen, and Gerhard Tröster. 2009. Wearable EOG goggles: Seamless sensing and context-awareness in everyday environments. Journal of Ambient Intelligence and Smart Environments 1, 2 (2009), 157--171.
21. Andreas Bulling, Jamie A Ward, Hans Gellersen, and Gerhard Troster. 2011. Eye movement analysis for activity recognition using electrooculography. IEEE transactions on pattern analysis and machine intelligence 33, 4 (2011), 741--753.
22. Michael Chau and Margrit Betke. 2005. Real time eye tracking and blink detection with usb cameras. Technical Report. Boston University Computer Science Department.
23. Elizabeth J. Corwin, Laura Cousino Klein, and Kristin Rickelman. 2002. Predictors of Fatigue in Healthy Young Adults: Moderating Effects of Cigarette Smoking and Gender. Biological Research For Nursing 3, 4 (2002), 222--233, PMID: 12184665.
24. Anirban Dasgupta, Anjith George, SL Happy, and Aurobinda Routray. 2013. A Vision-Based System for Monitoring the Loss of Attention in Automotive Drivers. IEEE Trans. Intelligent Transportation Systems 14, 4 (2013), 1825--1838.
25. Tomas Drutarovsky and Andrej Fogelton. 2014. Eye blink detection using variance of motion vectors. In European Conference on Computer Vision. Springer, 436--448.
26. John D. Fisk, Amanda Pontefract, Paul G. Ritvo, Catherine J. Archibald, and T.J. Murray. 1994. The Impact of Fatigue on Patients with Multiple Sclerosis. Canadian Journal of Neurological Sciences / Journal Canadien des Sciences Neurologiques 21, 1 (001 002 1994), 9--14.
27. J. Friedman, T. Hastie, and R. Tibshirani. 2001. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Series in Statistics, Vol. 1.
28. Wolfgang Fuhl, Thiago Santini, David Geisler, Thomas Kübler, Wolfgang Rosenstiel, and Enkelejda Kasneci. 2016. Eyes wide open? eyelid location and eye aperture estimation for pervasive eye tracking in real-world scenarios. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct. ACM, 1656--1665.
29. Kyosuke Fukuda, John A Stern, Timothy B Brown, and Michael B Russo. 2005. Cognition, blinks, eye-movements, and pupillary movements during performance of a running memory task. Aviation, space, and environmental medicine 76, 7 (2005), C75--C85.
30. X. Y. Gao, Y. F. Zhang, W. L. Zheng, and B. L. Lu. 2015. Evaluating driving fatigue detection algorithms using eye tracking glasses. In 2015 7th International IEEE/EMBS Conference on Neural Engineering (NER). 767--770.
31. I Garcia, Sebastian Bronte, Luis Miguel Bergasa, Javier Almazán, and J Yebes. 2012. Vision-based drowsiness detector for real driving conditions. In Intelligent Vehicles Symposium (IV), 2012 IEEE. IEEE, 618--623.
32. N. Goel, M. Basner, H. Rao, and D. F. Dinges. 2013. Circadian rhythms, sleep deprivation, and human performance. Prog Mol Biol Transl Sci 119 (2013), 155--190.
33. Kristen Grauman, Margrit Betke, James Gips, and Gary R Bradski. 2001. Communication via eye blinks-detection and duration analysis in real time. In Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on, Vol. 1. IEEE, I--I.
34. Martijn Haak, Steven Bos, Sacha Panic, and LJM Rothkrantz. 2009. Detecting stress using eye blinks and brain activity from EEG signals. Proceeding of the 1st driver car interaction and interface (DCII 2008) (2009), 35--60.
35. E. Havlikova, J. Rosenberger, I. Nagyova, B. Middel, T. Dubayova, Z. Gdovinova, J. P. Van Dijk, and J. W. Groothoff. 2008. Impact of fatigue on quality of life in patients with Parkinsonfis disease. European Journal of Neurology 15, 5 (2008), 475--480.
36. Tianyi Hong and Huabiao Qin. 2007. Drivers drowsiness detection in embedded system. In Vehicular Electronics and Safety, 2007. ICVES. IEEE International Conference on. IEEE, 1--5.
37. Chin-Shun Hsieh and Cheng-Chi Tai. 2013. An improved and portable eye-blink duration detection system to warn of driver fatigue. Instrumentation Science 8 Technology 41, 5 (2013), 429--444.
38. Michael Ingre, Torbjörn Ákerstedt, Björn Peters, Anna Anund, and Göran Kecklund. 2006. Subjective sleepiness, simulated driving performance and blink duration: examining individual differences. Journal of Sleep Research 15, 1 (2006), 47--53.
39. Amir-Homayoun Javadi, Zahra Hakimi, Morteza Barati, Vincent Walsh, and Lili Tcheang. 2015. SET: a pupil detection method using sinusoidal approximation. Frontiers in neuroengineering 8 (2015).
40. Xianta Jiang, Geoffrey Tien, Da Huang, Bin Zheng, and M Stella Atkins. 2013. Capturing and evaluating blinks from video-based eyetrackers. Behavior research methods 45, 3 (2013), 656--663.
41. Hughes JR and Hatsukami D. 1986. Signs and symptoms of tobacco withdrawal. Archives of General Psychiatry 43, 3 (1986), 289--294.
42. Hughes JR, Higgins ST, Bickel WK, and et al. 1991. Caffeine self-administration, withdrawal, and adverse effects among coffee drinkers. Archives of General Psychiatry 48, 7 (1991), 611--617.
43. Won Oh Lee, Eui Chul Lee, and Kang Ryoung Park. 2010. Blink detection robust to various facial poses. Journal of neuroscience methods 193, 2 (2010), 356--372.
44. I. S. Lobentanz, S. Asenbaum, K. Vass, C. Sauter, G. Klsch, H. Kollegger, W. Kristoferitsch, and J. Zeitlhofer. 2004. Factors influencing quality of life in multiple sclerosis patients: disability, depressive mood, fatigue and sleep quality. Acta Neurologica Scandinavica 110, 1 (2004), 6--13.
45. Amol M Malla, Paul R Davidson, Philip J Bones, Richard Green, and Richard D Jones. 2010. Automated video-based measurement of eye closure for detecting behavioral microsleep. In Engineering in Medicine and Biology Society (EMBC), 2010 Annual International Conference of the IEEE. IEEE, 6741--6744.
46. R Martins and JM Carvalho. 2015. Eye blinking as an indicator of fatigue and mental load-a systematic review. Occupational Safety and Hygiene III (2015), 231--235.
47. Addison Mayberry, Pan Hu, Benjamin Marlin, Christopher Salthouse, and Deepak Ganesan. 2014. iShadow: Design of a Wearable, Real-time Mobile Gaze Tracker. In Proceedings of the 12th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys ‘14). ACM, New York, NY, USA, 82--94.
48. Addison Mayberry, Yamin Tun, Pan Hu, Duncan Smith-Freedman, Deepak Ganesan, Benjamin M. Marlin, and Christopher Salthouse. 2015. CIDER: Enabling Robustness-Power Tradeoffs on a Computational Eyeglass. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (MobiCom ‘15). ACM, New York, NY, USA, 400--412.
49. Lindsey K McIntire, John P McIntire, R Andy McKinley, and Chuck Goodyear. 2014. Detection of vigilance performance with pupillometry. In Proceedings of the Symposium on Eye Tracking Research and Applications. ACM, 167--174.
50. T. R. Mendoza, X. S. Wang, C. S. Cleeland, M. Morrissey, B. A. Johnson, J. K. Wendt, and S. L. Huber. 1999. The rapid assessment of fatigue severity in cancer patients: use of the Brief Fatigue Inventory. Cancer 85, 5 (Mar 1999), 1186--1196.
51. Inga Meyhöfer, Veena Kumari, Antje Hill, Nadine Petrovsky, and Ulrich Ettinger. 2016. Sleep deprivation as an experimental model system for psychosis: Effects on smooth pursuit, prosaccades, and antisaccades. Journal of Psychopharmacology (2016), 0269881116675511.
52. Tim Morris, Paul Blenkhorn, and Farhan Zaidi. 2002. Blink detection for real-time eye tracking. Journal of Network and Computer Applications 25, 2 (2002), 129--143.
53. TL Morris and James C Miller. 1996. Electrooculographic and performance indices of fatigue during simulated flight. Biological psychology 42, 3 (1996), 343--360.
54. M Patel, SKL Lal, Diarmuid Kavanagh, and Peter Rossiter. 2011. Applying neural network analysis on heart rate variability data to assess driver fatigue. Expert systems with Applications 38, 6 (2011), 7235--7242.
55. Leo Pauly and Deepa Sankar. 2015. Detection of drowsiness based on hog features and svm classifiers. In Research in Computational Intelligence and Communication Networks (ICRCICN), 2015 IEEE International Conference on. IEEE, 181--186.
56. Marco Pedrotti, Shengguang Lei, Jeronimo Dzaack, and Matthias Rötting. 2011. A data-driven algorithm for offline pupil signal preprocessing and eyeblink detection in low-speed eye-tracking protocols. Behavior Research Methods 43, 2 (2011), 372--383.
57. A. Picot, A. Caplier, and S. Charbonnier. 2009. Comparison between EOG and high frame rate camera for drowsiness detection. In Applications of Computer Vision (WACV), 2009 Workshop on. 1--6.
58. Antoine Picot, Alice Caplier, and Sylvie Charbonnier. 2009. Comparison between EOG and high frame rate camera for drowsiness detection. In Applications of Computer Vision (WACV), 2009 Workshop on. IEEE, 1--6.
59. R. Schleicher, N. Galley, S. Briest, and L. Galley. 2008. Blinks and saccades as indicators of fatigue in sleepiness warnings: looking tired? Ergonomics 51, 7 (2008), 982--1010, PMID: 18568959.
60. Robert Schleicher, Niels Galley, Susanne Briest, and Lars Galley. 2008. Blinks and saccades as indicators of fatigue in sleepiness warnings: looking tired? Ergonomics 51, 7 (2008), 982--1010.
61. Paul Smith, Mubarak Shah, and Niels da Vitoria Lobo. 2003. Determining driver visual attention with one camera. IEEE transactions on intelligent transportation systems 4, 4 (2003), 205--218.
62. M. D. Stein and P. D. Friedmann. 2005. Disturbed sleep and its relationship to alcohol use. Subst Abus 26, 1 (Mar 2005), 1--13.
63. J. A. Stern, D. Boyer, David J. Schroeder, United States., and Civil Aeromedical Institute. 1994. Blink rate as a measure of fatigue {microform}: a review / J.A. Stern, D. Boyer, D.J. Schroeder. U.S. Dept. of Transportation, Federal Aviation Administration, Office of Aviation Medicine; Available to the public through the National Technical Information Service Washington, D.C.: Springfield, Va. i, 12 p.; pages. http://purl.access.gpo.gov/GPO/LPS84339
64. Federico M Sukno, Sri-Kaushik Pavani, Constantine Butakoff, and Alejandro F Frangi. 2009. Automatic assessment of eye blinking patterns through statistical shape models. In International Conference on Computer Vision Systems. Springer, 33--42.
65. Louis Tijernia, Stoltzfus Duane Gleckler, Mark nad, Scott Johnston, Michael J. Goodman, and Walter W. Wierwille. 1999. A Preliminary Assessment of Algorithms for Drowsy and Inattentive Driver Detection on the Road. Technical Report. Springfield VA.
66. Marc Tonsen, Xucong Zhang, Yusuke Sugano, and Andreas Bulling. 2016. Labelled pupils in the wild: a dataset for studying pupil detection in unconstrained environments. In Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research 8 Applications. ACM, 139--142.
67. W. W. Wierwille, S. S. Wreggit, C. L. Kirn, L. A. Ellsworth, and R. J. Fairbanks. 1994. Research on Vehicle-Based Driver Status/Performance Monitoring; Development, Validation, and Refinement of Algorithms For Detection of Driver Drowsiness. Technical Report. Springfield VA.
68. Ann Williamson. 2007. Predictors of Psychostimulant Use by Long-Distance Truck Drivers. American Journal of Epidemiology 166, 11 (2007), 1320.
69. Junwen Wu and Mohan M Trivedi. 2007. Simultaneous eye tracking and blink detection with interactive particle filters. EURASIP Journal on Advances in Signal Processing 2008, 1 (2007), 823695.
70. Cui Xu, Ying Zheng, and Zengfu Wang. 2008. Efficient eye states detection in real-time for drowsy driving monitoring system. In Information and Automation, 2008. ICIA 2008. International Conference on. IEEE, 170--174.
71. Chuang-Wen You, Nicholas D Lane, Fanglin Chen, Rui Wang, Zhenyu Chen, Thomas J Bao, Martha Montes-de Oca, Yuting Cheng, Mu Lin, Lorenzo Torresani, and others. 2013. Carsafe app: Alerting drowsy and distracted drivers using dual cameras on smartphones. In Proceeding of the 11th annual international conference on Mobile systems, applications, and services. ACM, 13--26.

Index Terms

• Applied computing

  • Life and medical sciences

    • Consumer health

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

        • Interest point and salient region detections

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI design and evaluation methods

      • User studies

  • Ubiquitous and mobile computing

    • Ubiquitous and mobile devices

      • Mobile devices