PupilScreen: Using Smartphones to Assess Traumatic Brain Injury

Abstract

Before a person suffering from a traumatic brain injury reaches a medical facility, measuring their pupillary light reflex (PLR) is one of the few quantitative measures a clinician can use to predict their outcome. We propose PupilScreen, a smartphone app and accompanying 3D-printed box that combines the repeatability, accuracy, and precision of a clinical device with the ubiquity and convenience of the penlight test that clinicians regularly use in emergency situations. The PupilScreen app stimulates the patient's eyes using the smartphone's flash and records the response using the camera. The PupilScreen box, akin to a head-mounted virtual reality display, controls the eyes' exposure to light. The recorded video is processed using convolutional neural networks that track the pupil diameter over time, allowing for the derivation of clinically relevant measures. We tested two different network architectures and found that a fully convolutional neural network was able to track pupil diameter with a median error of 0.30 mm. We also conducted a pilot clinical evaluation with six patients who had suffered a TBI and found that clinicians were almost perfect when separating unhealthy pupillary light reflexes from healthy ones using PupilScreen alone.

References

1. Alireza Abdolvahabi, Brandon W. Taylor, Rebecca L. Holden, Elizabeth V Shaw, Alex Kentsis, Carlos Rodriguez-Galindo, Shizuo Mukai, and Bryan F. Shaw. Colorimetric and longitudinal analysis of leukocoria in recreational photographs of children with retinoblastoma. PloS one, 8(10):e76677, oct 2013.
2. Andrew Bastawrous, Hillary K. Rono, Iain A. T. Livingstone, Helen A. Weiss, Stewart Jordan, Hannah Kuper, and Matthew J. Burton. Development and Validation of a Smartphone-Based Visual Acuity Test (Peek Acuity) for Clinical Practice and Community-Based Fieldwork. JAMA Ophthalmology, 133(8):930, aug 2015.
3. Matthias Behrends, Claus U Niemann, and Merlin D Larson. Infrared pupillometry to detect the light reflex during cardiopulmonary resuscitation: A case series. Resuscitation, 83(10):1223--1228, 2012.
4. Barry P Boden, Robin L Tacchetti, Robert C Cantu, Sara B Knowles, and Frederick O Mueller. Catastrophic Head Injuries in High School and College Football Players. The American Journal of Sports Medicine, 35(7):1075--1081, mar 2007.
5. Steven P Broglio, Michael S Ferrara, Stephen N Macciocchi, Ted A Baumgartner, and Ronald Elliott. Test-retest reliability of computerized concussion assessment programs. Journal of Athletic Training, 42(4):509--514, 2007.
6. Jose E Capó-Aponté, Thomas G Urosevich, David V Walsh, Leonard A Temme, and Aaron K Tarbett. Pupillary Light Reflex as an Objective Biomarker for Early Identification of Blast-Induced mTBI. Journal of Spine, 2013.
7. Centers for Disease Control. TBI: Get the Facts, 2016.
8. Rama Chellappa. The changing fortunes of pattern recognition and computer vision. Image and Vision Computing, 55(1):3--5, 2016.
9. David Couret, Delphine Boumaza, Coline Grisotto, Thibaut Triglia, Lionel Pellegrini, Philippe Ocquidant, Nicolas J Bruder, and Lionel J Velly. Reliability of standard pupillometry practice in neurocritical care: an observational, double-blinded study. Critical Care, 20(1):99, dec 2016.
10. Mark Faul, Likang Xu, MM Wald, and VG Coronado. Traumatic Brain Injury in the United States. Technical report, 2010.
11. JD Fischer and DJ van den Heever. Portable video-oculography device for implementation in side-line concussion assessments: a prototype. In Proc. EMBC ‘16, 2016.
12. Wolfgang Fuhl, Thomas Kübler, Katrin Sippel, Wolfgang Rosenstiel, and Enkelejda Kasneci. ExCuSe: Robust Pupil Detection in Real-World Scenarios. In International Conference on Computer Analysis of Images and Patterns, pages 39--51. Springer International Publishing, 2015.
13. Wolfgang Fuhl, Thiago Santini, Gjergji Kasneci, and Enkelejda Kasneci. PupilNet: Convolutional Neural Networks for Robust Pupil Detection. (1-10), 2016.
14. Wolfgang Fuhl, Thiago C Santini, Thomas Kübler, and Enkelejda Kasneci. ElSe : Ellipse Selection for Robust Pupil Detection in Real-World Environments. In Eye Tracking Research 8 Applications, pages 123--130, 2016.
15. Kristin M Galetta, Lauren E Brandes, Karl Maki, Mark S Dziemianowicz, Eric Laudano, Megan Allen, Kathy Lawler, Brian Sennett, Douglas Wiebe, Steve Devick, Leonard V Messner, Steven L Galetta, and Laura J Balcer. Journal of the Neurological Sciences The King-Devick test and sports-related concussion : Study of a rapid visual screening tool in a collegiate cohort. Journal of the neurological sciences, 309(1-2):34--39, 2011.
16. Kristin M Galetta, Mengling Liu, Danielle F Leong, Rachel E Ventura, Steven L Galetta, and Laura J Balcer. The King-Devick test of rapid number naming for concussion detection: meta-analysis and systematic review of the literature. Concussion, 1(2):cnc.15.8, 2015.
17. Matthew S. Galetta, Kristin M. Galetta, Jim McCrossin, James A Wilson, Stephen Moster, Steven L Galetta, Laura J Balcer, Gary W Dorshimer, and Christina L Master. Saccades and memory: Baseline associations of the King-Devick and SCAT2 SAC tests in professional ice hockey players. Journal of the Neurological Sciences, 328(1-2):28--31, 2013.
18. Mario E Giardini, Iain A T Livingstone, Stewart Jordan, Nigel M Bolster, Tunde Peto, Matthew Burton, and Andrew Bastawrous. A smartphone based ophthalmoscope. Proc. EMBC '14, 2014:2177--2180, 2014.
19. Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
20. Kimberly G Harmon, Jonathan A Drezner, Matthew Gammons, Kevin M Guskiewicz, Mark Halstead, Stanley A Herring, Jeffrey S Kutcher, Andrea Pana, Margot Putukian, and William O Roberts. American Medical Society for Sports Medicine position statement: concussion in sport. British Journal of Sports Medicine, 47(1):15--26, jan 2013.
21. Moritz Kassner, William Patera, and Andreas Bulling. Pupil: an open source platform for pervasive eye tracking and mobile gaze-based interaction. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing Adjunct Publication - UbiComp '14 Adjunct, pages 1151--1160, New York, New York, USA, 2014. ACM Press.
22. Jeff Klingner, Rakshit Kumar, and Pat Hanrahan. Measuring the task-evoked pupillary response with a remote eye tracker. Proceedings of the 2008 symposium on Eye tracking research 8 applications - ETRA '08, 1(212):69, 2008.
23. Merlin D Larson and Matthias Behrends. Portable Infrared Pupillometry. Anesthesia 8 Analgesia, 120(6):1242--1253, 2015.
24. Merlin D Larson and Isobel Muhiudeen. Pupillometric analysis of the 'absent light reflex'. Archives of neurology, 52(4):369--72, 1995.
25. Everett Lawson, Jason Boggess, Siddharth Khullar, Alex Olwal, Gordon Wetzstein, and Ramesh Raskar. Computational retinal imaging via binocular coupling and indirect illumination. In Proc. SIGGRAPH '12, page 51, 2012.
26. Hopin Lee, S John Sullivan, Anthony G Schneiders, Osman Hassan Ahmed, Arun Prasad Balasundaram, David Williams, Willem H Meeuwisse, and Paul McCrory. Smartphone and tablet apps for concussion road warriors (team clinicians): a systematic review for practical users. British journal of sports medicine, 49(8):1--2, apr 2014.
27. Dongheng Li, David Winfield, and Derrick J Parkhurst. Starburst: A hybrid algorithm for video-based eye tracking combining feature-based and model-based approaches. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops, volume 3, pages 79--79. IEEE, 2005.
28. Joiiiv R Lindsay. The significance of a positional nystagmus in otoneurological diagnosis. The Laryngoscope, 55(10):527--551, 1945.
29. Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 07-12-June, pages 3431--3440, mar 2015.
30. Alex Mariakakis, Megan A. Banks, Lauren Phillipi, Lei Yu, James Taylor, and Shwetak N. Patel. Biliscreen: smartphone-based scleral jaundice monitoring for liver and pancreatic disorders. In Proceedings of the 2017 ACM Interactive, Mobile, Wearable, Ubiquitous Technologies. ACM, 2017.
31. F Martínez-Ricarte, A Castro, MA Poca, J Sahuquillo, L Expósito, M Arribas, and J Aparicio. Infrared pupillometry. Basic principles and their application in the non-invasive monitoring of neurocritical patients. Neurología (Barcelona, Spain), 28(1):41--51, 2013.
32. Jun Maruta, Stephanie W. Lee, Emily F. Jacobs, and Jamshid Ghajar. A unified science of concussion. Annals of the New York Academy of Sciences, 1208(1):58--66, oct 2010.
33. Jun Maruta, J L Tong, Stephanie W Lee, Zarah Iqbal, Alison Schonberger, and Jamshid Ghajar. EYE-TRAC: monitoring attention and utility for mTBI. Sensing Technologies for Global Health, Military Medicine, Disaster Response, and Environmental Monitoring Ii: And Biometric Technology for Human Identification Ix, 8371:11, 2012.
34. Alex Marvez. Players could try to beat concussion tests | FOX Sports, 2011.
35. Paul McCrory, Willem H Meeuwisse, Mark Aubry, Bob Cantu, Jiri Dvorak, Ruben Echemendia, Lars Engebretsen, Karen Johnson, Jeffery S Kutcher, Martin Raftery, Allen Sills, Brian W Benson, Gavin A Davis, Richard G Ellenbogen, Kevin Guskiewicz, Stanley A Herring, Grant L Iverson, Barry D Jordan, James Kissick, Michael McCrea, Andrew S McIntosh, David Maddocks, Michael Makdissi, Laura Purcell, Margot Putukain, Kathryn Schneider, Charles H Tator, and Michael Turner. Consensus Statement on Concussion in Sport: The 4th International Conference on Concussion in Sport Held in Zurick, November 2012. British Journal of Sports Medicine, 47(2):250--258, apr 2013.
36. Michele Meeker, Rose Du, Peter Bacchetti, Claudio M Privitera, Merlin D Larson, Martin C Holland, and Geoffrey Manley. Pupil examination: validity and clinical utility of an automated pupillometer. The Journal of neuroscience nursing : journal of the American Association of Neuroscience Nurses, 37(1):34--40, 2005.
37. Tim Morris, Paul Blenkhorn, and Farhan Zaidi. Blink detection for real-time eye tracking. Journal of Network and Computer Applications, 25(2):129--143, 2002.
38. Rosemarie Scolaro Moser, Grant L Iverson, Ruben J Echemendia, Mark R Lovell, Philip Schatz, Frank M Webbe, Ronald M Ruff, Jeffrey T Barth, Donna K. NAN Policy and Planning Committee, Shane S. Donna K. Broshek, Shane S. Bush, Sandra P. Koffler, Cecil R. Reynolds, Cheryl H. Silver, Sandra P. Koffler, Cecil R. Reynolds, and Cheryl H. Silver. Neuropsychological evaluation in the diagnosis and management of sports-related concussion. Archives of clinical neuropsychology : the official journal of the National Academy of Neuropsychologists, 22(8):909--16, nov 2007.
39. Dai-Wai M Olson, Sonja Stutzman, Ciji Saju, Margaret Wilson, Weidan Zhao, and Venkatesh Aiyagari. Interrater Reliability of Pupillary Assessments. Neurocritical Care, 24(2):251--257, apr 2016.
40. Vitor F Pamplona, Ankit Mohan, Manuel M Oliveira, and Ramesh Raskar. NETRA: interactive display for estimating refractive errors and focal range. ACM transactions on graphics (TOG), 29(4):77, 2010.
41. Vitor F Pamplona, Erick B Passos, Jan Zizka, Manuel M Oliveira, Everett Lawson, Esteban Clua, and Ramesh Raskar. Catra: cataract probe with a lightfield display and a snap-on eyepiece for mobile phones. In Proc. SIGGRAPH ‘11, pages 7--11, 2011.
42. Stephen M Pizer, E. Philip Amburn, John D Austin, Robert Cromartie, Ari Geselowitz, Trey Greer, Bart ter Haar Romeny, John B Zimmerman, and Karel Zuiderveld. Adaptive histogram equalization and its variations. Computer Vision, Graphics, and Image Processing, 39(3):355--368, 1987.
43. Sohail Rafiqi, Chatchai Wangwiwattana, Jasmine Kim, Ephrem Fernandez, Suku Nair, and Eric C. Larson. PupilWare: towards pervasive cognitive load measurement using commodity devices. In Proceedings of the 8th ACM International Conference on Pervasive Technologies Related to Assistive Environments - PETRA '15, pages 1--8, New York, New York, USA, 2015. ACM Press.
44. Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
45. Lech Swirski, Andreas Bulling, and Neil Dodgson. Robust real-time pupil tracking in highly off-axis images. Etra, pages 1--4, 2012.
46. William R Taylor, Jeff W Chen, Hal Meltzer, Thomas A Gennarelli, Cynthia Kelbch, Sharen Knowlton, Jenny Richardson, Matthew J Lutch, Azadeh Farin, Kathryn N Hults, and Lawrence F Marshall. Quantitative pupillometry, a new technology: normative data and preliminary observations in patients with acute head injury. Technical note. Journal of neurosurgery, 98(1):205--213, 2003.
47. Preethi Thiagarajan and Kenneth J Ciuffreda. Pupillary responses to light in chronic non-blast-induced mTBI. Brain Injury, 29(12):1420--1425, oct 2015.
48. Fabian Timm and Erhardt Barth. Accurate Eye Centre Localisation by Means of Gradients. In VISAPP, pages 125--130, 2011.
49. James Q Truong and Kenneth J Ciuffreda. Comparison of pupillary dynamics to light in the mild traumatic brain injury (mTBI) and normal populations. Brain Injury, 30(11):1378--1389, sep 2016.
50. James Q Truong and Kenneth J Ciuffreda. Objective Pupillary Correlates of Photosensitivity in the Normal and Mild Traumatic Brain Injury Populations. Military Medicine, 181(10):1382--1390, oct 2016.
51. James Q Truong and Kenneth J Ciuffreda. Quantifying pupillary asymmetry through objective binocular pupillometry in the normal and mild traumatic brain injury (mTBI) populations. Brain Injury, 30(11):1372--1377, sep 2016.
52. James Q Truong, Nabin R Joshi, and Kenneth J Ciuffreda. Influence of refractive error on pupillary dynamics in the normal and mild traumatic brain injury (mTBI) populations. Journal of Optometry, mar 2017.
53. H Kenneth Walker, W Dallas Hall, and J Willis Hurst. Clinical Methods. Butterworths, 1990.
54. E Wood, T Baltruaitis, X Zhang, Y Sugano, P Robinson, and A Bulling. Rendering of Eyes for Eye-Shape Registration and Gaze Estimation, 2015.
55. Erroll Wood and Andreas Bulling. Eyetab: Model-based gaze estimation on unmodified tablet computers. Etra, pages 3--6, 2014.
56. Sahar F Zafar and Jose I Suarez. Automated pupillometer for monitoring the critically ill patient: A critical appraisal, 2014.

Index Terms

• Applied computing

  • Life and medical sciences

    • Consumer health

• Human-centered computing

  • Ubiquitous and mobile computing

    • Ubiquitous and mobile devices

      • Smartphones