skip to main content
10.1145/3242587.3242635acmconferencesArticle/Chapter ViewAbstractPublication PagesuistConference Proceedingsconference-collections
research-article
Public Access

Self-Powered Gesture Recognition with Ambient Light

Published:11 October 2018Publication History

ABSTRACT

We present a self-powered module for gesture recognition that utilizes small, low-cost photodiodes for both energy harvesting and gesture sensing. Operating in the photovoltaic mode, photodiodes harvest energy from ambient light. In the meantime, the instantaneously harvested power from individual photodiodes is monitored and exploited as clues for sensing finger gestures in proximity. Harvested power from all photodiodes are aggregated to drive the whole gesture-recognition module including the micro-controller running the recognition algorithm. We design robust, lightweight algorithm to recognize finger gestures in the presence of ambient light fluctuations. We fabricate two prototypes to facilitate user's interaction with smart glasses and smart watch. Results show 99.7%/98.3% overall precision/recall in recognizing five gestures on glasses and 99.2%/97.5% precision/recall in recognizing seven gestures on the watch. The system consumes 34.6 µW/74.3 µW for the glasses/watch and thus can be powered by the energy harvested from ambient light. We also test system's robustness under varying light intensities, light directions, and ambient light fluctuations, where the system maintains high recognition accuracy (> 96%) in all tested settings.

Skip Supplemental Material Section

Supplemental Material

suppl.mov

Supplemental video

p595-li.mp4

References

  1. Roland Aigner, Daniel Wigdor, Hrvoje Benko, Michael Haller, David Lindlbauer, Alexandra Ion, Shengdong Zhao, and Jeffrey Tzu Kwan Valino Koh. 2012. Understanding Mid-Air Hand Gestures?: A Study of Human Preferences in Usage of Gesture Types for HCI. Tech. Rep. MSR-TR-2012--11: 10. Retrieved from https://www.microsoft.com/en-us/research/publication/understanding-mid-air-hand-gestures-a-study-of-human-preferences-in-usage-of-gesture-types-for-hci/Google ScholarGoogle Scholar
  2. Apple. Apple Watch Gestures.Google ScholarGoogle Scholar
  3. Shigeo Asahi, Haruyuki Teranishi, Kazuki Kusaki, Toshiyuki Kaizu, and Takashi Kita. 2017. Two-step photon up-conversion solar cells. Nature Communications 8.Google ScholarGoogle Scholar
  4. Shahid Ayub, Sharadha Kariyawasam, Mahsa Honary, and Bahram Honary. 2013. Visible light ID system for indoor localization. Wireless, Mobile and Multimedia Networks (ICWMMN 2013), 5th IET International Conference on: 254--257.Google ScholarGoogle Scholar
  5. Andreas Braun, Reiner Wichert, Arjan Kuijper, and Dieter W. Fellner. 2015. Capacitive proximity sensing in smart environments. Journal of Ambient Intelligence and Smart Environments 7, 4: 483--510.Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Michael Buckland and Fredric Gey. 1994. The relationship between Recall and Precision. Journal of the American Society for Information Science 45, 1: 12--19. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Alex Butler, Shahram Izadi, and Steve Hodges. 2008. SideSight?: Multi- " touch " interaction around small devices. UIST '08: Proceedings of the 21st annual ACM symposium on User interface software and technology 23, 21: 201--204. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Xiang 'Anthony' Chen, Julia Schwarz, Chris Harrison, Jennifer Mankoff, and Scott E. Hudson. 2014. Air+Touch: Interweaving Touch & In-Air Gestures. Proceedings of the ACM Symposium on User Interface Software and Technology, UIST 2014: 519--525. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Gabe Cohn, Sidhant Gupta, Tien-Jui Lee, Dan Morris, Joshua R Smith, Matthew S Reynolds, Desney S Tan, and Shwetak N Patel. 2012. An Ultra-low-power Human Body Motion Sensor Using Static Electric Field Sensing. Proceedings of the 2012 ACM Conference on Ubiquitous Computing: 99--102. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Andrea Colaço, Ahmed Kirmani, Hye Soo Yang, Nan-Wei Gong, Chris Schmandt, and Vivek K. Goyal. 2013. Mime: Compact, Low-Power 3D Gesture Sensing for Interaction with Head-Mounted Displays. Proceedings of the 26th annual ACM symposium on User interface software and technology - UIST '13: 227--236. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. M. A. Cowell, B. P. Lechene, P. Raffone, J. W. Evans, A. C. Arias, and P. K. Wright. 2016. Wireless sensor node demonstrating indoor-light energy harvesting and voltage-triggered duty cycling. In Journal of Physics: Conference Series.Google ScholarGoogle Scholar
  12. Artem Dementyev and Joseph A. Paradiso. 2014. WristFlex: Low-Power Gesture Input with Wrist-Worn Pressure Sensors. Proceedings of the 27th annual ACM symposium on User interface software and technology - UIST '14: 161--166. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Christine Dierk, Molly Jane, Pearce Nicholas, and Eric Paulos. 2018. AlterWear?: Battery-Free Wearable Displays for Opportunistic Interactions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Michal Karol Dobrzynski, Ramon Pericet-Camara, and Dario Floreano. 2012. Vision tape-a flexible compound vision sensor for motion detection and proximity estimation. IEEE Sensors Journal 12, 5: 1131--1139.Google ScholarGoogle ScholarCross RefCross Ref
  15. Google. Google Glass Gestures. Retrieved from https://support.google.com/glass/answer/3064184?hl=enGoogle ScholarGoogle Scholar
  16. Tobias Grosse-Puppendahl, Steve Hodges, Nicholas Chen, John Helmes, Stuart Taylor, James Scott, Josh Fromm, and David Sweeney. 2016. Exploring the Design Space for Energy-Harvesting Situated Displays. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology - UIST '16. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Sidhant Gupta, Daniel Morris, Shwetak Patel, and Desney Tan. 2012. SoundWave: Using the Doppler Effect to Sense Gestures. Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems - CHI '12: 1911--1914. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Naveed U L Hassan, Aqsa Naeem, and Muhammad Adeel Pasha. 2014. Indoor Positioning Using Visible LED Lights: A Survey. ACM Transactions on Sensor Networks 11, 2: 1--24. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Steve Hodges, Shahram Izadi, Alex Butler, Alban Rrustemi, and Bill Buxton. 2007. ThinSight. Proceedings of the 20th annual ACM symposium on User interface software and technology - UIST '07: 259.Google ScholarGoogle Scholar
  20. Huawei. Huawei Smartwatch Bezel Gestures.Google ScholarGoogle Scholar
  21. J. Kim, S. Yun and Y. Kim. 2016. Low-power motion gesture sensor with a partially open cavity package. Opt. Express 24: 10537--10546.Google ScholarGoogle ScholarCross RefCross Ref
  22. Jun Gong, Yang Zhang , Xia Zhou and Xing-Dong Yang. 2017. Pyro: Thumb-Tip Gesture Recognition Using Pyroelectric Infrared Sensing. In In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17), 553--563. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Bryce Kellogg, Vamsi Talla, and Shyamnath Gollakota. 2014. Bringing Gesture Recognition To All Devices. Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14): 303--316. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Bryce Kellogg, Vamsi Talla, Joshua R. Smith, and Shyamnath Gollakot. 2017. PASSIVE WI-FI: Bringing Low Power to Wi-Fi Transmissions. GetMobile: Mobile Computing and Communications 20, 3: 38--41. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Jungsoo Kim, Jiasheng He, Kent Lyons, and Thad Starner. 2007. The Gesture Watch: A wireless contact-free Gesture based wrist interface. In Proceedings - International Symposium on Wearable Computers, ISWC, 15--22. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Kwangtaek Kim, Joongrock Kim, Jaesung Choi, Junghyun Kim, and Sangyoun Lee. 2015. Depth camera-based 3D hand gesture controls with immersive tactile feedback for natural mid-air gesture interactions. Sensors (Switzerland) 15, 1: 1022--1046.Google ScholarGoogle ScholarCross RefCross Ref
  27. Alexander O. Korotkevich, Zhanna S. Galochkina, Olga Lavrova, and Evangelos A. Coutsias. 2015. On the comparison of energy sources: Feasibility of radio frequency and ambient light harvesting. Renewable Energy 81: 804--807.Google ScholarGoogle ScholarCross RefCross Ref
  28. Sven Kratz and Michael Rohs. 2009. Hoverflow: exploring around-device interaction with IR distance sensors. on Human-Computer Interaction with Mobile Devices, Figure 1: 1--4. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Tianxing Li, Qiang Liu and Xia Zhou. 2017. Ultra-Low Power Gaze Tracking for Virtual Reality. In the 15th ACM Conference on Embedded Network Sensor Systems (SenSys '17). Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Tianxing Li, Chuankai An, Zhao Tian, Andrew T. Campbell, and Xia Zhou. 2015. Human Sensing Using Visible Light Communication. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking - MobiCom '15, 331--344. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Tianxing Li, Qiang Liu, and Xia Zhou. 2016. Practical Human Sensing in the Light. In Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services - MobiSys '16, 71--84. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Yi Li. 2012. Hand gesture recognition using Kinect. Software Engineering and Service Science (ICSESS), 2012 IEEE 3rd International Conference on: 196--199.Google ScholarGoogle ScholarCross RefCross Ref
  33. Jaime Lien, Nicholas Gillian, M Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, Ivan Poupyrev, and Google Atap. 2016. Soli: Ubiquitous Gesture Sensing with Millimeter Wave Radar. ACM Trans. Graph. Article 35, 10: 1--19. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Mingyu Liu, Mathieu Nancel, and Daniel Vogel. 2015. Gunslinger: Subtle Arms-Down Mid-Air Interaction. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology - UIST '15, 63--71. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Shenwei Liu and F Guimbretière. 2012. FlexAura: a flexible near-surface range sensor. Proc. UIST. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Shang Ma, Qiong Liu, and Phillip C.Y. Sheu. 2018. Foglight: Visible Light-Enabled Indoor Localization System for Low-Power IoT Devices. IEEE Internet of Things Journal 5, 1: 175--185.Google ScholarGoogle ScholarCross RefCross Ref
  37. Anders Markussen, Mikkel Rønne Jakobsen, and Kasper Hornbæk. 2014. Vulture: a mid-air word-gesture keyboard. Proceedings of the 32nd annual ACM conference on Human factors in computing systems - CHI '14: 1073--1082. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. A Mayberry, Y Tun, P Hu, D Smith-Freedman, D Ganesan, B M Marlin, and C Salthouse. 2015. CIDER: Enabling robustness-power tradeoffs on a computational eyeglass. 21st Annual International Conference on Mobile Computing and Networking, MobiCom 2015. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. John Mulgrew, Bernard, Grant, Peter, Thompson. 2002. Digital Signal Processing: Concepts and Applications.Google ScholarGoogle Scholar
  40. Saman Naderiparizi, Aaron N. Parks, Zerina Kapetanovic, Benjamin Ransford, and Joshua R. Smith. 2015. WISPCam: A battery-free RFID camera. In 2015 IEEE International Conference on RFID, RFID 2015.Google ScholarGoogle ScholarCross RefCross Ref
  41. Shree K. Nayar, Daniel C. Sims, and Mikhail Fridberg. 2015. Towards Self-Powered Cameras. In 2015 IEEE International Conference on Computational Photography, ICCP 2015 - Proceedings.Google ScholarGoogle Scholar
  42. P. H. Niemenlehto. 2009. Constant false alarm rate detection of saccadic eye movements in electro-oculography. Computer Methods and Programs in Biomedicine 96, 2: 158--171. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Masa Ogata, Yuta Sugiura, Hirotaka Osawa, and Michita Imai. 2012. iRing: Intelligent Ring Using Infrared Reflection. In Proceedings of the 25th annual ACM symposium on User interface software and technology - UIST '12, 131--136. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Parth H. Pathak, Xiaotao Feng, Pengfei Hu, and Prasant Mohapatra. 2015. Visible Light Communication, Networking, and Sensing: A Survey, Potential and Challenges. IEEE Communications Surveys and Tutorials 17, 2047--2077.Google ScholarGoogle ScholarCross RefCross Ref
  45. Qifan Pu, Sidhant Gupta, Shyamnath Gollakota, and Shwetak Patel. 2013. Whole-home gesture recognition using wireless signals. In Proceedings of the 19th annual international conference on Mobile computing & networking - MobiCom '13, 27. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Dongseok Ryu, Dugan Um, Philip Tanofsky, Do Hyong Koh, Young Sam Ryu, and Sungchul Kang. 2010. T-less: A novel touchless human-machine interface based on infrared proximity sensing. In IEEE/RSJ 2010 International Conference on Intelligent Robots and Systems, IROS 2010 - Conference Proceedings, 5220--5225.Google ScholarGoogle Scholar
  47. C. Scharf, L. L., Demeure. 1991. Statistical signal processing: detection, estimation, and time series analysis. Addison-Wesley Reading, MA.Google ScholarGoogle Scholar
  48. Jie Song, Gábor Sörös, Fabrizio Pece, Sean Ryan Fanello, Shahram Izadi, Cem Keskin, and Otmar Hilliges. 2014. In-air gestures around unmodified mobile devices. In Proceedings of the 27th annual ACM symposium on User interface software and technology - UIST '14, 319--329. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Li Sun, Souvik Sen, Dimitrios Koutsonikolas, and Kyu-Han Kim. 2015. WiDraw: Enabling Hands-free Drawing in the Air on Commodity WiFi Devices. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking - MobiCom '15, 77--89. Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Vamsi Talla, Bryce Kellogg, Shyamnath Gollakota, and Joshua R. Smith. 2017. Battery-Free Cellphone. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 2: 1--20. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Danhang Tang, Tsz Ho Yu, and Tae Kyun Kim. 2013. Real-time articulated hand pose estimation using semi-supervised transductive regression forests. In Proceedings of the IEEE International Conference on Computer Vision, 3224--3231. Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Thorlab. Photodiode Tutorial.Google ScholarGoogle Scholar
  53. Hoang Truong, Phuc Nguyen, Anh Nguyen, Nam Bui, and Tam Vu. 2017. Capacitive Sensing 3D-printed Wristband for Enriched Hand Gesture Recognition. In Proceedings of the 2017 Workshop on Wearable Systems and Applications - WearSys '17, 11--15. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Satoshi Tsuji. 2012. A tactile and proximity sensor by optical and electrical measurement. In Proceedings of IEEE Sensors.Google ScholarGoogle Scholar
  55. Ambuj Varshney, Andreas Soleiman, Luca Mottola, and Thiemo Voigt. 2017. Battery-free Visible Light Sensing. In Proceedings of the 4th ACM Workshop on Visible Light Communication Systems - VLCS '17, 3--8. Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. R. J.M. Vullers, R. van Schaijk, I. Doms, C. Van Hoof, and R. Mertens. 2009. Micropower energy harvesting. Solid-State Electronics.Google ScholarGoogle Scholar
  57. Juan Pablo Wachs, Mathias Kölsch, Helman Stern, and Yael Edan. 2011. Vision-based hand-gesture applications. Communications of the ACM 54, 2: 60. Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Anusha Withana, Roshan Peiris, Nipuna Samarasekara, and Suranga Nanayakkara. 2015. zSense: Enabling Shallow Depth Gesture Recognition for Greater Input Expressivity on Smart Wearables. Proceedings of the ACM CHI'15 Conference on Human Factors in Computing Systems 1: 3661--3670. Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. Xing-Dong Yang, Tovi Grossman, Daniel Wigdor, and George Fitzmaurice. 2012. Magic Finger: Always-Available Input through Finger Instrumentation. Proceedings of the 25th annual ACM symposium on User interface software and technology - UIST '12: 147--156. Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. Liang Yin, Xiping Wu, and Harald Haas. 2016. Indoor visible light positioning with angle diversity transmitter. In 2015 IEEE 82nd Vehicular Technology Conference, VTC Fall 2015 - Proceedings.Google ScholarGoogle Scholar
  61. Bei Yuan and Eelke Folmer. 2008. Blind hero: enabling guitar hero for the visually impaired. Proceedings of the 10th international ACM SIGACCESS conference on Computers and accessibility: 169--176. Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. Chi Zhang, Josh Tabor, Jialiang Zhang, and Xinyu Zhang. 2015. Extending Mobile Interaction Through Near-Field Visible Light Sensing. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking - MobiCom '15, 345--357. Google ScholarGoogle ScholarDigital LibraryDigital Library
  63. Chi Zhang and Xinyu Zhang. 2016. LiTell: Robust Indoor Localization Using Unmodified Light Fixtures. Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking - MobiCom '16: 230--242. Google ScholarGoogle ScholarDigital LibraryDigital Library
  64. Chen Zhao, Ke-Yu Chen, Md Tanvir Islam Aumi, Shwetak Patel, and Matthew S. Reynolds. 2014. SideSwipe. In Proceedings of the 27th annual ACM symposium on User interface software and technology - UIST '14, 527--534. Google ScholarGoogle ScholarDigital LibraryDigital Library
  65. Xia Zhou and Andrew T. Campbell. 2014. Visible light networking and sensing. HotWireless 2014 - Proceedings of the 1st ACM MobiCom Workshop on Hot Topics in Wireless: 55--59. Google ScholarGoogle ScholarDigital LibraryDigital Library
  66. BPW34. https://www.osram.com/os/ecat/DIL%20SMT%20BPW%2034%20S/com/en/class_pim_web_catalog_103489/global/prd_pim_device_2219543/.Google ScholarGoogle Scholar
  67. ADG736. http://www.analog.com/media/en/technical-documentation/data-sheets/ADG736.pdf.Google ScholarGoogle Scholar
  68. 74HC4514. http://www.ti.com/lit/ds/schs280c/schs280c.pdf.Google ScholarGoogle Scholar
  69. LTC3106. http://www.analog.com/en/products/power-management/energy-harvesting/ltc3106.html.Google ScholarGoogle Scholar
  70. MINI-M4 for MSP432. https://www.mikroe.com/mini-msp432.Google ScholarGoogle Scholar
  71. MSP432P401R. http://www.ti.com/lit/ds/symlink/msp432p401r.pdf.Google ScholarGoogle Scholar
  72. Monsoon. http://msoon.github.io/powermonitor/PowerTool/doc/Power%20Monitor%20Manual.pdf.Google ScholarGoogle Scholar
  73. TS881. http://www.st.com/resource/en/datasheet/ts881.pdf.Google ScholarGoogle Scholar
  74. Xbox Kinect. https://www.xbox.com/en-US/kinect.Google ScholarGoogle Scholar
  75. Leap Motion. https://www.leapmotion.com/.Google ScholarGoogle Scholar
  76. Point Grab. http://www.pointgrab.com/.Google ScholarGoogle Scholar
  77. Crunch Fish. http://crunchfish.com/.Google ScholarGoogle Scholar
  78. APDS-9130. https://www.broadcom.com/products/optical-sensors/proximity-sensors/apds-9130.Google ScholarGoogle Scholar
  79. APDS-9190. https://www.broadcom.com/products/optical-sensors/proximity-sensors/apds-9190.Google ScholarGoogle Scholar
  80. ads7042. http://www.ti.com/lit/ds/symlink/ads7042.pdf.Google ScholarGoogle Scholar

Index Terms

  1. Self-Powered Gesture Recognition with Ambient Light

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in
    • Published in

      cover image ACM Conferences
      UIST '18: Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology
      October 2018
      1016 pages
      ISBN:9781450359481
      DOI:10.1145/3242587

      Copyright © 2018 ACM

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 11 October 2018

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article

      Acceptance Rates

      UIST '18 Paper Acceptance Rate80of375submissions,21%Overall Acceptance Rate842of3,967submissions,21%

      Upcoming Conference

      UIST '24

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader