ABSTRACT
We present a self-powered module for gesture recognition that utilizes small, low-cost photodiodes for both energy harvesting and gesture sensing. Operating in the photovoltaic mode, photodiodes harvest energy from ambient light. In the meantime, the instantaneously harvested power from individual photodiodes is monitored and exploited as clues for sensing finger gestures in proximity. Harvested power from all photodiodes are aggregated to drive the whole gesture-recognition module including the micro-controller running the recognition algorithm. We design robust, lightweight algorithm to recognize finger gestures in the presence of ambient light fluctuations. We fabricate two prototypes to facilitate user's interaction with smart glasses and smart watch. Results show 99.7%/98.3% overall precision/recall in recognizing five gestures on glasses and 99.2%/97.5% precision/recall in recognizing seven gestures on the watch. The system consumes 34.6 µW/74.3 µW for the glasses/watch and thus can be powered by the energy harvested from ambient light. We also test system's robustness under varying light intensities, light directions, and ambient light fluctuations, where the system maintains high recognition accuracy (> 96%) in all tested settings.
Supplemental Material
- Roland Aigner, Daniel Wigdor, Hrvoje Benko, Michael Haller, David Lindlbauer, Alexandra Ion, Shengdong Zhao, and Jeffrey Tzu Kwan Valino Koh. 2012. Understanding Mid-Air Hand Gestures?: A Study of Human Preferences in Usage of Gesture Types for HCI. Tech. Rep. MSR-TR-2012--11: 10. Retrieved from https://www.microsoft.com/en-us/research/publication/understanding-mid-air-hand-gestures-a-study-of-human-preferences-in-usage-of-gesture-types-for-hci/Google Scholar
- Apple. Apple Watch Gestures.Google Scholar
- Shigeo Asahi, Haruyuki Teranishi, Kazuki Kusaki, Toshiyuki Kaizu, and Takashi Kita. 2017. Two-step photon up-conversion solar cells. Nature Communications 8.Google Scholar
- Shahid Ayub, Sharadha Kariyawasam, Mahsa Honary, and Bahram Honary. 2013. Visible light ID system for indoor localization. Wireless, Mobile and Multimedia Networks (ICWMMN 2013), 5th IET International Conference on: 254--257.Google Scholar
- Andreas Braun, Reiner Wichert, Arjan Kuijper, and Dieter W. Fellner. 2015. Capacitive proximity sensing in smart environments. Journal of Ambient Intelligence and Smart Environments 7, 4: 483--510.Google ScholarDigital Library
- Michael Buckland and Fredric Gey. 1994. The relationship between Recall and Precision. Journal of the American Society for Information Science 45, 1: 12--19. Google ScholarDigital Library
- Alex Butler, Shahram Izadi, and Steve Hodges. 2008. SideSight?: Multi- " touch " interaction around small devices. UIST '08: Proceedings of the 21st annual ACM symposium on User interface software and technology 23, 21: 201--204. Google ScholarDigital Library
- Xiang 'Anthony' Chen, Julia Schwarz, Chris Harrison, Jennifer Mankoff, and Scott E. Hudson. 2014. Air+Touch: Interweaving Touch & In-Air Gestures. Proceedings of the ACM Symposium on User Interface Software and Technology, UIST 2014: 519--525. Google ScholarDigital Library
- Gabe Cohn, Sidhant Gupta, Tien-Jui Lee, Dan Morris, Joshua R Smith, Matthew S Reynolds, Desney S Tan, and Shwetak N Patel. 2012. An Ultra-low-power Human Body Motion Sensor Using Static Electric Field Sensing. Proceedings of the 2012 ACM Conference on Ubiquitous Computing: 99--102. Google ScholarDigital Library
- Andrea Colaço, Ahmed Kirmani, Hye Soo Yang, Nan-Wei Gong, Chris Schmandt, and Vivek K. Goyal. 2013. Mime: Compact, Low-Power 3D Gesture Sensing for Interaction with Head-Mounted Displays. Proceedings of the 26th annual ACM symposium on User interface software and technology - UIST '13: 227--236. Google ScholarDigital Library
- M. A. Cowell, B. P. Lechene, P. Raffone, J. W. Evans, A. C. Arias, and P. K. Wright. 2016. Wireless sensor node demonstrating indoor-light energy harvesting and voltage-triggered duty cycling. In Journal of Physics: Conference Series.Google Scholar
- Artem Dementyev and Joseph A. Paradiso. 2014. WristFlex: Low-Power Gesture Input with Wrist-Worn Pressure Sensors. Proceedings of the 27th annual ACM symposium on User interface software and technology - UIST '14: 161--166. Google ScholarDigital Library
- Christine Dierk, Molly Jane, Pearce Nicholas, and Eric Paulos. 2018. AlterWear?: Battery-Free Wearable Displays for Opportunistic Interactions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Google ScholarDigital Library
- Michal Karol Dobrzynski, Ramon Pericet-Camara, and Dario Floreano. 2012. Vision tape-a flexible compound vision sensor for motion detection and proximity estimation. IEEE Sensors Journal 12, 5: 1131--1139.Google ScholarCross Ref
- Google. Google Glass Gestures. Retrieved from https://support.google.com/glass/answer/3064184?hl=enGoogle Scholar
- Tobias Grosse-Puppendahl, Steve Hodges, Nicholas Chen, John Helmes, Stuart Taylor, James Scott, Josh Fromm, and David Sweeney. 2016. Exploring the Design Space for Energy-Harvesting Situated Displays. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology - UIST '16. Google ScholarDigital Library
- Sidhant Gupta, Daniel Morris, Shwetak Patel, and Desney Tan. 2012. SoundWave: Using the Doppler Effect to Sense Gestures. Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems - CHI '12: 1911--1914. Google ScholarDigital Library
- Naveed U L Hassan, Aqsa Naeem, and Muhammad Adeel Pasha. 2014. Indoor Positioning Using Visible LED Lights: A Survey. ACM Transactions on Sensor Networks 11, 2: 1--24. Google ScholarDigital Library
- Steve Hodges, Shahram Izadi, Alex Butler, Alban Rrustemi, and Bill Buxton. 2007. ThinSight. Proceedings of the 20th annual ACM symposium on User interface software and technology - UIST '07: 259.Google Scholar
- Huawei. Huawei Smartwatch Bezel Gestures.Google Scholar
- J. Kim, S. Yun and Y. Kim. 2016. Low-power motion gesture sensor with a partially open cavity package. Opt. Express 24: 10537--10546.Google ScholarCross Ref
- Jun Gong, Yang Zhang , Xia Zhou and Xing-Dong Yang. 2017. Pyro: Thumb-Tip Gesture Recognition Using Pyroelectric Infrared Sensing. In In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17), 553--563. Google ScholarDigital Library
- Bryce Kellogg, Vamsi Talla, and Shyamnath Gollakota. 2014. Bringing Gesture Recognition To All Devices. Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14): 303--316. Google ScholarDigital Library
- Bryce Kellogg, Vamsi Talla, Joshua R. Smith, and Shyamnath Gollakot. 2017. PASSIVE WI-FI: Bringing Low Power to Wi-Fi Transmissions. GetMobile: Mobile Computing and Communications 20, 3: 38--41. Google ScholarDigital Library
- Jungsoo Kim, Jiasheng He, Kent Lyons, and Thad Starner. 2007. The Gesture Watch: A wireless contact-free Gesture based wrist interface. In Proceedings - International Symposium on Wearable Computers, ISWC, 15--22. Google ScholarDigital Library
- Kwangtaek Kim, Joongrock Kim, Jaesung Choi, Junghyun Kim, and Sangyoun Lee. 2015. Depth camera-based 3D hand gesture controls with immersive tactile feedback for natural mid-air gesture interactions. Sensors (Switzerland) 15, 1: 1022--1046.Google ScholarCross Ref
- Alexander O. Korotkevich, Zhanna S. Galochkina, Olga Lavrova, and Evangelos A. Coutsias. 2015. On the comparison of energy sources: Feasibility of radio frequency and ambient light harvesting. Renewable Energy 81: 804--807.Google ScholarCross Ref
- Sven Kratz and Michael Rohs. 2009. Hoverflow: exploring around-device interaction with IR distance sensors. on Human-Computer Interaction with Mobile Devices, Figure 1: 1--4. Google ScholarDigital Library
- Tianxing Li, Qiang Liu and Xia Zhou. 2017. Ultra-Low Power Gaze Tracking for Virtual Reality. In the 15th ACM Conference on Embedded Network Sensor Systems (SenSys '17). Google ScholarDigital Library
- Tianxing Li, Chuankai An, Zhao Tian, Andrew T. Campbell, and Xia Zhou. 2015. Human Sensing Using Visible Light Communication. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking - MobiCom '15, 331--344. Google ScholarDigital Library
- Tianxing Li, Qiang Liu, and Xia Zhou. 2016. Practical Human Sensing in the Light. In Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services - MobiSys '16, 71--84. Google ScholarDigital Library
- Yi Li. 2012. Hand gesture recognition using Kinect. Software Engineering and Service Science (ICSESS), 2012 IEEE 3rd International Conference on: 196--199.Google ScholarCross Ref
- Jaime Lien, Nicholas Gillian, M Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, Ivan Poupyrev, and Google Atap. 2016. Soli: Ubiquitous Gesture Sensing with Millimeter Wave Radar. ACM Trans. Graph. Article 35, 10: 1--19. Google ScholarDigital Library
- Mingyu Liu, Mathieu Nancel, and Daniel Vogel. 2015. Gunslinger: Subtle Arms-Down Mid-Air Interaction. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology - UIST '15, 63--71. Google ScholarDigital Library
- Shenwei Liu and F Guimbretière. 2012. FlexAura: a flexible near-surface range sensor. Proc. UIST. Google ScholarDigital Library
- Shang Ma, Qiong Liu, and Phillip C.Y. Sheu. 2018. Foglight: Visible Light-Enabled Indoor Localization System for Low-Power IoT Devices. IEEE Internet of Things Journal 5, 1: 175--185.Google ScholarCross Ref
- Anders Markussen, Mikkel Rønne Jakobsen, and Kasper Hornbæk. 2014. Vulture: a mid-air word-gesture keyboard. Proceedings of the 32nd annual ACM conference on Human factors in computing systems - CHI '14: 1073--1082. Google ScholarDigital Library
- A Mayberry, Y Tun, P Hu, D Smith-Freedman, D Ganesan, B M Marlin, and C Salthouse. 2015. CIDER: Enabling robustness-power tradeoffs on a computational eyeglass. 21st Annual International Conference on Mobile Computing and Networking, MobiCom 2015. Google ScholarDigital Library
- John Mulgrew, Bernard, Grant, Peter, Thompson. 2002. Digital Signal Processing: Concepts and Applications.Google Scholar
- Saman Naderiparizi, Aaron N. Parks, Zerina Kapetanovic, Benjamin Ransford, and Joshua R. Smith. 2015. WISPCam: A battery-free RFID camera. In 2015 IEEE International Conference on RFID, RFID 2015.Google ScholarCross Ref
- Shree K. Nayar, Daniel C. Sims, and Mikhail Fridberg. 2015. Towards Self-Powered Cameras. In 2015 IEEE International Conference on Computational Photography, ICCP 2015 - Proceedings.Google Scholar
- P. H. Niemenlehto. 2009. Constant false alarm rate detection of saccadic eye movements in electro-oculography. Computer Methods and Programs in Biomedicine 96, 2: 158--171. Google ScholarDigital Library
- Masa Ogata, Yuta Sugiura, Hirotaka Osawa, and Michita Imai. 2012. iRing: Intelligent Ring Using Infrared Reflection. In Proceedings of the 25th annual ACM symposium on User interface software and technology - UIST '12, 131--136. Google ScholarDigital Library
- Parth H. Pathak, Xiaotao Feng, Pengfei Hu, and Prasant Mohapatra. 2015. Visible Light Communication, Networking, and Sensing: A Survey, Potential and Challenges. IEEE Communications Surveys and Tutorials 17, 2047--2077.Google ScholarCross Ref
- Qifan Pu, Sidhant Gupta, Shyamnath Gollakota, and Shwetak Patel. 2013. Whole-home gesture recognition using wireless signals. In Proceedings of the 19th annual international conference on Mobile computing & networking - MobiCom '13, 27. Google ScholarDigital Library
- Dongseok Ryu, Dugan Um, Philip Tanofsky, Do Hyong Koh, Young Sam Ryu, and Sungchul Kang. 2010. T-less: A novel touchless human-machine interface based on infrared proximity sensing. In IEEE/RSJ 2010 International Conference on Intelligent Robots and Systems, IROS 2010 - Conference Proceedings, 5220--5225.Google Scholar
- C. Scharf, L. L., Demeure. 1991. Statistical signal processing: detection, estimation, and time series analysis. Addison-Wesley Reading, MA.Google Scholar
- Jie Song, Gábor Sörös, Fabrizio Pece, Sean Ryan Fanello, Shahram Izadi, Cem Keskin, and Otmar Hilliges. 2014. In-air gestures around unmodified mobile devices. In Proceedings of the 27th annual ACM symposium on User interface software and technology - UIST '14, 319--329. Google ScholarDigital Library
- Li Sun, Souvik Sen, Dimitrios Koutsonikolas, and Kyu-Han Kim. 2015. WiDraw: Enabling Hands-free Drawing in the Air on Commodity WiFi Devices. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking - MobiCom '15, 77--89. Google ScholarDigital Library
- Vamsi Talla, Bryce Kellogg, Shyamnath Gollakota, and Joshua R. Smith. 2017. Battery-Free Cellphone. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 2: 1--20. Google ScholarDigital Library
- Danhang Tang, Tsz Ho Yu, and Tae Kyun Kim. 2013. Real-time articulated hand pose estimation using semi-supervised transductive regression forests. In Proceedings of the IEEE International Conference on Computer Vision, 3224--3231. Google ScholarDigital Library
- Thorlab. Photodiode Tutorial.Google Scholar
- Hoang Truong, Phuc Nguyen, Anh Nguyen, Nam Bui, and Tam Vu. 2017. Capacitive Sensing 3D-printed Wristband for Enriched Hand Gesture Recognition. In Proceedings of the 2017 Workshop on Wearable Systems and Applications - WearSys '17, 11--15. Google ScholarDigital Library
- Satoshi Tsuji. 2012. A tactile and proximity sensor by optical and electrical measurement. In Proceedings of IEEE Sensors.Google Scholar
- Ambuj Varshney, Andreas Soleiman, Luca Mottola, and Thiemo Voigt. 2017. Battery-free Visible Light Sensing. In Proceedings of the 4th ACM Workshop on Visible Light Communication Systems - VLCS '17, 3--8. Google ScholarDigital Library
- R. J.M. Vullers, R. van Schaijk, I. Doms, C. Van Hoof, and R. Mertens. 2009. Micropower energy harvesting. Solid-State Electronics.Google Scholar
- Juan Pablo Wachs, Mathias Kölsch, Helman Stern, and Yael Edan. 2011. Vision-based hand-gesture applications. Communications of the ACM 54, 2: 60. Google ScholarDigital Library
- Anusha Withana, Roshan Peiris, Nipuna Samarasekara, and Suranga Nanayakkara. 2015. zSense: Enabling Shallow Depth Gesture Recognition for Greater Input Expressivity on Smart Wearables. Proceedings of the ACM CHI'15 Conference on Human Factors in Computing Systems 1: 3661--3670. Google ScholarDigital Library
- Xing-Dong Yang, Tovi Grossman, Daniel Wigdor, and George Fitzmaurice. 2012. Magic Finger: Always-Available Input through Finger Instrumentation. Proceedings of the 25th annual ACM symposium on User interface software and technology - UIST '12: 147--156. Google ScholarDigital Library
- Liang Yin, Xiping Wu, and Harald Haas. 2016. Indoor visible light positioning with angle diversity transmitter. In 2015 IEEE 82nd Vehicular Technology Conference, VTC Fall 2015 - Proceedings.Google Scholar
- Bei Yuan and Eelke Folmer. 2008. Blind hero: enabling guitar hero for the visually impaired. Proceedings of the 10th international ACM SIGACCESS conference on Computers and accessibility: 169--176. Google ScholarDigital Library
- Chi Zhang, Josh Tabor, Jialiang Zhang, and Xinyu Zhang. 2015. Extending Mobile Interaction Through Near-Field Visible Light Sensing. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking - MobiCom '15, 345--357. Google ScholarDigital Library
- Chi Zhang and Xinyu Zhang. 2016. LiTell: Robust Indoor Localization Using Unmodified Light Fixtures. Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking - MobiCom '16: 230--242. Google ScholarDigital Library
- Chen Zhao, Ke-Yu Chen, Md Tanvir Islam Aumi, Shwetak Patel, and Matthew S. Reynolds. 2014. SideSwipe. In Proceedings of the 27th annual ACM symposium on User interface software and technology - UIST '14, 527--534. Google ScholarDigital Library
- Xia Zhou and Andrew T. Campbell. 2014. Visible light networking and sensing. HotWireless 2014 - Proceedings of the 1st ACM MobiCom Workshop on Hot Topics in Wireless: 55--59. Google ScholarDigital Library
- BPW34. https://www.osram.com/os/ecat/DIL%20SMT%20BPW%2034%20S/com/en/class_pim_web_catalog_103489/global/prd_pim_device_2219543/.Google Scholar
- ADG736. http://www.analog.com/media/en/technical-documentation/data-sheets/ADG736.pdf.Google Scholar
- 74HC4514. http://www.ti.com/lit/ds/schs280c/schs280c.pdf.Google Scholar
- LTC3106. http://www.analog.com/en/products/power-management/energy-harvesting/ltc3106.html.Google Scholar
- MINI-M4 for MSP432. https://www.mikroe.com/mini-msp432.Google Scholar
- MSP432P401R. http://www.ti.com/lit/ds/symlink/msp432p401r.pdf.Google Scholar
- Monsoon. http://msoon.github.io/powermonitor/PowerTool/doc/Power%20Monitor%20Manual.pdf.Google Scholar
- TS881. http://www.st.com/resource/en/datasheet/ts881.pdf.Google Scholar
- Xbox Kinect. https://www.xbox.com/en-US/kinect.Google Scholar
- Leap Motion. https://www.leapmotion.com/.Google Scholar
- Point Grab. http://www.pointgrab.com/.Google Scholar
- Crunch Fish. http://crunchfish.com/.Google Scholar
- APDS-9130. https://www.broadcom.com/products/optical-sensors/proximity-sensors/apds-9130.Google Scholar
- APDS-9190. https://www.broadcom.com/products/optical-sensors/proximity-sensors/apds-9190.Google Scholar
- ads7042. http://www.ti.com/lit/ds/symlink/ads7042.pdf.Google Scholar
Index Terms
Self-Powered Gesture Recognition with Ambient Light
Recommendations
Gesture Recognition Using Ambient Light
There is a growing interest in the scientific community to develop techniques for humans to communicate with the computing that is embedding into our environments. Researchers are already exploring ubiquitous modalities, such as radio frequency signals, ...
Multi-scenario gesture recognition using Kinect
CGAMES '12: Proceedings of the 2012 17th International Conference on Computer Games: AI, Animation, Mobile, Interactive Multimedia, Educational & Serious Games (CGAMES)Hand gesture recognition (HGR) is an important research topic because some situations require silent communication with sign languages. Computational HGR systems assist silent communication, and help people learn a sign language. In this article, a ...
Recognizing Gestures with Ambient Light
S3 '18: Proceedings of the 10th on Wireless of the Students, by the Students, and for the Students WorkshopThere is growing interest in the research community to develop techniques for humans to communicate with the computing that is embedding into our environments. Researchers are exploring various modalities, such as radio-frequency signals, to develop ...
Comments