ExtraSensory App: Data Collection In-the-Wild with Rich User Interface to Self-Report Behavior

ABSTRACT

We introduce a mobile app for collecting in-the-wild data, including sensor measurements and self-reported labels describing people's behavioral context (e.g., driving, eating, in class, shower). Labeled data is necessary for developing context-recognition systems that serve health monitoring, aging care, and more. Acquiring labels without observers is challenging and previous solutions compromised ecological validity, range of behaviors, or amount of data. Our user interface combines past and near-future self-reporting of combinations of relevant context-labels. We deployed the app on the personal smartphones of 60 users and analyzed quantitative data collected in-the-wild and qualitative user-experience reports. The interface's flexibility was important to gain frequent, detailed labels, support diverse behavioral situations, and engage different users: most preferred reporting their past behavior through a daily journal, but some preferred reporting what they're about to do. We integrated insights from this work back into the app, which we make available to researchers for conducting in-the-wild studies.

References

1. Frank Bentley and Konrad Tollmar. 2013. The Power of Mobile Notifications to Increase Wellbeing Logging Behavior. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 1095--1098.
2. Daniel Castro, Steven Hickson, Vinay Bettadapura, Edison Thomaz, Gregory Abowd, Henrik Christensen, and Irfan Essa. 2015. Predicting daily activities from egocentric images using deep learning. In Proceedings of the 2015 ACM International Symposium on Wearable Computers. ACM, 75--82.
3. Tanzeem Choudhury, Sunny Consolvo, Beverly Harrison, Jeffrey Hightower, Anthony LaMarca, Louis LeGrand, Ali Rahimi, Adam Rea, G Bordello, Bruce Hemingway, and others. 2008. The mobile sensing platform: An embedded activity recognition system. IEEE Pervasive Computing 7, 2 (2008).
4. Sunny Consolvo, David W McDonald, Tammy Toscos, Mike Y Chen, Jon Froehlich, Beverly Harrison, Predrag Klasnja, Anthony LaMarca, Louis LeGrand, Ryan Libby, and others. 2008. Activity sensing in the wild: a field trial of ubifit garden. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1797--1806.
5. Mihaly Csikszentmihalyi and Reed Larson. 2014. Validity and reliability of the experience-sampling method. In Flow and the foundations of positive psychology. Springer, 35--54.
6. Yujie Dong, Jenna Scisco, Mike Wilson, Eric Muth, and Adam Hoover. 2014. Detecting periods of eating during free-living by tracking wrist motion. IEEE journal of biomedical and health informatics 18, 4 (2014), 1253--1260.
7. Katherine Ellis, Suneeta Godbole, Jacqueline Kerr, and Gert Lanckriet. 2014. Multi-Sensor physical activity recognition in free-living. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication. ACM, 431--440.
8. Raghu Kiran Ganti, Soundararajan Srinivasan, and Aca Gacic. 2010. Multisensor fusion in smartphones for lifestyle monitoring. In 2010 International Conference on Body Sensor Networks. IEEE, 36--43.
9. John J Guiry, Pepijn van de Ven, and John Nelson. 2014. Multi-Sensor Fusion for Enhanced Contextual Awareness of Everyday Activities with Ubiquitous Devices. Sensors 14 (2014), 5687--5701.
10. Erik B Hekler, Predrag Klasnja, Vicente Traver, and Monique Hendriks. 2013. Realizing effective behavioral management of health: the metamorphosis of behavioral science methods. IEEE pulse 4, 5 (2013), 29--34.
11. Daniel Kahneman, Alan B Krueger, David A Schkade, Norbert Schwarz, and Arthur A Stone. 2004. A survey method for characterizing daily life experience: The day reconstruction method. Science 306, 5702 (2004), 1776--1780.
12. Jacqueline Kerr, Ruth E Patterson, Katherine Ellis, Suneeta Godbole, Eileen Johnson, Gert Lanckriet, and John Staudenmayer. 2016. Objective Assessment of Physical Activity: Classifiers for Public Health. Medicine and science in sports and exercise 48, 5 (2016), 951--957.
13. Adil Mehmood Khan, Ali Tufail, Asad Masood Khattak, and Teemu H Laine. 2014. Activity recognition on smartphones via sensor-fusion and kda-based svms. International Journal of Distributed Sensor Networks 2014 (2014).
14. Abby C King, Eric B Hekler, Lauren A Grieco, Sandra J Winter, Jylana L Sheats, Matthew P Buman, Banny Banerjee, Thomas N Robinson, and Jesse Cirimele. 2013. Harnessing different motivational frames via mobile phones to promote daily physical activity and reduce sedentary behavior in aging adults. PloS one 8, 4 (2013), e62613.
15. Kai Kunze and Paul Lukowicz. 2014. Sensor placement variations in wearable activity recognition. IEEE Pervasive Computing 13, 4 (2014), 32--41.
16. Matthew L Lee and Anind K Dey. 2015. Sensor-based observations of daily living for aging in place. Personal and Ubiquitous Computing 19, 1 (2015), 27--43.
17. Beth Logan and others. 2000. Mel Frequency Cepstral Coefficients for Music Modeling.. In ISMIR.
18. Gloria Mark, Shamsi T. Iqbal, Mary Czerwinski, Paul Johns, and Akane Sano. 2016. Neurotics Can'T Focus: An in Situ Study of Online Multitasking in the Workplace. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 1739--1744.
19. Abhinav Mehrotra, Veljko Pejovic, Jo Vermeulen, Robert Hendley, and Mirco Musolesi. 2016. My Phone and Me: Understanding People's Receptivity to Mobile Notifications. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 1021--1032.
20. Hamed Pirsiavash and Deva Ramanan. 2012. Detecting activities of daily living in first-person camera views. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on. IEEE, 2847--2854.
21. Susanna Pirttikangas, Kaori Fujinami, and Tatsuo Nakajima. 2006. Feature selection and activity recognition from wearable sensors. In International Symposium on Ubiquitious Computing Systems. Springer, 516--527.
22. Tauhidur Rahman, Mi Zhang, Stephen Voida, and Tanzeem Choudhury. 2014. Towards Accurate Non-Intrusive Recollection of Stress Levels Using Mobile Sensing and Contextual Recall. In International Conference on Pervasive Computing Technologies for Healthcare.
23. Christie Napa Scollon, Chu-Kim Prieto, and Ed Diener. 2009. Experience sampling: promises and pitfalls, strength and weaknesses. In Assessing well-being. Springer, 157--180.
24. Allan Stisen, Henrik Blunck, Sourav Bhattacharya, Thor Siiger Prentow, Mikkel Baun Kjærgaard, Anind Dey, Tobias Sonne, and Mads Møller Jensen. 2015. Smart Devices are Different: Assessing and MitigatingMobile Sensing Heterogeneities for Activity Recognition. In Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems. ACM, 127--140.
25. Edison Thomaz, Irfan Essa, and Gregory D Abowd. 2015a. A practical approach for recognizing eating moments with wrist-mounted inertial sensing. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 1029--1040.
26. Edison Thomaz, Cheng Zhang, Irfan Essa, and Gregory D Abowd. 2015b. Inferring Meal Eating Activities in Real World Settings from Ambient Sounds: A Feasibility Study. In Proceedings of the 20th International Conference on Intelligent User Interfaces. ACM, 427--431.
27. Yonatan Vaizman, Katherine Ellis, and Gert Lanckriet. 2017a. Recognizing Detailed Human Context In-the-Wild from Smartphones and Smartwatches. IEEE Pervasive Computing 16, 4 (2017).
28. Yonatan Vaizman, Nadir Weibel, and Gert Lanckriet. 2017b. Behavioral Context Recognition In-the-Wild: Unified Model for Multi-Modal Sensors and Multi-Label Classification. Under review for Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 4 (2017).
29. Evan Welbourne and Emmanuel Munguia Tapia. 2014. CrowdSignals: a call to crowdfund the community's largest mobile dataset. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication. ACM, 873--877.
30. Hao Yan and Ted Selker. 2000. Context-aware office assistant. In Proceedings of the 5th international conference on Intelligent user interfaces. ACM, 276--279.
31. Yi-Hsuan Yang and Yuan-Ching Teng. 2015. Quantitative Study of Music Listening Behavior in a Smartphone Context. ACM Trans. Interact. Intell. Syst. 5, 3, Article 14 (Sept. 2015), 30 pages.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)