Assessing activity recognition feedback in long-term psychology trials

ABSTRACT

The physical activities we perform throughout our daily lives tell a great deal about our goals, routines, and behavior, and as such, have been known for a while to be a key indicator for psychiatric disorders. This paper focuses on the use of a wrist-watch with integrated inertial sensors. The algorithms that deal with the data from these sensors can automatically detect the activities that the patient performed from characteristic motion patterns. Such a system can be deployed for several weeks continuously and can thus provide the consulting psychiatrist an insight in their patient's behavior and changes thereof. Since these algorithms will never be flawless, however, a remaining question is how we can support the psychiatrist in assigning confidence to these automatic detections. To this end, we present a study where visualizations at three levels from a detection algorithm are used as feedback, and examine which of these are the most helpful in conveying what activities the patient has performed. Results show that just visualizing the classifier's output performs the best, but that user's confidence in these automated predictions can be boosted significantly by visualizing earlier pre-processing steps.

References

1. O. Amft, H. Junker, and G. Troster. 2005. Detection of eating and drinking arm gestures using inertial body-worn sensors. In Proceedings of the 2005 International Symposium on Wearable Computers (ISWC '05). 160--163.
2. P. Bech, T. G. Bolwig, P. Kramp, and O. J. Rafaelsen. 1979. The Bech-Rafaelsen Mania Scale and the Hamilton Depression Scale. Acta psychiatrica Scandinavica 59, 4 (Apr 1979), 420--430.
3. Eugen Berlin and Kristof Van Laerhoven. 2012. Detecting Leisure Activities with Dense Motif Discovery. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing (UbiComp '12). ACM, 250--259.
4. CamNtech. 2015. MotionWatch 8. (2015). http://www.camntech.com/products/motionwatch/motionwatch-8-overview Accessed: August 2015.
5. K.-H. Chang, M. Y. Chen, and J. Canny. 2007. Tracking free-weight exercises. In Proceedings of the 9th International Conference on Ubiquitous Computing (UbiComp '07). Springer-Verlag, 19--37.
6. T. Choudhury, S. Consolvo, B. Harrison, and others. 2008. The Mobile Sensing Platform: An Embedded Activity Recognition System. IEEE Pervasive Computing 7, 2 (2008), 32--41.
7. Sunny Consolvo, David W. McDonald, Tammy Toscos, Mike Y. Chen, Jon Froehlich, and others. 2008. Activity Sensing in the Wild: A Field Trial of Ubifit Garden. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '08). ACM, 1797--1806.
8. P. Corkum, R. Tannock, H. Moldofsky, S. Hogg-Johnson, and T. Humphries. 2001. Actigraphy and parental ratings of sleep in children with attention-deficit/hyperactivity disorder (ADHD). Sleep 24, 3 (May 2001), 303--312.
9. Andrea Cuttone, MichaelKai Petersen, and JakobEg Larsen. 2014. Four Data Visualization Heuristics to Facilitate Reflection in Personal Informatics. In Universal Access in Human-Computer Interaction. Design for All and Accessibility Practice, Vol. 8516. Springer Intl' Publishing, 541--552.
10. Daniel Epstein, Felicia Cordeiro, Elizabeth Bales, James Fogarty, and Sean Munson. 2014. Taming Data Complexity in Lifelogs: Exploring Visual Cuts of Personal Informatics Data. In Proc. of the 2014 Conference on Designing Interactive Systems (DIS '14). ACM, 667-676. DOI: http://dx.doi.org/10.1145/2598510.2598558
11. Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. 2010. The Pascal Visual Object Classes (VOC) Challenge. International Journal of Computer Vision 88, 2 (2010), 303-338. DOI: http://dx.doi.org/10.1007/s11263-009-0275-4
12. Raffay Hamid, S. Maddi, A. Bobick, and I. Essa. 2007. Structure from Statistics - Unsupervised Activity Analysis using Suffix Trees. In Proceedings of the IEEE 11th International Conference on Computer Vision (ICCV '07). 1--8.
13. T. Holleczek, J. Schoch, B. Arnrich, and G. Troster. 2010. Recognizing turns and other snowboarding activities with a gyroscope. In Proceedings of the 2010 International Symposium on Wearable Computers (ISWC '10). 1--8.
14. Kevin Huang, Patrick J. Sparto, Sara Kiesler, Asim Smailagic, Jennifer Mankoff, and Dan Siewiorek. 2014. A Technology Probe of Wearable In-home Computer-assisted Physical Therapy. In Proceedings of the 32Nd Annual ACM Conference on Human Factors in Computing Systems (CHI '14). ACM, 2541-2550. DOI: http://dx.doi.org/10.1145/2556288.2557416
15. T. Huynh and B. Schiele. 2005. Analyzing features for activity recognition. In Proceedings of the 2005 joint conference on Smart objects and ambient intelligence: innovative context-aware services: usages and technologies (sOc-EUSAI '05). 159--163.
16. S. Intille, K. Larson, E. Tapia, and others. 2006. Using a live-in laboratory for ubiquitous computing research. In Proceedings of the 4th International Conference on Pervasive Computing (PERVASIVE' 06). Springer-Verlag, 349--365.
17. Rohit Ashok Khot, Jeewon Lee, Deepti Aggarwal, Larissa Hjorth, and Florian 'Floyd' Mueller. 2015. TastyBeats: Designing Palatable Representations of Physical Activity. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, 2933-2942. DOI: http://dx.doi.org/10.1145/2702123.2702197
18. Jakob Eg Larsen, Andrea Cuttone, and Sune Lehmann Jørgensen. 2013. QS Spiral: Visualizing Periodic Quantified Self Data. In Proceedings of CHI 2013 Workshop on Personal Informatics in the Wild: Hacking Habits for Health and Happiness.
19. J. Lester, T. Choudhury, N. Kern, G. Borriello, and B. Hannaford. 2005. A hybrid discriminative/generative approach for modeling human activities. In Proceedings of the 19th international joint conference on Artificial intelligence (IJCAI '05). 766--772.
20. James J. Lin, Lena Mamykina, Silvia Lindtner, Gregory Delajoux, and Henry B. Strub. Fish'N'Steps: Encouraging Physical Activity with an Interactive Computer Game. In Proceedings of the 8th International Conference on Ubiquitous Computing (UbiComp '06). Springer-Verlag, 261--278.
21. M. Mahdaviani and T. Choudhury. 2008. Fast and Scalable Training of Semi-Supervised CRFs with Application to Activity Recognition. In NIPS '07. MIT Press, 977--984.
22. D. Minnen, T. Starner, I. Essa, and C. Isbell. 2006. Discovering Characteristic Actions from On-Body Sensor Data. In Proceedings of the 2006 International Symposium on Wearable Computers (ISWC '06). 11--18.
23. Andreas Möller, Matthias Kranz, Barbara Schmid, Luis Roalter, and Stefan Diewald. 2013. Investigating Self-reporting Behavior in Long-term Studies. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, 2931--2940.
24. N. Oliver and E. Horvitz. 2005. A comparison of HMMs and dynamic bayesian networks for recognizing office activities. In Proceedings of the 10th international conference on User Modeling (UM '05). Springer-Verlag, 199--209.
25. J. Partridge, K.; Begole. 2009. Activity-based Advertising: Techniques and Challenges. In Workshop on Pervasive Advertising.
26. D. J. Patterson, D. Fox, H. Kautz, and M. Philipose. 2005. Fine-grained activity recognition by aggregating abstract object usage. In Proceedings of the 2005 International Symposium on Wearable Computers (ISWC '05). 44--51.
27. M. Philipose, K. Fishkin, M. Perkowitz, D. Patterson, D. Fox, H. Kautz, and D. Hahnel. 2004. Inferring activities from interactions with objects. IEEE Pervasive Computing 3, 4 (2004), 50--57.
28. T. Stiefmeier, D. Roggen, G. Ogris, and P. Lukowicz. 2008. Wearable Activity Tracking in Car Manufacturing. IEEE Pervasive Computing 7, 2 (2008), 42--50.
29. D. Tacconi, O. Mayora, P. Lukowicz, B. Arnrich, G. Tröster, and C. Haring. 2007. On the feasibility of using activity recognition and context aware interaction to support early diagnosis of bipolar disorder. In Ubiwell Workshop '07. 206--209.
30. M. H. Teicher. 1995. Actigraphy and motion analysis: new tools for psychiatry. Harvard review of psychiatry 3, 1 (Jun 1995), 18--35.
31. A. Vahdatpour, N. Amini, and M. Sarrafzadeh. 2009. Toward unsupervised activity discovery using multi-dimensional motif detection in time series. In Proceedings of the 21st International Jont Conference on Artifical Intelligence (IJCAI'09). 1261--1266.
32. K. Van Laerhoven, E. Berlin, and B. Schiele. 2009. Enabling Efficient Time Series Analysis for Wearable Activity Data. In Proceedings of the 8th International Conference on Machine Learning and Applications (ICMLA '09). IEEE, 392--397.
33. K. Van Laerhoven, H.-W. Gellersen, and Y. G. Malliaris. 2006. Long term activity monitoring with a wearable sensor node. In Proceedings of the International Workshop on Wearable and Implantable Body Sensor Networks (BSN '06). 174--177.
34. F. Wilhelm, M. Pfaltz, and P. Grossman. 2006. Continuous electronic data capture of physiology, behavior and experience in real life: towards ecological momentary assessment of emotion. Interacting with Computers 18, 2 (2006), 171--186.

Index Terms

• Human-centered computing

  • Ubiquitous and mobile computing

  • Visualization