Multimodal Indoor Social Interaction Sensing and Real-time Feedback for Behavioural Intervention

ABSTRACT

Social interactions play an important role in people's personal as well as working life. Interactions come in various forms, identifiable mainly by duration and proximity. The ability to detect and distinguish interactions can often shed light over worktask performance, epidemic spreading, personal relationship development, use of space and more. Questionnaires and direct observations have often been used as mechanisms to identify interactions, however, these are either very expensive in terms of staff time, yield very coarse grained information or do not scale. Technology has started cutting costs by allowing automatic detection, however precise interaction identification often requires individuals to wear custom hardware. The aim of my work is to exploit the capabilities of off-the-shelf wearable devices (i.e. smart watches and fitness trackers) to build a social interactions sensing platform which offers accuracy and scalability. To this end, non-verbal behaviours, such as, body language, will be considered in addition to the occurrence of the interactions (individuals involved, duration and location) with the objective of providing unobtrusive real-time feedback.

References

1. N. Aharony, W. Pan, C. Ip, I. Khayal, and A. Pentland. Social fmri: Investigating and shaping social mechanisms in the real world. Pervasive and Mobile Computing, 2011.
2. J. Alvina, S. T. Perrault, T. Roumen, S. Zhao, M. Azh, and M. Fjeld. Omnivib: Towards cross-body spatiotemporal vibrotactile notifications for mobile phones. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 2015.
3. T. Baur, I. Damian, F. Lingenfelser, J. Wagner, and E. André. Nova: Automated analysis of nonverbal signals in social interactions. In Human Behavior Understanding. Springer, 2013.
4. N. M. Bradburn, S. Sudman, E. Blair, and C. Stocking. Question threat and response bias. Public Opinion Quarterly, 1978.
5. C. Brown, C. Efstratiou, I. Leontiadis, D. Quercia, and C. Mascolo. Tracking serendipitous interactions: How individual cultures shape the office. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work. ACM, 2014.
6. C. Brown, C. Efstratiou, I. Leontiadis, D. Quercia, C. Mascolo, J. Scott, and P. Key. The architecture of innovation: Tracking face-to-face interactions with ubicomp technologies. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 2014.
7. R. S. Burt. Structural holes and good ideas. American journal of sociology, 110(2), 2004.
8. J. M. Cabero, V. Molina, I. Urteaga, F. Liberal, and J. L. Martın. Acquisition of human traces with Bluetooth technology: Challenges and proposals. Ad Hoc Networks, 12, 2014.
9. D. R. Carney, A. J. Cuddy, and A. J. Yap. Power posing brief nonverbal displays affect neuroendocrine levels and risk tolerance. Psychological Science, 2010.
10. T. K. Choudhury. Sensing and modeling human networks. PhD thesis, Massachusetts Institute of Technology, 2003.
11. I. Damian, C. S. S. Tan, T. Baur, J. Schöning, K. Luyten, and E. André. Augmenting social interactions: Realtime behavioural feedback using social signal processing techniques. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 2015.
12. A.-L. Fayard and J. Weeks. Photocopiers and water-coolers: The affordances of informal interaction. Organization studies, 2007.
13. W. Huang, Y.-S. Kuo, P. Pannuto, and P. Dutta. Opo: a wearable sensor for capturing high-fidelity face-to-face interactions. In Proceedings of the 12th ACM Conference on Embedded Network Sensor Systems. ACM, 2014.
14. P. Jeffrey and A. McGrath. Sharing serendipity in the workplace. In Proceedings of the Third International Conference on Collaborative Virtual Environments. ACM, 2000.
15. S. Liu, Y. Jiang, and A. Striegel. Face-to-Face Proximity Estimation Using Bluetooth On Smartphones. IEEE Transactions on Mobile Computing, 2014.
16. P. Lopes, A. Ion, W. Mueller, D. Hoffmann, P. Jonell, and P. Baudisch. Proprioceptive interaction. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 2015.
17. M. S. Mast. On the importance of nonverbal communication in the physician--patient interaction. Patient education and counseling, 2007.
18. T. V. McGovern and H. E. Tinsley. Interviewer evaluations of interviewee nonverbal behavior. Journal of Vocational Behavior, 1978.
19. D. O. Olguın and A. S. Pentland. Social sensors for automatic data collection. Proceedings of the 14th Americas Conference on Information Systems (AMCIS), 2008.
20. A. Pentland. The new science of building great teams. Harvard Business Review, 90, 2012.
21. I. Rossberg-Gempton and G. D. Poole. The effect of open and closed postures on pleasant and unpleasant emotions. The Arts in psychotherapy, 1993.
22. T. Roumen, S. T. Perrault, and S. Zhao. Notiring: A comparative study of notification channels for wearable interactive rings. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 2015.
23. K. Sailer, R. Pachilova, and C. Brown. Human versus machine-testing validity and insights of manual and automated data gathering methods in complex buildings. In Proceedings of the 9th International Space Syntax Symposium. Sejong University Press, 2013.
24. J. B. Stryker and M. D. Santoro. Facilitating face-to-face communication in high-tech teams. Research-Technology Management, 2012.
25. W. F. Whyte. Street corner society: The social structure of an Italian slum. University of Chicago Press, 2012.
26. E. Yoneki and J. Crowcroft. Epimap: Towards quantifying contact networks for understanding epidemiology in developing countries. Ad Hoc Networks, 2014.

Index Terms

• Information systems

  • Information systems applications