ProxemicUI: object-oriented middleware and event model for proxemics-aware applications on large displays

ABSTRACT

Interpersonal spatial relationships ("proxemics") play an important role when people collaborate or work near one another. While state-of-art tracking systems and toolkits have been demonstrated that are able to provide proxemics data, application developers require the ability to define their own custom proxemics events (i.e., the meaning behind specific spatial configurations) rather than conduct manual tests in the UI layer. At the same time, proxemics-aware applications involving shared displays need mechanisms to tightly integrate UI events and proxemics events. This requires a middleware framework that allows developers to define proxemics events by composing low-level proxemics data and optionally UI events. In this paper, we present a proximity-based event model suited to collaboration around large displays, and a corresponding framework for building proxemics-aware applications. The design of this model is derived from a review of prior work, and direct experience implementing and evaluating a proxemics-aware interactive tabletop application (a museum information kiosk).

References

1. Michelle Annett, Tovi Grossman, Daniel Wigdor, and George Fitzmaurice. 2011. Medusa: a proximity-aware multi-touch tabletop. In Proceedings of the 24th annual ACM symposium on User interface software and technology (UIST '11), 337--346. DOI=10.1145/2047196.2047240 http://doi.acm.org/10.1145/2047196.2047240
2. Alec Azad, Jaime Ruiz, Daniel Vogel, Mark Hancock, and Edward Lank. 2012. Territoriality and behaviour on and around large vertical publicly-shared displays. In Proceedings of the Designing Interactive Systems Conference (DIS '12), 468--477. DOI=10.1145/2317956.2318025 http://doi.acm.org/10.1145/2317956.2318025
3. Till Ballendat, Nicolai Marquardt, and Saul Greenberg. 2010. Proxemic interaction: designing for a proximity and orientation-aware environment. In ACM International Conference on Interactive Tabletops and Surfaces (ITS '10), 121--130. DOI=10.1145/1936652.1936676 http://doi.acm.org/10.1145/1936652.1936676
4. W. Keith Edwards, Mark W. Newman, and Erika Shehan Poole. 2010. The infrastructure problem in HCI. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). ACM, New York, NY, USA, 423--432. DOI=http://dx.doi.org/10.1145/1753326.1753390
5. Shenfeng Fei, Andruid Kerne, Ajit Jain, Andrew M. Webb, and Yin Qu. 2013. Positioning portals with peripheral NFC tags to embody trans-surface interaction. In Proceedings of the 2013 ACM international conference on Interactive tabletops and surfaces(ITS'13),317--320. DOI=10.1145/2512349.2514593 http://doi.acm.org/10.1145/2512349.2514593
6. Saul Greenberg, Nicolai Marquardt, Till Ballendat, Rob Diaz-Marino, and Miaosen Wang. 2011. Proxemic interactions: the new ubicomp?. interactions 18, 1 (Jan 2011), 42--50. DOI=10.1145/1897239.1897250 http://doi.acm.org/10.1145/1897239.1897250
7. Edward T. Hall. 1966. The Hidden Dimension. Doubleday.
8. Uta Hinrichs, Sheelagh Carpendale, and Stacey D. Scott. 2006. Evaluating the effects of fluid interface components on tabletop collaboration. In Proceedings of the working conference on Advanced visual interfaces (AVI '06), 27--34. DOI=10.1145/1133265.1133272 http://doi.acm.org/10.1145/1133265.1133272
9. Gang Hu, Derek Reilly, Mohammed Alnusayri, Ben Swinden, and Qigang Gao. 2014. DT-DT: Top-down Human Activity Analysis for Interactive Surface Applications. In Proceedings of the Ninth ACM International Conference on Interactive Tabletops and Surfaces (ITS '14), 167--176. DOI=10.1145/2669485.2669501 http://doi.acm.org/10.1145/2669485.2669501
10. Abhijit Karnik, Diego Martinez Plasencia, Walterio Mayol-Cuevas, and Sriram Subramanian. 2012. PiVOT: personalized view-overlays for tabletops. In Proceedings of the 25th annual ACM symposium on User interface software and technology (UIST '12), 271--280. DOI=10.1145/2380116.2380151 http://doi.acm.org/10.1145/2380116.2380151
11. Adam Kendon (1990). Conducting Interaction: Patterns of behavior in focused encounters. Studies in Interactional Sociolinguistics (7), Cambridge University Press.
12. Ahmed Kharrufa, Roberto Martinez-Maldonado, Judy Kay, and Patrick Olivier. 2013. Extending tabletop application design to the classroom. In Proceedings of the 2013 ACM international conference on Interactive tabletops and surfaces (ITS '13), 115--124. DOI=10.1145/2512349.2512816 http://doi.acm.org/10.1145/2512349.2512816
13. Daniel Klinkhammer, Markus Nitsche, Marcus Specht, and Harald Reiterer. 2011. Adaptive personal territories for co-located tabletop interaction in a museum setting. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS '11), 107--110. DOI=10.1145/2076354.2076375 http://doi.acm.org/10.1145/2076354.2076375
14. Nicolai Marquardt, Robert Diaz-Marino, Sebastian Boring, and Saul Greenberg. 2011. The proximity toolkit: prototyping proxemic interactions in ubiquitous computing ecologies. In Proceedings of the 24th annual ACM symposium on User interface software and technology (UIST '11), 315--326. DOI=10.1145/2047196.2047238 http://doi.acm.org/10.1145/2047196.2047238
15. Nicolai Marquardt, Till Ballendat, Sebastian Boring, Saul Greenberg, and Ken Hinckley. 2012. Gradual engagement: facilitating information exchange between digital devices as a function of proximity. In Proceedings of the 2012 ACM international conference on Interactive tabletops and surfaces (ITS '12). ACM, New York, NY, USA, 31--40. DOI=http://dx.doi.org/10.1145/2396636.2396642
16. Paul Marshall, Yvonne Rogers, and Nadia Pantidi. 2011. Using F-formations to analyse spatial patterns of interaction in physical environments. In Proceedings of CSCW '11. 445--454.
17. Paul Marshall, Richard Morris, Yvonne Rogers, Stefan Kreitmayer, and Matt Davies. 2011. Rethinking 'multi-user': an in-the-wild study of how groups approach a walk-up-and-use tabletop interface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11), 3033-3042. DOI=10.1145/1978942.1979392 http://doi.acm.org/10.1145/1978942.1979392
18. Max Möllers and Jan Borchers. 2011. TaPS widgets: interacting with tangible private spaces. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS '11), 75--78. DOI=10.1145/2076354.2076369 http://doi.acm.org/10.1145/2076354.2076369
19. Meredith Ringel Morris, Anqi Huang, Andreas Paepcke, and Terry Winograd. 2006. Cooperative gestures: multi-user gestural interactions for co-located groupware. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '06), 1201--1210. DOI=10.1145/1124772.1124952 http://doi.acm.org/10.1145/1124772.1124952
20. Kenton O'Hara, Jesper Kjeldskov, and Jeni Paay (2011). Blended Interaction Spaces for Distributed Team Collaboration. ACM TOCHI 18(1).
21. Anne Marie Piper, Eileen O'Brien, Meredith Ringel Morris, and Terry Winograd. 2006. SIDES: a cooperative tabletop computer game for social skills development. In Proceedings of the 2006 20th anniversary conference on Computer supported cooperative work (CSCW '06), 1--10. DOI=10.1145/1180875.1180877 http://doi.acm.org/10.1145/1180875.1180877
22. Jochen Rick, Amanda Harris, Paul Marshall, Rowanne Fleck, Nicola Yuill, and Yvonne Rogers. 2009. Children designing together on a multi-touch tabletop: an analysis of spatial orientation and user interactions. In Proceedings of the 8th International Conference on Interaction Design and Children (IDC '09), 106--114. DOI=10.1145/1551788.1551807 http://doi.acm.org/10.1145/1551788.1551807
23. Constantin Schmidt, Jörg Müller, and Gilles Bailly. 2013. Screenfinity: extending the perception area of content on very large public displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13), 1719--1728. DOI=10.1145/2470654.2466227 http://doi.acm.org/10.1145/2470654.2466227
24. Stacey D. Scott, M. Sheelagh T. Carpendale, and Kori M. Inkpen. 2004. Territoriality in collaborative tabletop workspaces. In Proceedings of the 2004 ACM conference on Computer supported cooperative work (CSCW '04), 294--303. DOI=10.1145/1031607.1031655 http://doi.acm.org/10.1145/1031607.1031655
25. Stacey D. Scott. 2003. Territory-based interaction techniques for tabletop collaboration. In Proceedings of Conference Companion of the ACM Symposium on User Interface Software and Technology (UIST '03).
26. Anthony Tang, Melanie Tory, Barry Po, Petra Neumann, and Sheelagh Carpendale. 2006. Collaborative coupling over tabletop displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '06), 1181--1190. DOI=10.1145/1124772.1124950 http://doi.acm.org/10.1145/1124772.1124950
27. Daniel Vogel and Ravin Balakrishnan. 2004. Interactive public ambient displays: transitioning from implicit to explicit, public to personal, interaction with multiple users. In Proceedings of the 17th annual ACM symposium on User interface software and technology (UIST '04), 137--146. DOI=10.1145/1029632.1029656 http://doi.acm.org/10.1145/1029632.1029656
28. Miaosen Wang, Sebastian Boring, and Saul Greenberg. 2012. Proxemic peddler: a public advertising display that captures and preserves the attention of a passerby. In Proceedings of the 2012 International Symposium on Pervasive Displays (PerDis '12). ACM, New York, NY, USA, Article 3, 6 pages. DOI=http://dx.doi.org/10.1145/2307798.2307801
29. Daniel Wigdor, Dennis Wixon. 2011. Brave NUI World: designing natural user interfaces for touch and gesture. Morgan Kaufmann Publisher.
30. Mike Wu and Ravin Balakrishnan. 2003. Multi-finger and whole hand gestural interaction techniques for multi-user tabletop displays. In Proceedings of the 16th annual ACM symposium on User interface software and technology (UIST '03), 193--202. DOI=10.1145/964696.964718 http://doi.acm.org/10.1145/964696.964718

Index Terms

• Human-centered computing

  • Collaborative and social computing

  • Human computer interaction (HCI)