A screen-space formulation for 2D and 3D direct manipulation

ABSTRACT

Rotate-Scale-Translate (RST) interactions have become the de facto standard when interacting with two-dimensional (2D) contexts in single-touch and multi-touch environments. Because the use of RST has thus far focused almost entirely on 2D, there are not yet standard techniques for extending these principles into three dimensions. In this paper we describe a screen-space method which fully captures the semantics of the traditional 2D RST multi-touch interaction, but also allows us to extend these same principles into three-dimensional (3D) interaction. Just like RST allows users to directly manipulate 2D contexts with two or more points, our method allows the user to directly manipulate 3D objects with three or more points. We show some novel interactions, which take perspective into account and are thus not available in orthographic environments. Furthermore, we identify key ambiguities and unexpected behaviors that arise when performing direct manipulation in 3D and offer solutions to mitigate the difficulties each presents. Finally, we show how to extend our method to meet application-specific control objectives, as well as show our method working in some example environments.

References

1. Agarawala, A. and Balakrishnan, R. 2006. Keepin' it real: pushing the desktop metaphor with physics, piles and the pen. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Montréal, Québec, Canada, April 22--27, 2006). R. Grinter, T. Rodden, P. Aoki, E. Cutrell, R. Jeffries, and G. Olson, Eds. CHI '06. ACM, New York, NY, 1283--1292.
2. Chen, M., Mountford, S. J., and Sellen, A. 1988. A study in interactive 3-D rotation using 2-D control devices. In Proceedings of the 15th Annual Conference on Computer Graphics and interactive Techniques R. J. Beach, Ed. SIGGRAPH '88. ACM, New York, NY, 121--129.
3. Cutler, L. D., Fröhlich, B., and Hanrahan, P. 1997. Two-handed direct manipulation on the responsive workbench. In Proceedings of the 1997 Symposium on interactive 3D Graphics (Providence, Rhode Island, United States, April 27--30, 1997). SI3D '97. ACM, New York, NY, 107--114. DOI=http://doi.acm.org/10.1145/253284.253315
4. Fröhlich, B., Tramberend, H., Beers, A., Agrawala, M., and Baraff, D. 2000. Physically-Based Manipulation on the Responsive Workbench. In Proceedings of the IEEE Virtual Reality 2000 Conference (March 18--22, 2000). VR. IEEE Computer Society, Washington, DC, 5.
5. Gleicher, M. and Witkin, A. 1992. Through-the-Lens Camera Control. In Proceedings of the 19th Annual Conference on Computer Graphics and Interactive Techniques J. J. Thomas, Ed. SIGGRAPH '92. ACM, New York, NY, 331--340.
6. Grossman, T., Wigdor, D., and Balakrishnan, R. 2004. Multi-finger gestural interaction with 3d volumetric displays. In Proceedings of the 17th Annual ACM Symposium on User interface Software and Technology (Santa Fe, NM, USA, October 24--27, 2004). UIST '04. ACM, New York, NY, 61--70. DOI=http://doi.acm.org/10.1145/1029632.1029644
7. Hancock, M., Carpendale, S., and Cockburn, A. 2007. Shallow-depth 3d interaction: design and evaluation of one-, two- and three-touch techniques. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (San Jose, California, USA, April 28--May 03, 2007). CHI '07. ACM, New York, NY, 1147--1156.
8. Kyung, M., Kim, M., and Hong, S. J. 1996. A new approach to through-the-lens camera control. Graph. Models Image Process. 58, 3 (May. 1996), 262--285.
9. Nielson, G. M. and Olsen, D. R. 1987. Direct manipulation techniques for 3D objects using 2D locator devices. In Proceedings of the 1986 Workshop on interactive 3D Graphics (Chapel Hill, North Carolina, United States). F. Crow and S. M. Pizer, Eds. SI3D '86. ACM, New York, NY, 175--182.
10. Nocedal, J. and Wright, M. 1999. Numerical Optimization. Springer, New York.
11. Shoemake, K. 1992. ARCBALL: a user interface for specifying three-dimensional orientation using a mouse. In Proceedings of the Conference on Graphics interface '92 (Vancouver, British Columbia, Canada). K. S. Booth and A. Fournier, Eds. Morgan Kaufmann Publishers, San Francisco, CA, 151--156.
12. Snibbe, S. S., Herndon, K. P., Robbins, D. C., Conner, D. B., and van Dam, A. 1992. Using deformations to explore 3D widget design. In Proceedings of the 19th Annual Conference on Computer Graphics and interactive Techniques J. J. Thomas, Ed. SIGGRAPH '92. ACM, New York, NY, 351--352.
13. Wilson, A. D., Izadi, S., Hilliges, O., Garcia-Mendoza, A., and Kirk, D. 2008. Bringing physics to the surface. In Proceedings of the 21st Annual ACM Symposium on User interface Software and Technology (Monterey, CA, USA, October 19--22, 2008). UIST '08. ACM, New York, NY, 67--76.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction devices

      • Touch screens

    • Interaction paradigms