A camera-based interface for interaction with mobile handheld computers

ABSTRACT

Recent advances in mobile computing allow the users to deal with 3D interactive graphics on handheld computers. Although the computing resources and screen resolutions grow steadily, user interfaces for handheld computers do not change significantly. Consequently, we designed a new 3-DOF interface adapted to the characteristics of handheld computers. This interface tracks the movement of a target that the user holds behind the screen by analyzing the video stream of the handheld computer camera. The position of the target is directly inferred from the color-codes that are printed on it using an efficient algorithm. The users can easily interact in real-time in a mobile setting. The visualization of the data is good as the target does not occlude the screen and the interaction techniques are not dependent on the orientation of the handheld computer. We used the interface in several test applications for the visualization of large images such as maps, the manipulation of 3D models, and the navigation in 3D scenes. This new interface favors the development of 2D and 3D interactive applications on handheld computers.

References

1. Balakrishnan, R., and Hinckley, K. 1999. The role of kinesthetic reference frames in two-handed input performance. In Proceedings of the 12th annual ACM symposium on User interface software and technology, ACM Press, 171--178.
2. Bederson, B. B., and Hollan, J. D. 1994. Pad++: A zooming graphical interface for exploring alternate interface physics. In Proceedings of the 7th Annual ACM Symposium on User Interface Software and Technology (UIST'94), 17--26.
3. Buxton, W., and Myers, B. 1986. A study in two-handed input. In Proceedings of the SIGCHI conference on Human factors in computing systems, ACM Press, 321--326.
4. Dusan, S., Gadbois, G., and Flanagan, J. 2003. Multi-modal interaction on pda's integrating speech and pen inputs. In Proceedings of EUROSPEECH, 2225--2228.
5. Encarnao, L., Bimber, O., Schmalstieg, D., and Chandler, S. 1999. A translucent sketchpad for the virtual table exploring motion-based gesture recognition. In Proceedings of Eurographics '99, 179--190.
6. Fallman, D., Lund, A., and Wiberg, M. 2004. Scrollpad: Tangible scrolling with mobile devices. In Proceedings of the 37th Annual Hawaii International Conference on System Sciences (HICSS'04) - Track 9 (To appear).
7. Fitzmaurice, G. W., Zhai, S., and Chignell, M. H. 1993. Virtual reality for palmtop computers. ACM Trans. Inf. Syst. 11, 3, 197--218.
8. Fitzmaurice, G. W. 1993. Situated information spaces and spatially aware palmtop computers. Communications of the ACM.
9. Gong, J., and Tarasewich, P. 2004. Guidelines for handheld device interface design. In Proceedings of DSI 2004 (To appear).
10. Hinckley, K., Sinclair, M., Hanson, E., Szeliski, R., and Conway, M. 1999. The VideoMouse: a camera-based multi-degree-of-freedom input device. In UIST '99: Proceedings of the 12th annual ACM symposium on User interface software and technology, ACM Press, 103--112.
11. Hinckley, K., Pierce, J. S., Sinclair, M., and Horvitz, E. 2000. Sensing techniques for mobile interaction. In Proceedings of the 13th annual ACM symposium on User Interface Software and Technology (UIST 2000), 91--100.
12. Igarashi, T., and Hinckley, K. 2000. Speed-dependent automatic zooming for browsing large documents. In Proceedings of the 13th annual ACM symposium on User Interface Software and Technology (UIST 2000), ACM Press, 139--148.
13. Kjedskov, J., and Skov, M. B. 2002. Interaction design for handheld computers. In Proceedings of the 5th Asian Pacific Conference on Human-Computer Interaction, APCHI 2002, Science Press.
14. Krum, D. M., Melby, R., Ribarsky, W., and Hodges, L. 2003. Isometric pointer interfaces for wearable 3D visualization. In CHI '03 extended abstracts on Human factors in computing systems, ACM Press, 774--775.
15. Mine, M. 1998. Exploiting proprioception in virtual-environment interaction. PhD thesis, University of North Carolina at Chapel Hill.
16. Pierce, J., Mahaney, H., and Abowd, G. 2003. Opportunistic annexing for handheld devices: Opportunities and challenges. Tech. rep., Georgia Institute of Technology GIT-GVU-03-31.
17. Pouderoux, J., Gonzato, J.-C., Guitton, P., and Granier, X., 2004. A software for reconstructing 3d-terrains from scanned maps. ACM SIGGRAPH 2004 Sketches and Applications, http://iparla.labri.fr/softwares/automnt/.
18. Rantakokko, T., and Plomp, J. 2003. An adaptive map-based interface for situated services. In Proceedings of Smart Object Conference.
19. Rekimoto, J., and Nagao, K. 1995. The world through the computer: Computer augmented interaction with real world environments. In ACM Symposium on User Interface Software and Technology (UIST 1995), 29--36.
20. Rekimoto, J. 1996. Tilting operations for small screen interfaces. In ACM Symposium on User Interface Software and Technology (UIST 96), 167--168.
21. Rohs, M. 2004. Real-world interaction with camera-phones. In 2nd International Symposium on Ubiquitous Computing Systems (UCS 2004).
22. Shneiderman, B. 1997. Designing the User Interface: Strategies for Effective Human-Computer Interaction. Addison-Wesley Longman Publishing Co., Inc.
23. Wagner, D., and Schmalstieg, D. 2003. First steps towards handheld augmented reality. In Proceedings of Seventh IEEE International Symposium on Wearable Computers.
24. Woods, E., Mason, P., and Billinghurst, M. 2003. Magic-mouse: an inexpensive 6-degree-of-freedom mouse. In Proceedings of the 1st international conference on Computer graphics and interactive techniques in Austalasia and South East Asia, ACM Press, 285--286.
25. Yee, K.-P. 2003. Peephole displays: pen interaction on spatially aware handheld computers. In Proceedings of the conference on Human factors in computing systems, ACM Press, 1--8.

Index Terms

• Hardware

  • Communication hardware, interfaces and storage

    • Buses and high-speed links

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction paradigms