Large document, small screen: a camera driven scroll and zoom control for mobile devices

ABSTRACT

We present a three degree-of-freedom control designed for viewing large documents and images on a mobile device equipped with a camera. Tracking natural features detected in the camera's field of view, we can roughly estimate the motion of the device, using the results to scroll and zoom the current document. Central to our implementation is the manner by which we amplify the motion, allowing the user to scroll through large portions of the document with minimal hand movement. Then, using a Hidden Markov Model, we determine when the user is scrolling, zooming, or some combination of the two, thus providing smoother, more fluid control. We demonstrate a prototype of our 3DOF control that can easily navigate documents that are many times larger than the display area, and show how it might be incorporated into a larger document retrieval application.

REFERENCES

• 1: Ballagas, R., Rohs, M., and Sheridan, J. G. 2005. Mobile phones as pointing devices. In Workshop on Pervasive Mobile Interaction Devices (PERMID).
• 2: Bouguet, J. Y., 2001. Pyramidal implementation of the Lucas Kanade feature tracker. (http://sourceforge.net/projects/opencvlibrary).
• 3: Capin, T., Haro, A., Setlur, V., and Wilkinson, S. 2006. Camera-based virtual environment interaction on mobile devices. Lecture Notes in Computer Science 4263, 765--773.
• 4: George W. Fitzmaurice, Situated information spaces and spatially aware palmtop computers, Communications of the ACM, v.36 n.7, p.39-49, July 1993 [doi>10.1145/159544.159566]
• 5: Martin Hachet , Joachim Pouderoux , Pascal Guitton, A camera-based interface for interaction with mobile handheld computers, Proceedings of the 2005 symposium on Interactive 3D graphics and games, April 03-06, 2005, Washington, District of Columbia [doi>10.1145/1053427.1053438]
• 6: Seokhee Jeon , Jane Hwang , Gerard J. Kim , Mark Billinghurst, Interaction techniques in large display environments using hand-held devices, Proceedings of the ACM symposium on Virtual reality software and technology, November 01-03, 2006, Limassol, Cyprus [doi>10.1145/1180495.1180516]
• 7: Haruhisa Kato , Akio Yoneyama , Yasuhiro Takishima, An intuitive interface based on camera parameters for portable devices, ACM SIGGRAPH 2006 Research posters, July 30-August 03, 2006, Boston, Massachusetts [doi>10.1145/1179622.1179787]
• 8: Kun, L. A., and Lee, K. H. 2005. Enhancing map viewing on symbian series 60 smartphones using camera as 2d input device. Tech. rep., University of Cape Town.
• 9: Xu Liu , David Doermann , Huiping Li, Fast camera motion estimation for hand-held devices and applications, Proceedings of the 4th international conference on Mobile and ubiquitous multimedia, December 08-10, 2005, Christchurch, New Zealand [doi>10.1145/1149488.1149505]
• 10: Lucas, B. D., and Kanade, T. 1981. An iterative image registration technique with an application to stereo vision. In Proceedings of the 7th International Joint Conference on Artificial Intelligence, 674--679.
• 11: Mooser, J., You, S., and Neumann, U. 2007. Real-time object tracking for augmented reality combining graph cuts and optical flow. In Proceedings of the IEEE and ACM International Symposium on Mixed and Augmented Reality, 145--152.
• 12: Pocket Motion. http://www.pocketmotion.com.
• 13: Michael Rohs , Georg Essl, Which one is better?: information navigation techniques for spatially aware handheld displays, Proceedings of the 8th international conference on Multimodal interfaces, November 02-04, 2006, Banff, Alberta, Canada [doi>10.1145/1180995.1181016]
• 14: Michael Rohs , Georg Essl, CaMus²: optical flow and collaboration in camera phone music performance, Proceedings of the 7th international conference on New interfaces for musical expression, June 06-10, 2007, New York, New York [doi>10.1145/1279740.1279770]
• 15: Shi, J., and Tomasi, C. 1994. Good features to track. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'94).
• 16: Jingtao Wang , John Canny, TinyMotion: camera phone based interaction methods, CHI '06 extended abstracts on Human factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1125451.1125526]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis
          Subjects: Tracking

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation)

  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.6 Methodology and Techniques
          Subjects: Interaction techniques

Keywords:
UI techniques, mobile device interaction, tracking