PenLight: combining a mobile projector and a digital pen for dynamic visual overlay

ABSTRACT

Digital pen systems, originally designed to digitize annotations made on physical paper, are evolving to permit a wider variety of applications. Although the type and quality of pen feedback (e.g., haptic, audio, and visual) have a huge impact on advancing the digital pen technology, dynamic visual feedback has yet to be fully investigated. In parallel, miniature projectors are an emerging technology with the potential to enhance visual feedback for small mobile computing devices. In this paper we present the PenLight system, which is a testbed to explore the interaction design space and its accompanying interaction techniques in a digital pen embedded with a spatially-aware miniature projector. Using our prototype, that simulates a miniature projection (via a standard video projector), we visually augment paper documents, giving the user immediate access to additional information and computational tools. We also show how virtual ink can be managed in single and multi-user environments to aid collaboration and data management. User evaluation with professional architects indicated promise of our proposed techniques and their potential utility in the paper-intensive domain of architecture.

References

1. Anoto, Development Guide for Service Enabled by Anoto Functionality. 2002, Anoto.
2. Apitz, G. and F. Guimbretiere. CrossY: A Crossing-Based Drawing Application. Proceedings of UIST'04, pp. 3 -- 12.
3. Arai, T., D. Aust, and S.E. Hudson. PaperLink: a technique for hyperlinking from real paper to electronic content. Proceedings of CHI'97, pp. 327 -- 334.
4. Arvo, J. and K. Novins. Fluid sketches: continuous recognition and morphing of simple hand-drawn shapes. Proceedings of UIST'00, pp. 73--80.
5. Baudisch, P. and R. Rosenholtz, Halo: a technique for visualizing off-screen objects, in Proceedings of the SIGCHI conference on Human factors in computing systems. 2003, ACM: Ft. Lauderdale, Florida, USA.
6. Bier, E.A., M.C. Stone, K. Pier, W. Buxton, and T.D. DeRose. Toolglass and magic lenses: the see-through interface. Proceedings of SigGraph'93, pp. 73 -- 80.
7. Cao, X. and R. Balakrishnan. Interacting with dynamically defined information spaces using a handheld projector and a pen. Proceedings of UIST '06, pp. 225--234.
8. Cao, X., C. Forlines, and R. Balakrishnan, Multi-user interaction using handheld projectors, in Proceedings of UIST'07, ACM: Newport, Rhode Island, USA. p. 43--52.
9. Eastman, C., P. Teicholz, R. Sacks, and K. Liston, BIM Handbook: A Guide to Building Information Modeling for Owners, Managers, Designers, Engineers and Contractors. 2008: Wiley.
10. Erwin, D.E., Further Evidence for Two Components in Visual Persistence. Journal of Experimental Psychology: Human Perception and Performance, 2, 2, 191--209, 1976.
11. Fitzmaurice, G.W., Situated information spaces and spatially aware palmtop computers. Communications of the ACM, 1993. 36(7): p. 39 -- 49.
12. Flagg, M. and J.M. Rehg, Projector-guided painting, in Proceedings of UIST'06, ACM: Montreux, Switzerland. p. 235--244.
13. Flypentop, Flypentop Computer http://www.flypentop.com/.
14. Forlines, C., R. Balakrishnan, P. Beardsley, J.v. Baar, and R. Raskar, Zoom-and-pick: facilitating visual zooming and precision pointing with interactive handheld projectors, in Proceedings of the UIST'05, ACM: Seattle, WA, USA. p. 73--82.
15. Grossman, T., K. Hinckley, P. Baudisch, M. Agrawala, and R. Balakrishnan, Hover widgets: using the tracking state to extend the capabilities of pen-operated devices, in Proceedings of the SIGCHI '06, ACM: Montreal, Quebec, Canada. p. 861--870.
16. Guimbretiere, F. Paper Augmented Digital Documents. Proceedings of UIST'03, pp. 51 -- 60.
17. Heikkila, J. and O. Silven. A Four-step Camera Calibration Procedure with Implicit Image Correction. Proceedings of CVPR'97, pp.
18. Heiner, J.M., S.E. Hudson, and K. Tanaka. Linking and messaging from real paper in the Paper PDA. Proceedings of UIST'99, pp. 179 -- 186.
19. Hinckley, K., R. Pausch, J.C. Goble, and N.F. Kassell, A survey of design issues in spatial input, in Proceedings of the 7th annual ACM symposium on User interface software and technology. 1994, ACM: Marina del Rey, California, United States.
20. Holman, D., R. Vertegaal, M. Altosaar, N. Troje, and D. Johns, Paper windows: interaction techniques for digital paper, in Proceedings of the SIGCHI conference on Human factors in computing systems. 2005, ACM: Portland, Oregon, USA.
21. Keisuke, T., K. Itaru, and O. Yuichi, A nested marker for augmented reality, in ACM SIGGRAPH 2006 Sketches. 2006, ACM: Boston, Massachusetts.
22. Lee, C.J., H.P. Dietz, D. Leigh, S.W. Yerazunis, and E.S. Hudson. Haptic pen: a tactile feedback stylus for touch screens. Proceedings of UIST'04, pp. 291--294.
23. Liao, C., F. Guimbretière, and C.E. Loeckenhoff. Pen-top feedback for paper-based interfaces. Proceedings of UIST '06, pp. 201--210.
24. Liao, C., F. Guimbretière, K. Hinckley, and J. Hollan, Papiercraft: A gesture-based command system for interactive paper. ACM Trans. Comput.-Hum. Interact., 2008. 14(4): p. 1--27.
25. LiveScribe, http://www.livescribe.com/
26. Light Blue Optics http://www.lightblueoptics.com/
27. Mackay, W.E., D.S. Pagani, L. Faber, B. Inwood, P. Launiainen, L. Brenta, and V. Pouzol. Ariel: augmenting paper engineering drawings. Proceedings of CHI'95, pp. 421 -- 422.
28. Mackay, W.E., G. Pothier, C. Letondal, K. Bøegh, and H.E. Sørensen. The missing link: augmenting biology laboratory notebooks. Proceedings of UIST'02, pp. 41 -- 50.
29. Norman, D., Psychology of everyday things. 1988: Basic Books.
30. Perlin, K. and J. Meyer, Nested user interface components, in Proceedings of the 12th annual ACM symposium on User interface software and technology. 1999, ACM: Asheville, North Carolina, United States. p. 11--18.
31. Phelps, T.A. and R. Wilensky. Multivalent Annotations. Proceedings of ECDL'97, pp. 287 -- 303.
32. Signer, B. and M.C. Norrie. PaperPoint: a paper-based presentation and interactive paper prototyping tool. Proceedings of TEI'07, pp. 57--64.
33. Subramanian, S., D. Aliakseyeu, and A. Lucero, Multi-layer interaction for digital tables, in Proceedings of UIST06, ACM: Montreux, Switzerland. p. 269--272.
34. Tsang, M., G. Fitzmaurice, G. Kurtenbach, A. Khan, and B. Buxton. Boom chameleon: simultaneous capture of 3D viewpoint, voice and gesture annotations on a spatially-aware display. Proceedings of UIST '02, pp. 111 -- 120.
35. MicroVision http://www.microvision.com/
36. Wellner, P., Interacting with paper on the DigitalDesk. Communications of the ACM, 1993. 36(7): p. 87 -- 96.
37. Yee, K. Peephole displays: pen interaction on spatially aware handheld computers. Proceedings of SIGCHI '03, pp. 1--8.
38. Yeh, R.B., C. Liao, S.R. Klemmer, F. Guimbretière, B. Lee, B. Kakaradov, J. Stamberger, and A. Paepcke. ButterflyNet: A Mobile Capture and Access System for Field Biology Research. Proceedings of CHI'06, pp.
39. Zhai, S., S. Conversy, M. Beaudouin-Lafon, and Y. Guiard, Human on-line response to target expansion, in Proceedings of the SIGCHI '03, ACM: Ft. Lauderdale, Florida, USA.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction devices

      • Touch screens