Charting past, present, and future research in ubiquitous computing

Abstract

The proliferation of computing into the physical world promises more than the ubiquitous availability of computing infrastructure; it suggest new paradigms of interaction inspired by constant access to information and computational capabilities. For the past decade, application-driven research on abiquitous computing (ubicomp) has pushed three interaction themes:natural interfaces, context-aware applications,andautomated capture and access. To chart a course for future research in ubiquitous computing, we review the accomplishments of these efforts and point to remaining research challenges. Research in ubiquitious computing implicitly requires addressing some notion of scale, whether in the number and type of devices, the physical space of distributed computing, or the number of people using a system. We posit a new area of applications research, everyday computing, focussed on scaling interaction with respect to time. Just as pushing the availiability of computing away from the traditional desktop fundamentally changes the relationship between humans and computers, providing continuous interaction moves computing from a localized tool to a constant companion. Designing for continous interaction requires addressing interruption and reumption of intreaction, representing passages of time and providing associative storage models. Inherent in all of these interaction themes are difficult issues in the social implications of ubiquitous computing and the challenges of evaluating> ubiquitious computing research. Although cumulative experience points to lessons in privacy, security, visibility, and control, there are no simple guidelines for steering research efforts. Akin to any efforts involving new technologies, evaluation strategies form a spectrum from technology feasibility efforts to long-term use studies—but a user-centric perspective is always possible and necessary

References

1. ABOWD, G. D. 1999. Classroom 2000: An experiment with the instruction of a living educational environment. IBM Syst. J. 38, 4, 508-530.
2. ABOWD, G. D., ATKESON, C. G., HONG, J., LONG, S., KOOPER, R., AND PINKERTON, M. 1997. Cyberguide: a mobile context-aware tour guide. Wireless Networks 3, 5, 421-433.
3. ABOWD, G. D., ATKESON, C. G., BROTHERTON, J., ENQVIST, T., GULLEY, P., AND LEMON, J. 1998a. Investigating the capture, integration and access problem of ubiquitous computing in an educational setting. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI '98, Los Angeles, CA, Apr. 18-23), M. E. Atwood, C.-M. Karat, A. Lund, J. Coutaz, and J. Karat, Eds. ACM Press/Addison-Wesley Publ. Co., New York, NY, 440-447.
4. ABOWD, G. D., BROTHERTON, J., AND BHALODIA, J. 1998b. Automated capture, integration, and visualization of multiple media streams. In Proceedings of the 1998 IEEE Conference on Multimedia and Computing Systems (San Antonio, TX), IEEE Computer Society Press, Los Alamitos, CA, 54-63.
5. BABER, C. AND HONE, K. S. 1993. Modeling error recovery and repair in automatic speech recognition. Int. J. Man-Mach. Stud. 39, 3 (Sept. 1993), 495-515.
6. BACHER, C. H. R. AND OTTMANN, T. H. 1996. Tools and services for authoring on the fly. In Proceedings of the Conference on Educational Multimedia and Hypermedia (ED-MEDIA '96, Boston, MA, June 7-12),
7. BEDERSON, B. B. 1995. Audio augmented reality: A prototype automated tour guide. In Conference Companion for the ACM Conference on Human Factors in Computing Systems (CHI '95, Denver, CO, May 7-11), J. Miller, I. Katz, R. Mack, and L. Marks, Eds. ACM Press, New York, NY, 210-211.
8. BROTHERTON, J. A., ABOWD, G., AND TRUONG, K. N. 1999. Supporting capture and access interfaces for informal and opportunistic meetings. Tech. Rep. GIT-GVU-99-06. Graphics, Visualization and Usability Center, Georgia Institute of Technology, Atlanta, GA.
9. BUSH, V. 1945. As we may think. Atlantic Mon. 176, I (July), 101-108. Also at h ttp ://www. th e a t l an ti c. co m/un bo un d/fi ash b ks/co m p u te r/b u shf . h tm.
10. CARD, S., MACKINLAY, J., AND SHNEIDERMAN, B. 1999. Readings in Information Visualization: Using Vision to Think. Morgan Kaufmann Publishers Inc., San Francisco, CA.
11. CARROLL, J. M. AND ROSSON, M. B. 1991. Deliberated evolution: Stalking the view matcher in design space articles. Hum. Comput. Interact. 6, 3-4, 281-318.
12. CHEVERST, K., MITCHELL, K., AND DAVIES, N. 1998. Design of an object model for a context sensitive tourist guide. In Proceedings of the Conference on Interactive Applications of Mobile Computing (IMB '98, Rostock, Germany, Nov.),
13. CHIU, P. AND WILCOX, L. 1998. A dynamic grouping technique for ink and audio notes. In Proceedings of the 11th Annual ACM Symposium on User Interface Software and Technology (UIST '98, San Francisco, CA, Nov. 1-4), E. Mynatt and R. J. K. Jacob, Eds. ACM Press, New York, NY, 195-202.
14. DAVIS, R. C., LANDAY, J. A., CHEN, V., HUANG, J., LEE, R. B., LI, F. C., LIN, J., MORREY, C. B., III, SCHLEIMER, B., PRICE, M. N., AND SCHILIT, B. N. 1999. NotePals: Lightweight note sharing by the group, for the group. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI '99, Pittsburgh, PA, May), ACM Press, New York, NY, 338 -345.
15. EDWARDS, W. K. AND MYNATT, E. D. 1997. Timewarp: techniques for autonomous collaboration. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI '97, Atlanta, GA, Mar. 22-27), S. Pemberton, Ed. ACM Press, New York, NY, 218-225.
16. ELROD, S., BRUCE, R., GOLD, R., GOLDBERG, D., HALASZ, F., JANSSEN, W., LEE, D., MCCALL, K., PEDERSEN, E., PIER, K., TANG, J., AND WELCH, B. 1992. Liveboard: a large interactive display supporting group meetings, presentations, and remote collaboration. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI '92, Monterey, CA, May 3-7), P. Bauersfeld, J. Bennett, and G. Lynch, Eds. ACM Press, New York, NY, 599-607.
17. FEINER, S., MACINTYRE, B., AND SELIGMANN, D. 1993. Knowledge-based augmented reality. Commun. ACM 36, 7 (July), 53-62.
18. FERTIG, S., FREEMAN, E., AND GELERNTER, D. 1996. Lifestreams: an alternative to the desktop metaphor. In Proceedings of the CHI '96 Conference Companion on Human Factors in Computing Systems: Common Ground (CHI '96, Vancouver, British Columbia, Canada, Apr. 13-18), M. J. Tauber, Ed. ACM Press, New York, NY, 410-411.
19. FITZMAURICE, G. W., ISHII, H., AND BUXTON, W. A. S. 1995. Bricks: Laying the foundations for graspable user interfaces. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI '95, Denver, CO, May 7-11), I. R. Katz, R. Mack, L. Marks, M. B. Rosson, and J. Nielsen, Eds. ACM Press/Addison-Wesley Publ. Co., New York, NY, 442-449.
20. FITZMAURICE, G. W., ZHAI, S., AND CHIGNELL, M. H. 1993. Virtual reality for palmtop computers. ACM Trans. Inf. Syst. 11, 3 (July 1993), 197-218.
21. FRANKISH, C., JONES, D., AND HAPESHI, K. 1992. Decline in accuracy of automatic speech recognition as a function of time on task: Fatigue or voice drift?. Int. J. Man-Mach. Stud. 36, 6 (June 1992), 797-816.
22. FRANKISH, C., HULL, R., AND MORGAN, P. 1995. Recognition accuracy and user acceptance of pen interfaces. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI '95, Denver, CO, May 7-11), I. R. Katz, R. Mack, L. Marks, M. B. Rosson, and J. Nielsen, Eds. ACM Press/Addison-Wesley Publ. Co., New York, NY, 503-510.
23. GOLDBERG, D. AND RICHARDSON, C. 1993. Touch-typing with a stylus. In Proceedings of the ACM Conference on Human Factors in Computing (INTERCHI '93, Amsterdam, The Netherlands, Apr. 24-29), S. Ashlund, A. Henderson, E. Hollnagel, K. Mullet, and T. White, Eds. ACM Press, New York, NY, 80-87.
24. GRUDIN, J. AND PALEN, L. 1997. Emerging groupware successes in major corporations: Studies of adoption and adaptation. In Worldwide Computing and Its Applications, T. Masuda, Y. Masunaga, and M. Tsukamoto, Eds. Springer Lecture Notes in Computer Science, vol. 1274. Springer-Verlag, New York, 142-153.
25. HARRISON, B. L., FISHKIN, K. P., GUJAR, A., MOCHON, C., AND WANT, R. 1998. Squeeze me, hold me, tilt me! An exploration of manipulative user interfaces. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI '98, Los Angeles, CA, Apr. 18-23), M. E. Atwood, C.-M. Karat, A. Lurid, J. Coutaz, and J. Karat, Eds. ACM Press/Addison-Wesley Publ. Co., New York, NY, 17-24.
26. HENDERSON, D. A. AND CARD, S. 1986. Rooms: The use of multiple virtual workspaces to reduce space contention in a window-based graphical user interface. ACM Trans. Graph. 5, 3 (July 1986), 211-243.
27. HINDUS, D. AND SCHMANDT, C. 1992. Ubiquitous audio: capturing spontaneous collaboration. In Proceedings of the ACM Conference on Computer-Supported Cooperative Work (CSCW '92, Toronto, Canada, Oct. 31-Nov. 4), M. Mantel and R. Baecker, Eds. ACM Press, New York, NY, 210-217.
28. ISHH, H. AND ULLMER, B. 1997. Tangible bits: Towards seamless interfaces between people, bits and atoms. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI '97, Atlanta, GA, Mar. 22-27), S. Pemberton, Ed. ACM Press, New York, NY, 234-241.
29. JEBARA, T., EYSTER, C., WEAVER, J., STARNER, T., AND PENTLAND, A. 1997. Stochasticks: Augmenting the billiards experience with probabilistic vision and wearable computers. In Proceedings of the IEEE International Symposium on Wearable Computing (Oct.), IEEE Computer Society, New York, NY.
30. LAMMING, M. AND FLYNN, M. 1994. "Forget-me-not" intimate computing in support of human memory. Tech. Rep. EPC-94-103. Rank Xerox, EuroPARC, Cambridge, UK.
31. MACINTYRE, B. AND FEINER, S. 1996. Future multimedia user interfaces. Multimedia Syst. 4, 5, 250-268.
32. MACINTYRE, B. AND MYNATT, E. n. 1998. Augmenting intelligent environments: Augmented reality as an interface to intelligent environments. In Intelligent Environments Symposium (Stanford, CA, Mar. 23-25), AAAI 1998 Spring Symposium Series AAAI Press, Menlo Park, CA.
33. MANKOFF, J., HUDSON, S., AND ABOWD, G. 2000. Providing integrated toolkit-level support for ambiguity in recognition-based interfaces. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI' 00, The Hauge, The Netherlands, Apr.), ACM Press, New York, NY, 368-375.
34. MARX, M. AND SCHMANDT, C. 1994. Putting people first: Specifying proper names in speech interfaces. In Proceedings of the 7th Annual ACM Symposium on User Interface Software and Technology (UIST '94, Marina del Rey, CA, Nov. 2-4), P. Szekely, Ed. ACM Press, New York, NY, 29-37.
35. MINNEMAN, S., HARRISON, S., JANSSEN, B., KURTENBACH, G., MORAN, T., SMITH, I., AND VAN MELLE, B. 1995. A confederation of tools for capturing and accessing collaborative activity. In Proceedings of the 3rd International Conference on Multimedia (Multimedia '95, San Francisco, CA, Nov. 5-9), P. Zellweger, Ed. ACM Press, New York, NY, 523-534.
36. MORAN, T. P., CHIU, P., HARRISON, S., KURTENBACH, G., MINNEMAN, S., AND VAN MELLE, W. 1996. Evolutionary engagement in an ongoing collaborative work process: A case study. In Proceedings of the 1996 ACM Conference on Computer-Spported Cooperative Work (CSCW '96, Boston, MA, Nov. 16-20), G. Olson, J. Olson, and M. S. Ackerman, Eds. ACM Press, New York, NY, 150-159.
37. MORAN, T. P., CHIU, P., AND VAN MELLE, W. 1997a. Pen-based interaction techniques for organizing material on an electronic whiteboard. In Proceedings of the lOth Annual ACM Symposium on User Interface Software and Technology (UIST '97, Banff, Alberta, Canada, Oct. 14-17), G. Robertson and C. Schmandt, Eds. ACM Press, New York, NY, 45-54.
38. MORAN, T. P., PALEN, L., HARRISON, S., CHIU, P., KIMBER, D., MINNEMAN, S., VAN MELLE, W., AND ZELLWEGER, P. 1997b. "I'll get that off the audio": A case study of salvaging multimedia meeting records. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI '97, Atlanta, GA, Mar. 22-27), S. Pemberton, Ed. ACM Press, New York, NY, 202-209.
39. MUKHOPADHYA, S. AND SMITH, B. 1999. Passive capture and structuring of lectures. In Proceedings of the ACM Conference on Multimedia (Multimedia '99, Orlando, FL), ACM Press, New York, NY, 477-487.
40. MYNATT, E. D. 1999. The writing on the wall. In Proceedings of the 7th IFIP Conference on Human-Computer Interaction (INTERACT '99), IFIP, Laxenburg, Austria.
41. MYNATT, E. D., BACK, M., WANT, R., BAER, M., AND ELLIS, J. B. 1998. Designing audio aura. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI '98, Los Angeles, CA, Apr. 18-23), M. E. Atwood, C.-M. Karat, A. Lund, J. Coutaz, and J. Karat, Eds. ACM Press/Addison-Wesley Publ. Co., New York, NY, 566-573.
42. MYNATT, E. D., IGARASHI, T., EDWARD, W. K., AND LAMARCA, A. 1999. Flatland: New dimensions in office whiteboards. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI '99, Pittsburgh, PA, May), ACM Press, New York, NY.
43. NORMAN, D.A. 1998. The Invisible Computer. MIT Press, Cambridge, MA.
44. OPPERMANN, R. AND SPECHT, M. 1998. Adaptive support for a mobile museum guide. In Proceedings of the Conference on Interactive Applications of Mobile Computing (IMB '98, Rostock, Germany, Nov.), Available at http://www.egd.igd.fhg.de/-imc98/proceedings.html.
45. PICARD, R.W. 1997. Affective Computing. MIT Press, Cambridge, MA.
46. REKIMOTO, J. 1999. Time-machine computing: A time-centric approach for the information environment. In Proceedings of the ACM Symposium on User Interface Software Technology (UIST '99, Asheville, NC, Nov.), ACM, New York, NY, 45-54.
47. REKIMOTO, J. AND NAGAO, K. 1995. The world through the computer: computer augmented interaction with real world environments. In Proceedings of the 8th ACM Symposium on User Interface and Software Technology (UIST '95, Pittsburgh, PA, Nov 14-17), G. Robertson, Ed. ACM Press, New York, NY, 29-36.
48. RHODES, B.J. 1997. The wearable remembrance agent: A system for augmented memory. Pers. Tech. J. 1, 4, 218-224.
49. RHODES, B. J. AND STARNER, T. E. 1996. Remembrance agent: A continuously running automated information retrieval system. In Proceedings of Practical Applications of Intelligent Agents and Multi-Agent Technology (PAAM),
50. RICHTER, H., SCHUCHHARD, P., AND ABOWD, G. D. 1999. Automated capture and retrieval of architectural rationale. In Online Proceedings of the 1st Working IFIP Conference on Software Architecture (San Antonio, TX), IFIP, Laxenburg, Austria. Full paper appears as Georgia Tech GVU Center Technical Report GIT-GVU-98-37.
51. SALBER, D., DEY, A. K., AND ABOWD, G. D. 1999. The context toolkit: Aiding the development of context-enabled applications. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI '99, Pittsburgh, PA, May), ACM Press, New York, NY, 434-441.
52. SCHMANDT, C. AND ARONS, B. 1985. PhoneSlave: A graphical telecommunications interface. Proc. Soc. Inf. Display 26, 1, 79-82.
53. SCHOMAKER, L. R. B. 1994. User-interface aspects in recognizing connected-cursive handwriting. In Proceedings of the IEE Colloquium on Handwriting and Pen-Based Input, The Institution of Electrical Engineers, London, UK.
54. SHIPMAN, F. M., MARSHALL, C. C., AND MORAN, T. P. 1995. Finding and using implicit structure in human-organized spatial layouts of information. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI '95, Denver, CO, May 7-11), I. R. Katz, R. Mack, L. Marks, M. B. Rosson, and J. Nielsen, Eds. ACM Press/Addison-Wesley Publ. Co., New York, NY, 346-353.
55. STARNER, T., MANN, S., RHODES, B., LEVINE, J., HEALEY, J., KIRSCH, D., PICARD, R., AND PENTLAND, A. 1997. Augmented reality throuh wearable computing. Presence: Teleoper. Virtual Environ. 6, 4.
56. STIFELMAN, L.J. 1996. Augmenting real-world objects: a paper-based audio notebook. In Proceedings of the CHI '96 Conference Companion on Human Factors in Computing Systems: Common Ground (CHI '96, Vancouver, British Columbia, Canada, Apr. 13-18), M. J. Tauber, Ed. ACM Press, New York, NY, 199-200.
57. STIFELMAN, L.J. 1997. The audio notebook: Paper and pen interaction with structured speech. Ph.D. Dissertation. MIT Media Lab, Cambridge, MA.
58. STREITZ, N. A., GEII}LER, J., HOLMER, T., KONOMI, S. I., M LLER-TOMFELDE, C., REISCHL, W., REXROTH, P., SEITZ, P., AND STEINMETZ, R. 1999. i-Land: An interactive landscape for creativity and innovation. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI '99, Pittsburgh, PA, May), ACM Press, New York, NY, 120-127.
59. TRUONG, K. N., ABOWD, G. D., AND BROTHERTON, J.A. 1999. Personalizing the capture of public experiences. In Proceedings of the ACM Symposium on User Interface Software Technology (UIST '99, Asheville, NC, Nov.), ACM, New York, NY, 121-130.
60. TURK, M., Ed. 1997. Proceedings of the Perceptual User Interfaces Workshop. (PUI '97, Banff, Alberta, Canada, Oct. 20-21). http://research.microsoft.com/PUIWorkshop97/.
61. TURK, M., Ed. 1998. Proceedings of the Perceptual User Interfaces Workshop. (PIU '98, San Francisco, CA, Nov. 5-6). http://research.microsoft.com/PUIWorkshop/PUI98.htm.
62. VAN BUSKmK, R. AND LALOMIA, M. 1995. The just noticeable difference of speech recognition accuracy. In Conference Companion for the ACM Conference on Human Factors in Computing Systems (CHI '95, Denver, CO, May 7-11), J. Miller, I. Katz, R. Mack, and L. Marks, Eds. ACM Press, New York, NY, 95.
63. WANT, R., HOPPER, A., FALC O, V., AND GIBBONS, J. 1992. The active badge location system. ACM Trans. Inf. Syst. 10, 1 (Jan. 1992), 91-102.
64. WANT, R., SCHILIT, B., ADAMS, N., GOLD, R., PETERSEN, K., ELLIS, J., GOLDBERG, D., AND WEISER, M. 1995. The PARCTab ubiquitous computing experiment. Tech. Rep. CSL-95-1. Xerox PARC, Palo Alto, CA.
65. WEHER, K. AND POON, A. 1994. Marquee: A tool for real-time video logging. In Proceedings of the ACM Conference on Human Factors in Computing Systems: "Celebrating Interdependence" (CHI '94, Boston, MA, Apr. 24-28), ACM Press, New York, NY, 58-64.
66. WEISER, M. 1991. The computer for the 21st century. Sci. Am. 265, 3 (Sept.), 94-104.
67. WEISER, M. 1993. Some computer science issues in ubiquitous computing. Commun. ACM 36, 7 (July), 75-84.
68. WELLNER, P. 1993. Interacting with paper on the DigitalDesk. Commun. ACM 36, 7 (July), 87-96.
69. WHITE, S., GUPTA, A., GRUDIN, J., KIMBERLY, G., AND SANOCKI, E. 1998. A software system for education at a distance: Case study results. Microsoft Research Tech. Rep. MSR-TR-98-61. Microsoft Corp., Redmond, WA. Revised in April 1999.
70. WHITTAKER, S., HYLAND, P., AND WILEY, M. 1994. FILOCHAT: handwritten notes provide access to recorded conversations. In Proceedings of the ACM Conference on Human Factors in Computing Systems: "Celebrating Interdependence" (CHI '94, Boston, MA, Apr. 24-28), ACM Press, New York, NY, 271-277.
71. WILCOX, L. D., SCHILIT, B. N., AND SAWHNEY, N. 1997. Dynomite: a dynamically organized ink and audio notebook. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI '97, Atlanta, GA, Mar. 22-27), S. Pemberton, Ed. ACM Press, New York, NY, 186-193.
72. WOLF, C. G., RHYNE, J. R., AND BRIGGS, L. K. 1992. Communication and information retrieval with a pen-based meeting support tool. In Proceedings of the ACM Conference on Computer- Supported Cooperative Work (CSCW '92, Toronto, Canada, Oct. 31-Nov. 4), M. Mantel and R. Baecker, Eds. ACM Press, New York, NY, 322-329.
73. ZAJICEK, M. AND HEWITT, J. 1990. An investigation into the use of error recovery dialogues in a user interface management system for speech recognition. In Proceedings of the IFIP 1990 Conference on Human-Computer Interaction (INTERACT '90), IFIP, Laxenburg, Austria, 755-760.

Index Terms

• Applied computing

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI design and evaluation methods

• Social and professional topics