Charting past, present, and future research in ubiquitous computing

ABSTRACT

The proliferation of computing into the physical world promises more than the ubiquitous availability of computing infrastructure; it suggest new paradigms of interaction inspired by constant access to information and computational capabilities. For the past decade, application-driven research on abiquitous computing (ubicomp) has pushed three interaction themes:natural interfaces, context-aware applications,andautomated capture and access. To chart a course for future research in ubiquitous computing, we review the accomplishments of these efforts and point to remaining research challenges. Research in ubiquitious computing implicitly requires addressing some notion of scale, whether in the number and type of devices, the physical space of distributed computing, or the number of people using a system. We posit a new area of applications research, everyday computing, focussed on scaling interaction with respect to time. Just as pushing the availiability of computing away from the traditional desktop fundamentally changes the relationship between humans and computers, providing continuous interaction moves computing from a localized tool to a constant companion. Designing for continous interaction requires addressing interruption and reumption of intreaction, representing passages of time and providing associative storage models. Inherent in all of these interaction themes are difficult issues in the social implications of ubiquitous computing and the challenges of evaluating> ubiquitious computing research. Although cumulative experience points to lessons in privacy, security, visibility, and control, there are no simple guidelines for steering research efforts. Akin to any efforts involving new technologies, evaluation strategies form a spectrum from technology feasibility efforts to long-term use studies—but a user-centric perspective is always possible and necessary

REFERENCES

• 1: G. D. Abowd, Classroom 2000: an experiment with the instrumentation of a living educational environment, IBM Systems Journal, v.38 n.4, p.508-530, Dec. 1999
• 2: Gregory D. Abowd , Christopher G. Atkeson , Jason Hong , Sue Long , Rob Kooper , Mike Pinkerton, Cyberguide: a mobile context-aware tour guide, Wireless Networks, v.3 n.5, p.421-433, Oct. 1997 [doi>10.1023/A:1019194325861]
• 3: Gregory D. Abowd , Christopher G. Atkeson , Jason Brotherton , Tommy Enqvist , Paul Gulley , Johan LeMon, Investigating the capture, integration and access problem of ubiquitous computing in an educational setting, Proceedings of the SIGCHI conference on Human factors in computing systems, p.440-447, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/274644.274704]
• 4: Jason A. Brotherton , Janak R. Bhalodia , Gregory D. Abowd, Automated Capture, Integration, and Visualization of Multiple Media Streams, Proceedings of the IEEE International Conference on Multimedia Computing and Systems, p.54, June 28-July 01, 1998 [doi>10.1109/MMCS.1998.693625]
• 5: C. Baber , K. S. Hone, Modeling error recovery and repair in automatic speech recognition, International Journal of Man-Machine Studies, v.39 n.3, p.495-515, Sept. 1993 [doi>10.1006/imms.1993.1070]
• 6: BACHER, C. H. R. AND OTTMANN, T. H. 1996. Tools and services for authoring on the fly. In Proceedings of the Conference on Educational Multimedia and Hypermedia (ED-MEDIA '96, Boston, MA, June 7-12),
• 7: Benjamin B. Bederson, Audio augmented reality: a prototype automated tour guide, Conference companion on Human factors in computing systems, p.210-211, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223355.223526]
• 8: BROTHERTON, J. A., ABOWD, G., AND TRUONG, K. N. 1999. Supporting capture and access interfaces for informal and opportunistic meetings. Tech. Rep. GIT-GVU-99-06. Graphics, Visualization and Usability Center, Georgia Institute of Technology, Atlanta, GA.
• 9: BUSH, V. 1945. As we may think. Atlantic Mon. 176, I (July), 101-108. Also at h ttp ://www. th e a t l an ti c. co m/un bo un d/fi ash b ks/co m p u te r/b u shf . h tm.
• 10: Stuart K. Card , Jock D. Mackinlay , Ben Shneiderman, Readings in information visualization: using vision to think, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1999
• 11: CARROLL, J. M. AND ROSSON, M. B. 1991. Deliberated evolution: Stalking the view matcher in design space articles. Hum. Comput. Interact. 6, 3-4, 281-318.
• 12: CHEVERST, K., MITCHELL, K., AND DAVIES, N. 1998. Design of an object model for a context sensitive tourist guide. In Proceedings of the Conference on Interactive Applications of Mobile Computing (IMB '98, Rostock, Germany, Nov.),
• 13: Patrick Chiu , Lynn Wilcox, A dynamic grouping technique for ink and audio notes, Proceedings of the 11th annual ACM symposium on User interface software and technology, p.195-202, November 01-04, 1998, San Francisco, California, United States [doi>10.1145/288392.288605]
• 14: Richard C. Davis , James A. Landay , Victor Chen , Jonathan Huang , Rebecca B. Lee , Frances C. Li , James Lin , Charles B. Morrey, III , Ben Schleimer , Morgan N. Price , Bill N. Schilit, NotePals: lightweight note sharing by the group, for the group, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.338-345, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303107]
• 15: W. Keith Edwards , Elizabeth D. Mynatt, Timewarp: techniques for autonomous collaboration, Proceedings of the SIGCHI conference on Human factors in computing systems, p.218-225, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258710]
• 16: Scott Elrod , Richard Bruce , Rich Gold , David Goldberg , Frank Halasz , William Janssen , David Lee , Kim McCall , Elin Pedersen , Ken Pier , John Tang , Brent Welch, Liveboard: a large interactive display supporting group meetings, presentations, and remote collaboration, Proceedings of the SIGCHI conference on Human factors in computing systems, p.599-607, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.143052]
• 17: Steven Feiner , Blair Macintyre , Dorée Seligmann, Knowledge-based augmented reality, Communications of the ACM, v.36 n.7, p.53-62, July 1993 [doi>10.1145/159544.159587]
• 18: Scott Fertig , Eric Freeman , David Gelernter, Lifestreams: an alternative to the desktop metaphor, Conference companion on Human factors in computing systems: common ground, p.410-411, April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/257089.257404]
• 19: George W. Fitzmaurice , Hiroshi Ishii , William A. S. Buxton, Bricks: laying the foundations for graspable user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems, p.442-449, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223964]
• 20: George W. Fitzmaurice , Shumin Zhai , Mark H. Chignell, Virtual reality for palmtop computers, ACM Transactions on Information Systems (TOIS), v.11 n.3, p.197-218, July 1993 [doi>10.1145/159161.159160]
• 21: Clive Frankish , Dylan Jones , Kevin Hapeshi, Decline in accuracy of automatic speech recognition as a function of time on task: fatigue or voice drift?, International Journal of Man-Machine Studies, v.36 n.6, p.797-816, June 1992 [doi>10.1016/0020-7373(92)90073-T]
• 22: Clive Frankish , Richard Hull , Pam Morgan, Recognition accuracy and user acceptance of pen interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems, p.503-510, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223972]
• 23: David Goldberg , Cate Richardson, Touch-typing with a stylus, Proceedings of the SIGCHI conference on Human factors in computing systems, p.80-87, April 24-29, 1993, Amsterdam, The Netherlands [doi>10.1145/169059.169093]
• 24: Jonathan Grudin , Leysia Palen, Emerging Groupware Successes in Major Corporations: Studies of Adoption and Adaptation, Proceedings of the International Conference on Worldwide Computing and Its Applications, p.142-153, March 10-11, 1997
• 25: Beverly L. Harrison , Kenneth P. Fishkin , Anuj Gujar , Carlos Mochon , Roy Want, Squeeze me, hold me, tilt me! An exploration of manipulative user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems, p.17-24, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/274644.274647]
• 26: D. Austin Henderson, Jr. , Stuart Card, Rooms: the use of multiple virtual workspaces to reduce space contention in a window-based graphical user interface, ACM Transactions on Graphics (TOG), v.5 n.3, p.211-243, July 1986 [doi>10.1145/24054.24056]
• 27: Debby Hindus , Chris Schmandt, Ubiquitous audio: capturing spontaneous collaboration, Proceedings of the 1992 ACM conference on Computer-supported cooperative work, p.210-217, November 01-04, 1992, Toronto, Ontario, Canada [doi>10.1145/143457.143481]
• 28: Hiroshi Ishii , Brygg Ullmer, Tangible bits: towards seamless interfaces between people, bits and atoms, Proceedings of the SIGCHI conference on Human factors in computing systems, p.234-241, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258715]
• 29: Tony Jebara , Cyrus Eyster , Josh Weaver , Thad Starner , Alex Pentland, Stochasticks: Augmenting the Billiards Experience with Probabilistic Vision and Wearable Computers, Proceedings of the 1st IEEE International Symposium on Wearable Computers, p.138, October 13-14, 1997
• 30: LAMMING, M. AND FLYNN, M. 1994. "Forget-me-not" intimate computing in support of human memory. Tech. Rep. EPC-94-103. Rank Xerox, EuroPARC, Cambridge, UK.
• 31: Blair MacIntyre , Steven Feiner, Future multimedia user interfaces, Multimedia Systems, v.4 n.5, p.250-268, Oct. 1996 [doi>10.1007/s005300050027]
• 32: MACINTYRE, B. AND MYNATT, E. n. 1998. Augmenting intelligent environments: Augmented reality as an interface to intelligent environments. In Intelligent Environments Symposium (Stanford, CA, Mar. 23-25), AAAI 1998 Spring Symposium Series AAAI Press, Menlo Park, CA.
• 33: Jennifer Mankoff , Scott E. Hudson , Gregory D. Abowd, Providing integrated toolkit-level support for ambiguity in recognition-based interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems, p.368-375, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332459]
• 34: Matt Marx , Chris Schmandt, Putting people first: specifying proper names in speech interfaces, Proceedings of the 7th annual ACM symposium on User interface software and technology, p.29-37, November 02-04, 1994, Marina del Rey, California, United States [doi>10.1145/192426.192439]
• 35: Scott Minneman , Steve Harrison , Bill Janssen , Gordon Kurtenbach , Thomas Moran , Ian Smith , Bill van Melle, A confederation of tools for capturing and accessing collaborative activity, Proceedings of the third ACM international conference on Multimedia, p.523-534, November 05-09, 1995, San Francisco, California, United States [doi>10.1145/217279.215316]
• 36: Thomas P. Moran , Patrick Chiu , Steve Harrison , Gordon Kurtenbach , Scott Minneman , William van Melle, Evolutionary engagement in an ongoing collaborative work process: a case study, Proceedings of the 1996 ACM conference on Computer supported cooperative work, p.150-159, November 16-20, 1996, Boston, Massachusetts, United States [doi>10.1145/240080.240243]
• 37: Thomas P. Moran , Patrick Chiu , William van Melle, Pen-based interaction techniques for organizing material on an electronic whiteboard, Proceedings of the 10th annual ACM symposium on User interface software and technology, p.45-54, October 14-17, 1997, Banff, Alberta, Canada [doi>10.1145/263407.263508]
• 38: Thomas P. Moran , Leysia Palen , Steve Harrison , Patrick Chiu , Don Kimber , Scott Minneman , William van Melle , Polle Zellweger, “I'll get that off the audio”: a case study of salvaging multimedia meeting records, Proceedings of the SIGCHI conference on Human factors in computing systems, p.202-209, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258704]
• 39: Sugata Mukhopadhyay , Brian Smith, Passive capture and structuring of lectures, Proceedings of the seventh ACM international conference on Multimedia (Part 1), p.477-487, October 30-November 05, 1999, Orlando, Florida, United States [doi>10.1145/319463.319690]
• 40: MYNATT, E. D. 1999. The writing on the wall. In Proceedings of the 7th IFIP Conference on Human-Computer Interaction (INTERACT '99), IFIP, Laxenburg, Austria.
• 41: Elizabeth D. Mynatt , Maribeth Back , Roy Want , Michael Baer , Jason B. Ellis, Designing audio aura, Proceedings of the SIGCHI conference on Human factors in computing systems, p.566-573, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/274644.274720]
• 42: Elizabeth D. Mynatt , Takeo Igarashi , W. Keith Edwards , Anthony LaMarca, Flatland: new dimensions in office whiteboards, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.346-353, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303108]
• 43: Donald A. Norman, The invisible computer, MIT Press, Cambridge, MA, 1998
• 44: OPPERMANN, R. AND SPECHT, M. 1998. Adaptive support for a mobile museum guide. In Proceedings of the Conference on Interactive Applications of Mobile Computing (IMB '98, Rostock, Germany, Nov.), Available at http://www.egd.igd.fhg.de/-imc98/proceedings.html.
• 45: Rosalind W. Picard, Affective computing, MIT Press, Cambridge, MA, 1997
• 46: Jun Rekimoto, Time-machine computing: a time-centric approach for the information environment, Proceedings of the 12th annual ACM symposium on User interface software and technology, p.45-54, November 07-10, 1999, Asheville, North Carolina, United States [doi>10.1145/320719.322582]
• 47: Jun Rekimoto , Katashi Nagao, The world through the computer: computer augmented interaction with real world environments, Proceedings of the 8th annual ACM symposium on User interface and software technology, p.29-36, November 15-17, 1995, Pittsburgh, Pennsylvania, United States [doi>10.1145/215585.215639]
• 48: RHODES, B.J. 1997. The wearable remembrance agent: A system for augmented memory. Pers. Tech. J. 1, 4, 218-224.
• 49: RHODES, B. J. AND STARNER, T. E. 1996. Remembrance agent: A continuously running automated information retrieval system. In Proceedings of Practical Applications of Intelligent Agents and Multi-Agent Technology (PAAM),
• 50: RICHTER, H., SCHUCHHARD, P., AND ABOWD, G. D. 1999. Automated capture and retrieval of architectural rationale. In Online Proceedings of the 1st Working IFIP Conference on Software Architecture (San Antonio, TX), IFIP, Laxenburg, Austria. Full paper appears as Georgia Tech GVU Center Technical Report GIT-GVU-98-37.
• 51: Daniel Salber , Anind K. Dey , Gregory D. Abowd, The context toolkit: aiding the development of context-enabled applications, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.434-441, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303126]
• 52: SCHMANDT, C. AND ARONS, B. 1985. PhoneSlave: A graphical telecommunications interface. Proc. Soc. Inf. Display 26, 1, 79-82.
• 53: SCHOMAKER, L. R. B. 1994. User-interface aspects in recognizing connected-cursive handwriting. In Proceedings of the IEE Colloquium on Handwriting and Pen-Based Input, The Institution of Electrical Engineers, London, UK.
• 54: Frank M. Shipman, III , Catherine C. Marshall , Thomas P. Moran, Finding and using implicit structure in human-organized spatial layouts of information, Proceedings of the SIGCHI conference on Human factors in computing systems, p.346-353, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223949]
• 55: STARNER, T., MANN, S., RHODES, B., LEVINE, J., HEALEY, J., KIRSCH, D., PICARD, R., AND PENTLAND, A. 1997. Augmented reality throuh wearable computing. Presence: Teleoper. Virtual Environ. 6, 4.
• 56: Lisa J. Stifelman, Augmenting real-world objects: a paper-based audio notebook, Conference companion on Human factors in computing systems: common ground, p.199-200, April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/257089.257279]
• 57: Lisa Joy Stifelman , Christopher M. Schmandt, The audio notebook: paper and pen interaction with structured speech, 1997
• 58: Norbert A. Streitz , Jörg Geißler , Torsten Holmer , Shin'ichi Konomi , Christian Müller-Tomfelde , Wolfgang Reischl , Petra Rexroth , Peter Seitz , Ralf Steinmetz, i-LAND: an interactive landscape for creativity and innovation, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.120-127, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303010]
• 59: Khai N. Truong , Gregory D. Abowd , Jason A. Brotherton, Personalizing the capture of public experiences, Proceedings of the 12th annual ACM symposium on User interface software and technology, p.121-130, November 07-10, 1999, Asheville, North Carolina, United States [doi>10.1145/320719.322593]
• 60: TURK, M., Ed. 1997. Proceedings of the Perceptual User Interfaces Workshop. (PUI '97, Banff, Alberta, Canada, Oct. 20-21). http://research.microsoft.com/PUIWorkshop97/.
• 61: TURK, M., Ed. 1998. Proceedings of the Perceptual User Interfaces Workshop. (PIU '98, San Francisco, CA, Nov. 5-6). http://research.microsoft.com/PUIWorkshop/PUI98.htm.
• 62: Ron Van Buskirk , Mary LaLomia, The just noticeable difference of speech recognition accuracy, Conference companion on Human factors in computing systems, p.95, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223355.223446]
• 63: Roy Want , Andy Hopper , Veronica Falcão , Jonathan Gibbons, The active badge location system, ACM Transactions on Information Systems (TOIS), v.10 n.1, p.91-102, Jan. 1992 [doi>10.1145/128756.128759]
• 64: WANT, R., SCHILIT, B., ADAMS, N., GOLD, R., PETERSEN, K., ELLIS, J., GOLDBERG, D., AND WEISER, M. 1995. The PARCTab ubiquitous computing experiment. Tech. Rep. CSL-95-1. Xerox PARC, Palo Alto, CA.
• 65: Karon Weher , Alex Poon, Marquee: a tool for real-time video logging, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.58-64, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191697]
• 66: WEISER, M. 1991. The computer for the 21st century. Sci. Am. 265, 3 (Sept.), 94-104.
• 67: Mark Weiser, Some computer science issues in ubiquitous computing, Communications of the ACM, v.36 n.7, p.75-84, July 1993 [doi>10.1145/159544.159617]
• 68: Pierre Wellner, Interacting with paper on the DigitalDesk, Communications of the ACM, v.36 n.7, p.87-96, July 1993 [doi>10.1145/159544.159630]
• 69: WHITE, S., GUPTA, A., GRUDIN, J., KIMBERLY, G., AND SANOCKI, E. 1998. A software system for education at a distance: Case study results. Microsoft Research Tech. Rep. MSR-TR-98-61. Microsoft Corp., Redmond, WA. Revised in April 1999.
• 70: Steve Whittaker , Patrick Hyland , Myrtle Wiley, FILOCHAT: handwritten notes provide access to recorded conversations, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.271-277, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191763]
• 71: Lynn D. Wilcox , Bill N. Schilit , Nitin Sawhney, Dynomite: a dynamically organized ink and audio notebook, Proceedings of the SIGCHI conference on Human factors in computing systems, p.186-193, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258700]
• 72: Catherine G. Wolf , James R. Rhyne , Laura K. Briggs, Communication and information retrieval with a pen-based meeting support tool, Proceedings of the 1992 ACM conference on Computer-supported cooperative work, p.322-329, November 01-04, 1992, Toronto, Ontario, Canada [doi>10.1145/143457.143539]
• 73: Mary Zajicek , Jill Hewitt, An investigation into the use of error recovery dialogues in a user interface management system for speech recognition, Proceedings of the IFIP TC13 Third Interational Conference on Human-Computer Interaction, p.755-760, August 27-31, 1990

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)

  J. Computer Applications

  K. Computing Milieux
  K.4 COMPUTERS AND SOCIETY

General Terms:
Human Factors

Keywords:
augmented reality, capture and access, context-aware applications, evaluation, everyday computing, natural interfaces, social implications, ubiquitous computing, user interfaces