Experiences with recombinant computing: Exploring ad hoc interoperability in evolving digital networks

ABSTRACT

This article describes an infrastructure that supports the creation of interoperable systems while requiring only limited prior agreements about the specific forms of communication between these systems. Conceptually, our approach uses a set of “meta-interfaces”—agreements on how to exchange new behaviors necessary to achieve compatibility at runtime, rather than requiring that communication specifics be built in at development time—to allow devices on the network to interact with one another. While this approach to interoperability can remove many of the system-imposed constraints that prevent fluid, ad hoc use of devices now, it imposes its own limitations on the user experience of systems that use it. Most importantly, since devices may be expected to work with peers about which they have no detailed semantic knowledge, it is impossible to achieve the sort of tight semantic integration that can be obtained using other approaches today, despite the fact that these other approaches limit interoperability. Instead, under our model, users must be tasked with performing the sense-making and semantic arbitration necessary to determine how any set of devices will be used together. This article describes the motivation and details of our infrastructure, its implications on the user experience, and our experience in creating, deploying, and using applications built with it over a period of several years.

REFERENCES

• 1: Balfanz, D., Smetters, D. K., Stewart, P., and Wong, H. C. 2002. Talking to strangers: Authentication in ad hoc wireless networks. In Proceedings of the Network and Distributed Systems Security Symposium (NDSS'02).
• 2: Renata Bandelloni , Giulio Mori , Fabio Paternò, Dynamic generation of web migratory interfaces, Proceedings of the 7th international conference on Human computer interaction with mobile devices & services, September 19-22, 2005, Salzburg, Austria [doi>10.1145/1085777.1085792]
• 3: Krishna A. Bharat , Luca Cardelli, Migratory applications, Proceedings of the 8th annual ACM symposium on User interface and software technology, p.132-142, November 15-17, 1995, Pittsburgh, Pennsylvania, United States [doi>10.1145/215585.215711]
• 4: Black, J. A., Edwards, W. K., Newman, M. W., Sedivy, J. Z., and Smith, T. F. 2003. Supporting extensible public display systems with Speakeasy. In Public and Situated Displays: Social and Interactional Aspects of Shared Display Technologies, K. O'Hara, et al. Eds., Kluwer Academic, Amsterdam.
• 5: Bluetooth Consortium. 2001. Specification of the Bluetooth System, Version 1.1 Core. http://www.bluetooth.com.
• 6: N. Borenstein , N. Freed, MIME (Multipurpose Internet Mail Extensions): Mechanisms for Specifying and Describing the Format of Internet Message Bodies, RFC Editor, 1992
• 7: Box, D., Ehnebuske, D., Kakivaya, G., Layman, A., Mendelsohn, N., Nielsen, H. F., Thatte, S., and Winer, D. 2000. W3C note: Simple object access protocol (SOAP) 1.1. World Wide Web Consortium.
• 8: Chakraborty, D. and Joshi, A. 2001. Dynamic service composition: state-of-the-art and research directions. Tech. rep.TR-CS-01-19, CSEE, University of Maryland, Baltimore.
• 9: Nicolas Ducheneaut , Trevor F. Smith , James Bo Begole , Mark W. Newman , Chris Beckmann, The orbital browser: composing ubicomp services using only rotation and selection, CHI '06 extended abstracts on Human factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1125451.1125523]
• 10: W. Keith Edwards , Mark W. Newman , Jana Sedivy , Trevor Smith , Shahram Izadi, Challenge: recombinant computing and the speakeasy approach, Proceedings of the 8th annual international conference on Mobile computing and networking, September 23-28, 2002, Atlanta, Georgia, USA [doi>10.1145/570645.570680]
• 11: W. Keith Edwards , Mark W. Newman , Jana Z. Sedivy , Trevor F. Smith , Dirk Balfanz , D. K. Smetters , H. Chi Wong , Shahram Izadi, Using speakeasy for ad hoc peer-to-peer collaboration, Proceedings of the 2002 ACM conference on Computer supported cooperative work, November 16-20, 2002, New Orleans, Louisiana, USA [doi>10.1145/587078.587114]
• 12: W. Keith Edwards , Victoria Bellotti , Anind K. Dey , Mark W. Newman, The challenges of user-centered design and evaluation for infrastructure, Proceedings of the SIGCHI conference on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/642611.642664]
• 13: Edwards, W. K., Newman, M. W., Sedivy, J. Z. and Smith, T. F. 2004. Supporting serendipitous integration in mobile computing environments. Int. J. Hum. Comput. Stud. 60, 666--700.
• 14: Edwards, W. K., Newman, M. W., Sedivy, J. Z., and Smith, T. F. 2005. An extensible set- top box platform for home media applications. IEEE Trans.Consumer Electron. 51, 4, 1175--1181.
• 15: W. Keith Edwards, Discovery Systems in Ubiquitous Computing, IEEE Pervasive Computing, v.5 n.2, p.70-77, April 2006 [doi>10.1109/MPRV.2006.28]
• 16: Fox, A., Goldberg, I., Gribble, S. D., Lee, D. C., Polito, A., and Brewer, E. A. 1998. Experience with Top Gun Wingman: A proxy-based graphical web browser for the 3com Palm Pilot. In Proceedings of the Middleware Conference.
• 17: David Gelernter, Generative communication in Linda, ACM Transactions on Programming Languages and Systems (TOPLAS), v.7 n.1, p.80-112, Jan. 1985 [doi>10.1145/2363.2433]
• 18: Li Gong, Inside Java 2 platform security architecture, API design, and implementation, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1999
• 19: Steven D. Gribble , Matt Welsh , Rob von Behren , Eric A. Brewer , David Culler , N. Borisov , S. Czerwinski , R. Gummadi , J. Hill , A. Joseph , R. H. Katz , Z. M. Mao , S. Ross , B. Zhao , Robert C. Holte, The Ninja architecture for robust Internet-scale systems and services373423, Computer Networks: The International Journal of Computer and Telecommunications Networking, v.35 n.4, p.473-497, March 2001 [doi>10.1016/S1389-1286(00)00179-1]
• 20: Harmonia Inc. 2000. User interface modeling language 2.0 draft specification. http://www.uiml.org/specs/uiml2/index.htm.
• 21: Richard Han , Veronique Perret , Mahmoud Naghshineh, WebSplitter: a unified XML framework for multi-device collaborative Web browsing, Proceedings of the 2000 ACM conference on Computer supported cooperative work, p.221-230, December 2000, Philadelphia, Pennsylvania, United States [doi>10.1145/358916.358993]
• 22: Todd D. Hodes , Randy H. Katz, A document-based framework for internet application control, Proceedings of the 2nd conference on USENIX Symposium on Internet Technologies and Systems, p.6-6, October 11-14, 1999, Boulder, Colorado
• 23: Andrew C. Huang , Benjamin C. Ling , John Barton , Armando Fox, Making computers disappear: appliance data services, Proceedings of the 7th annual international conference on Mobile computing and networking, p.108-121, July 2001, Rome, Italy [doi>10.1145/381677.381688]
• 24: Humble, J., Crabtree, A., Hemmings, T., Akesson, K.-P., Koleva, B., Rodden, T. and Hansson, P. 2003. Playing with the bits: User-configuration of ubiquitous domestic environments. In Proceedings of the 5th International Conference on Ubiquitous Computing.
• 25: Internet Engineering Task Force (IETF). 2005. Zeroconf Working Group. http://www.zeroconf.org.
• 26: Brad Johanson , Armando Fox , Terry Winograd, The Interactive Workspaces Project: Experiences with Ubiquitous Computing Rooms, IEEE Pervasive Computing, v.1 n.2, p.67-74, April 2002 [doi>10.1109/MPRV.2002.1012339]
• 27: Jeronimo, M. and Weast, J. 2003. UPnP Design by Example. Intel Press.
• 28: E. Kõcõman , A. Fox, Towards Zero-Code Service Composition, Proceedings of the Eighth Workshop on Hot Topics in Operating Systems, p.172, May 20-22, 2001
• 29: Gregor Kiczales , Jim Des Rivieres, The Art of the Metaobject Protocol, MIT Press, Cambridge, MA, 1991
• 30: Tim Kindberg , John Barton, A Web-based nomadic computing system, Computer Networks: The International Journal of Computer and Telecommunications Networking, v.35 n.4, p.443-456, March 2001 [doi>10.1016/S1389-1286(00)00181-X]
• 31: Tim Kindberg , Armando Fox, System Software for Ubiquitous Computing, IEEE Pervasive Computing, v.1 n.1, p.70-81, January 2002 [doi>10.1109/MPRV.2002.993146]
• 32: Keunwoo Lee , Anthony LaMarca , Craig Chambers, HydroJ: object-oriented pattern matching for evolvable distributed systems, Proceedings of the 18th annual ACM SIGPLAN conference on Object-oriented programing, systems, languages, and applications, October 26-30, 2003, Anaheim, California, USA
• 33: Barbara H. Liskov , Jeannette M. Wing, A behavioral notion of subtyping, ACM Transactions on Programming Languages and Systems (TOPLAS), v.16 n.6, p.1811-1841, Nov. 1994 [doi>10.1145/197320.197383]
• 34: Wendy E. Mackay, Patterns of sharing customizable software, Proceedings of the 1990 ACM conference on Computer-supported cooperative work, p.209-221, October 07-10, 1990, Los Angeles, California, United States [doi>10.1145/99332.99356]
• 35: Wendy E. Mackay, Triggers and barriers to customizing software, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.153-160, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108867]
• 36: Mao, Z. M. and Katz, R. H. 2002. Achieving service portability using self-adaptive data paths. IEEE Comm. 40, 1, 108--114.
• 37: Mark W. Newman , Shahram Izadi , W. Keith Edwards , Jana Z. Sedivy , Trevor F. Smith, User interfaces when and where they are needed: an infrastructure for recombinant computing, Proceedings of the 15th annual ACM symposium on User interface software and technology, October 27-30, 2002, Paris, France [doi>10.1145/571985.572009]
• 38: Mark W. Newman , Jana Z. Sedivy , Christine M. Neuwirth , W. Keith Edwards , Jason I. Hong , Shahram Izadi , Karen Marcelo , Trevor F. Smith, Designing for serendipity: supporting end-user configuration of ubiquitous computing environments, Proceedings of the 4th conference on Designing interactive systems: processes, practices, methods, and techniques, June 25-28, 2002, London, England [doi>10.1145/778712.778736]
• 39: Mark W. Newman , Nicolas Ducheneaut , W. Keith Edwards , Jana Z. Sedivy , Trevor F. Smith, Supporting the unremarkable: experiences with the obje Display Mirror, Personal and Ubiquitous Computing, v.11 n.7, p.523-536, October 2007 [doi>10.1007/s00779-006-0117-0]
• 40: Mark W. Newman , Ame Elliott , Trevor F. Smith, Providing an Integrated User Experience of Networked Media, Devices, and Services through End-User Composition, Proceedings of the 6th International Conference on Pervasive Computing, May 19-22, 2008, Sydney, Australia [doi>10.1007/978-3-540-79576-6_13]
• 41: Jeffrey Nichols , Brad A. Myers , Michael Higgins , Joseph Hughes , Thomas K. Harris , Roni Rosenfeld , Mathilde Pignol, Generating remote control interfaces for complex appliances, Proceedings of the 15th annual ACM symposium on User interface software and technology, October 27-30, 2002, Paris, France [doi>10.1145/571985.572008]
• 42: Object Management Group. 1995. CORBA: The common object request broker architecture, Rev. 2.0.
• 43: Ockerbloom, J. 1998. Mediating among diverse data formats. Tech.rep., CMU-CS-98-10, Carnegie Mellon University.
• 44: Dan R. Olsen, Jr. , Sean Jefferies , Travis Nielsen , William Moyes , Paul Fredrickson, Cross-modal interaction using XWeb, Proceedings of the 13th annual ACM symposium on User interface software and technology, p.191-200, November 06-08, 2000, San Diego, California, United States [doi>10.1145/354401.354764]
• 45: Omojokun, O. and Dewan, P. 2003. A high-level and flexible framework for dynamically composing networked devices. In Proceedings of the 5th IEEE Workshop on Mobile Computing Systems and Applications (WMCSA) .
• 46: Shankar Ponnekanti , Brian Lee , Armando Fox , Pat Hanrahan , Terry Winograd, ICrafter: A Service Framework for Ubiquitous Computing Environments, Proceedings of the 3rd international conference on Ubiquitous Computing, p.56-75, September 30-October 02, 2001, Atlanta, Georgia, USA
• 47: Shankar R. Ponnekanti , Armando Fox, Interoperability among independently evolving web services, Proceedings of the 5th ACM/IFIP/USENIX international conference on Middleware, October 18-22, 2004, Toronto, Canada
• 48: Rekimoto, J., Ayatsuka, Y., Kohno, M., and Oba, H. 2003. Proximal Interactions: A direct manipulation technique for wireless networking. In Proceedings of the INTERACT Conference. Richardson, T., Stafford-Fraser, Q., Wood, K. and Hopper, A. 1998. Virtual network computing. IEEE Internet Comput. 2, 1.
• 49: M. Rose, The Blocks Extensible Exchange Protocol Core, RFC Editor, 2001
• 50: D. K. Smetters , R. E. Grinter, Moving from the design of usable security technologies to the design of useful secure applications, Proceedings of the 2002 workshop on New security paradigms, September 23-26, 2002, Virginia Beach, Virginia [doi>10.1145/844102.844117]
• 51: Smetters, D. K., Balfanz, D., Durfee, G., Smith, T., and Lee, K. 2006. Instant matchmaking: Simple secure virtual extensions to ubiquitous computing environments. In Proceedings of the 8th International Conference on Ubiquitous Computing (UBICOMP). Lecture Notes in Computer Science, vol. 4206, Springer, Berlin, 477--494.
• 52: Frank Stajano , Ross J. Anderson, The Resurrecting Duckling: Security Issues for Ad-hoc Wireless Networks, Proceedings of the 7th International Workshop on Security Protocols, p.172-194, April 19-21, 1999
• 53: Sun Microsystems. 1999. Jini discovery and join specification.
• 54: Colin Swindells , Kori M. Inkpen , John C. Dill , Melanie Tory, That one there! Pointing to establish device identity, Proceedings of the 15th annual ACM symposium on User interface software and technology, October 27-30, 2002, Paris, France [doi>10.1145/571985.572007]
• 55: Truong, K. N., Huang, E. M., and Abowd, G. D. 2004. CAMP: A magnetic poetry interface for end-user programming of capture applications for the home. In Proceedings of the 6th International Conference on Ubiquitous Computing (UBICOMP). 143--160.
• 56: UPNP Forum. 2005. MediaServer V 1.0 and Media Renderer V 1.0. http://www.upnp.org/standardizeddcps/mediaserver.asp.
• 57: Universal Serial Bus Implementer's Forum. 2000. Universal serial bus revision 2.0 specification.
• 58: Venners, B. 2005. The ServiceUI API specification, version 1.1a. http://www.artima.com/jini/serviceui/Spec.html.
• 59: Stephen Voida , W. Keith Edwards , Mark W. Newman , Rebecca E. Grinter , Nicolas Ducheneaut, Share and share alike: exploring the user interface affordances of file sharing, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124806]
• 60: Jim Waldo, The Jini architecture for network-centric computing, Communications of the ACM, v.42 n.7, p.76-82, July 1999 [doi>10.1145/306549.306582]
• 61: Weiser, M. and Brown, J. S. 1996. Designing calm technology. http://powergrid.electriciti.com/1.01.
• 62: Alma Whitten , J. D. Tygar, Why Johnny can't encrypt: a usability evaluation of PGP 5.0, Proceedings of the 8th conference on USENIX Security Symposium, p.14-14, August 23-26, 1999, Washington, D.C.
• 63: Jeannette M. Wing , John Ockerbloom, Respectful Type Converters, IEEE Transactions on Software Engineering, v.26 n.7, p.579-593, July 2000 [doi>10.1109/32.859529]
• 64: Ann Wollrath , Roger Riggs , Jim Waldo, A distributed object model for the java^TM system, Proceedings of the 2nd conference on USENIX Conference on Object-Oriented Technologies (COOTS), p.17-17, June 17-21, 1996, Toronto, Ontario, Canada

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: User-centered design

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
        D.2.12 Interoperability
            Subjects: Distributed objects

  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human factors
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Prototyping; Theory and methods

General Terms:
Design, Human Factors, Standardization

Keywords:
Mobile code, Obje, Speakeasy, discovery, infrastructure, interoperability, recombinant computing, ubiquitous computing, user interfaces