Towards environment-to-environment (E2E) multimedia communication systems

ABSTRACT

We present an approach to connect multiple remote environments for natural interaction among people and objects. Focus of current communication and telepresence systems severely restrict user affordances in terms of movement, interaction, peripheral vision, spatio-semantic integrity and even information flow. These systems allow information transfer rather than experiential interaction. We propose Environment-to-Environment (E2E) as a new paradigm for communication which allows users to interact in natural manner using text, audio, and video by connecting environments. Each Environment is instrumented using as many different types of sensors as may be required to detect presence and activity of objects and this object position and activity information is used to direct multimedia information to be sent to other Environments as well as present incoming multimedia information on right displays and speakers. The mediation for the appropriate data capture and presentation is done by a scalable event-based multimodal information system. This paper describes the design principles for E2E communication, discusses system architecture, and gives our experience in implementing prototypes of such systems in telemedicine and office collaboration applications. We also discuss the research challenges and a road-map for creating more sophisticated E2E applications in near future.

REFERENCES

• 1: Abigail Sellen , Bill Buxton , John Arnott, Using spatial cues to improve videoconferencing, Proceedings of the SIGCHI conference on Human factors in computing systems, p.651-652, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.143070]
• 2: Pradeep K. Atrey , Mohan S. Kankanhalli , Ramesh Jain, Timeline-based information assimilation in multimedia surveillance and monitoring systems, Proceedings of the third ACM international workshop on Video surveillance & sensor networks, November 11-11, 2005, Hilton, Singapore [doi>10.1145/1099396.1099416]
• 3: Sara A. Bly , Steve R. Harrison , Susan Irwin, Media spaces: bringing people together in a video, audio, and computing environment, Communications of the ACM, v.36 n.1, p.28-46, Jan. 1993 [doi>10.1145/151233.151235]
• 4: Susanne Boll , Utz Westermann, MediÆther: an event space for context-aware multimedia experiences, Proceedings of the 2003 ACM SIGMM workshop on Experiential telepresence, November 07, 2003, Berkeley, California [doi>10.1145/982484.982488]
• 5: Jeffrey W. Chastine , Kristine Nagel , Ying Zhu , Luca Yearsovich, Understanding the design space of referencing in collaborative augmented reality environments, Proceedings of Graphics Interface 2007, May 28-30, 2007, Montreal, Canada [doi>10.1145/1268517.1268552]
• 6: Gamhewage C. de Silva , Toshihiko Yamasaki , Kiyoharu Aizawa, Evaluation of video summarization for a large number of cameras in ubiquitous home, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101329]
• 7: William Gaver , Thomas Moran , Allan MacLean , Lennart Lövstrand , Paul Dourish , Kathleen Carter , William Buxton, Realizing a video environment: EuroPARC's RAVE system, Proceedings of the SIGCHI conference on Human factors in computing systems, p.27-35, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142754]
• 8: Heidi Gregersen, The formal semantics of the timeER model, Proceedings of the 3rd Asia-Pacific conference on Conceptual modelling, p.35-44, January 16-19, 2006, Hobart, Australia
• 9: Christian Heath , Paul Luff, Disembodied conduct: communication through video in a multi-media office environment, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.99-103, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108859]
• 10: R. Jain. Out-of-the-box data engineering events in heterogeneous data environments. Data Engineering, 2003. Proceedings. 19th International Conference on, pages 8--21, 5-8 March 2003.
• 11: Ramesh Jain , Pilho Kim , Zhao Li, Experiential meeting system, Proceedings of the 2003 ACM SIGMM workshop on Experiential telepresence, November 07, 2003, Berkeley, California [doi>10.1145/982484.982486]
• 12: Norman P. Jouppi , Subu Iyer , Stan Thomas , April Slayden, BiReality: mutually-immersive telepresence, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027725]
• 13: M. Kankanhalli, J. Wang, and R. Jain. Experiential sampling on multiple data streams. Multimedia, IEEE Transactions on, 8(5):947--955, Oct. 2006.
• 14: Arun Katkere , Saied Moezzi , Don Y. Kuramura , Patrick Kelly , Ramesh Jain, Towards video-based immersive environments, Multimedia Systems, v.5 n.2, p.69-85, March 1997 [doi>10.1007/s005300050043]
• 15: Patrick H. Kelly , Arun Katkere , Don Y. Kuramura , Saied Moezzi , Shankar Chatterjee, An architecture for multiple perspective interactive video, Proceedings of the third ACM international conference on Multimedia, p.201-212, November 05-09, 1995, San Francisco, California, United States [doi>10.1145/217279.215267]
• 16: Cory D. Kidd , Robert Orr , Gregory D. Abowd , Christopher G. Atkeson , Irfan A. Essa , Blair MacIntyre , Elizabeth D. Mynatt , Thad Starner , Wendy Newstetter, The Aware Home: A Living Laboratory for Ubiquitous Computing Research, Proceedings of the Second International Workshop on Cooperative Buildings, Integrating Information, Organization, and Architecture, p.191-198, October 01-02, 1999
• 17: Wei Liu , Adrian David Cheok , Charissa Lim Mei-Ling , Yin-Leng Theng, Mixed reality classroom: learning from entertainment, Proceedings of the 2nd international conference on Digital interactive media in entertainment and arts, September 19-21, 2007, Perth, Australia [doi>10.1145/1306813.1306833]
• 18: Aditi Majumder , Rick Stevens, Perceptual photometric seamlessness in projection-based tiled displays, ACM Transactions on Graphics (TOG), v.24 n.1, p.118-139, January 2005 [doi>10.1145/1037957.1037964]
• 19: Gloria Mark , Steve Abrams , Nayla Nassif, Group-to-group distance collaboration: examining the "space between", Proceedings of the eighth conference on European Conference on Computer Supported Cooperative Work, p.99-118, September 14-18, 2003, Helsinki, Finland
• 20: G. Mark and P. DeFlorio. An experiment using life-size hdtv. In Proc. IEEE Workshop on Advanced Collaborative Environments (WACE), 2001.
• 21: Steven McCanne , Van Jacobson, vic: a flexible framework for packet video, Proceedings of the third ACM international conference on Multimedia, p.511-522, November 05-09, 1995, San Francisco, California, United States [doi>10.1145/217279.215315]
• 22: Simon Moncrieff , Svetha Venkatesh , Geoff West, Privacy and the access of information in a smart house environment, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291389]
• 23: David T. Nguyen , John Canny, Multiview: improving trust in group video conferencing through spatial faithfulness, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240846]
• 24: H. Packard. Hp halo overview, 2007.
• 25: Ramesh Raskar , Michael S. Brown , Ruigang Yang , Wei-Chao Chen , Greg Welch , Herman Towles , Brent Seales , Henry Fuchs, Multi-projector displays using camera-based registration, Proceedings of the conference on Visualization '99: celebrating ten years, p.161-168, October 1999, San Francisco, California, United States
• 26: Ramesh Raskar , Greg Welch , Matt Cutts , Adam Lake , Lev Stesin , Henry Fuchs, The office of the future: a unified approach to image-based modeling and spatially immersive displays, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.179-188, July 1998 [doi>10.1145/280814.280861]
• 27: A. A. Sawchuk , E. Chew , R. Zimmermann , C. Papadopoulos , C. Kyriakakis, From remote media immersion to Distributed Immersive Performance, Proceedings of the 2003 ACM SIGMM workshop on Experiential telepresence, November 07, 2003, Berkeley, California [doi>10.1145/982484.982506]
• 28: E. Schooler. A distributed architecture for multimedia conference control. Technical report, University of Southern California, 1991.
• 29: SecondLife. http://secondlife.com/.
• 30: R. Sheppard , W. Wu , Z. Yang , K. Nahrstedt , L. Wymore , G. Kurillo , R. Bajcsy , K. Mezur, New digital options in geographically distributed dance collaborations with TEEVE: tele-immersive environments for everybody, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291469]
• 31: R. Stults. Media space. Technical report, Xerox PARC, 1986.
• 32: R. Vertegaal, G. V. der Veer, and H. Vons. Effects of gaze on multiparty mediated communication. In Proc. Graphics Interface, pages 95--102, 2000.
• 33: Roel Vertegaal , Ivo Weevers , Changuk Sohn , Chris Cheung, GAZE-2: conveying eye contact in group video conferencing using eye-controlled camera direction, Proceedings of the SIGCHI conference on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/642611.642702]
• 34: Mark Weiser, The computer for the 21st century, ACM SIGMOBILE Mobile Computing and Communications Review, v.3 n.3, p.3-11, July 1999 [doi>10.1145/329124.329126]
• 35: G. U. Westermann and R. Jain. Events in multimedia electronic chronicles (e-chronicles). International Journal of Semantic Web and Information Systems (IJSWIS, 2(2):1--27, Oct. 2006.
• 36: Zhenyu Yang , Wanmin Wu , Klara Nahrstedt , Gregorij Kurillo , Ruzena Bajcsy, ViewCast: view dissemination and management for multi-party 3d tele-immersive environments, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291434]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems

Additional Classification:
  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS
      H.4.3 Communications Applications

General Terms:
Design, Human Factors

Keywords:
e2e, environment-to-environment communication, multimedia communication system