Understanding performance in coliseum, an immersive videoconferencing system

ABSTRACT

Coliseum is a multiuser immersive remote teleconferencing system designed to provide collaborative workers the experience of face-to-face meetings from their desktops. Five cameras are attached to each PC display and directed at the participant. From these video streams, view synthesis methods produce arbitrary-perspective renderings of the participant and transmit them to others at interactive rates, currently about 15 frames per second. Combining these renderings in a shared synthetic environment gives the appearance of having all participants interacting in a common space. In this way, Coliseum enables users to share a virtual world, with acquired-image renderings of their appearance replacing the synthetic representations provided by more conventional avatar-populated virtual worlds. The system supports virtual mobility---participants may move around the shared space---and reciprocal gaze, and has been demonstrated in collaborative sessions of up to ten Coliseum workstations, and sessions spanning two continents.Coliseum is a complex software system which pushes commodity computing resources to the limit. We set out to measure the different aspects of resource, network, CPU, memory, and disk usage to uncover the bottlenecks and guide enhancement and control of system performance. Latency is a key component of Quality of Experience for video conferencing. We present how each aspect of the system---cameras, image processing, networking, and display---contributes to total latency. Performance measurement is as complex as the system to which it is applied. We describe several techniques to estimate performance through direct light-weight instrumentation as well as use of realistic end-to-end measures that mimic actual user experience. We describe the various techniques and how they can be used to improve system performance for Coliseum and other network applications. This article summarizes the Coliseum technology and reports on issues related to its performance---its measurement, enhancement, and control.

REFERENCES

• 1: Baker, H. H., Tanguay, D., Sobel, I., Gelb, D., Goss, M. E., Culbertson, W. B., and Malzbender, T. 2002. The Coliseum Immersive Teleconferencing System. In Proceedings of the International Workshop on Immersive Telepresence (Juan Les Pins, France, Dec.). ACM, New York.
• 2: H. Harlyn Baker , Nina Bhatti , Donald Tanguay , Irwin Sobel , Dan Gelb , Michael E. Goss , John MacCormick , Kei Yuasa , W. Bruce Culbertson , Thomas Malzbender, Computation and performance issues In coliseum: an immersive videoconferencing system, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957118]
• 3: Baker, H. H. and Tanguay, D. 2004. Graphics-accelerated panoramic mosaicking from a video camera array. In Proceedings of Vision, Modeling, and Visualization Workshop (Stanford, Calif., Nov.). IOS Press. 133--140.
• 4: Joseph Buck , Soonhoi Ha , Edward A. Lee , David G. Messerschmitt, Ptolemy: a framework for simulating and prototyping heterogeneous systems, Readings in hardware/software co-design, Kluwer Academic Publishers, Norwell, MA, 2001
• 5: Milton Chen, Design of a virtual auditorium, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500147]
• 6: Devernay, F. and Faugeras, O. 1995. Automatic calibration and removal of distortion from scenes of structured environments. SPIE, 2567 (San Diego, Calif., July). 62--72.
• 7: Gharai, L., Perkins, C., Riley, R., and Mankin, A. 2002. Large scale video conferencing: A digital amphitheater. In Proceedings of the 8th International Conference on Distributed Multimedia Systems (San Francisco, Calif., Sept.).
• 8: Gordon, R. and Talley, S. 1999. Essential JMF: Java Media Framework. Prentice-Hall, Englewood Cliffs, N.J.
• 9: Peter Henderson , James H. Morris, Jr., A lazy evaluator, Proceedings of the 3rd ACM SIGACT-SIGPLAN symposium on Principles on programming languages, p.95-103, January 19-21, 1976, Atlanta, Georgia [doi>10.1145/800168.811543]
• 10: Hirsh, S., Sellen, A., and Brokopp, N. 2004. Why HP people do and don't use videoconferencing systems. Hewlett-Packard Laboratories, External Tech. Rep. HPL-2004-140R1.
• 11: Lanier, J. 2001. Virtually There. Scientific American, Apr., 66--75.
• 12: Marco Lohse , Michael Repplinger , Philipp Slusallek, An Open Middleware Architecture for Network-Integrated Multimedia, Proceedings of the Joint International Workshops on Interactive Distributed Multimedia Systems and Protocols for Multimedia Systems: Protocols and Systems for Interactive Distributed Multimedia, p.327-338, November 26-29, 2002
• 13: Longuet-Higgins, H. C. 1981. A computer algorithm for reconstructing a scene from two projections. Nature 293, 133--135.
• 14: Wojciech Matusik , Chris Buehler , Ramesh Raskar , Steven J. Gortler , Leonard McMillan, Image-based visual hulls, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.369-374, July 2000 [doi>10.1145/344779.344951]
• 15: P. J. Narayanan , Peter W. Rander , Takeo Kanade, Constructing Virtual Worlds Using Dense Stereo, Proceedings of the Sixth International Conference on Computer Vision, p.3, January 04-07, 1998
• 16: Mark D. Pesce, Programming Microsoft DirectShow for Digital Video, Television, and DVD, Microsoft Press, Redmond, WA, 2002
• 17: Pollefeys, M. 1999. Self-calibration and metric 3D reconstruction from uncalibrated image sequences. Ph.D. dissertation. ESAT-PSI, K.U. Leuven.
• 18: Prince, S., Cheok, A., Farbiz, F., Williamson, T., Johnson, N., Billinghurst M., and Kato, H. 2002. Real-time 3D interaction for augmented and virtual reality. In Proceedings of SIGGRAPH 2002. ACM, New York, 238.
• 19: Schreer, O., Brandenburg, N., Askar, S., and Trucco, E. 2001. A virtual 3D video-conferencing system providing semi-immersive telepresence: A real-time solution in hardware and software. In Proceedings of the International Conference on eWork and eBusiness. (Venice., Italy). 184--190.
• 20: Steven M. Seitz , Charles R. Dyer, Photorealistic Scene Reconstruction by Voxel Coloring, Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97), p.1067, June 17-19, 1997
• 21: Slabaugh, G., Schafer, R., and Hans, M. 2002. Image-based photo hulls. In Proceedings of the 1st International Symposium on 3D Processing, Visualization, and Transmission. (Padua, Italy). 704--708.
• 22: Tanguay, D., Gelb, D., and Baker, H. H. 2004. Nizza: A framework for developing real-time streaming multimedia applications. Hewlett-Packard Laboratories, Technical Report, HPL-2004-132.
• 23: James R. Wilcox, Videoconferencing, C M P Books, 2000
• 24: Zhengyou Zhang, A Flexible New Technique for Camera Calibration, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.11, p.1330-1334, November 2000 [doi>10.1109/34.888718]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS
      H.4.3 Communications Applications
          Subjects: Computer conferencing, teleconferencing, and videoconferencing

General Terms:
Algorithms, Design, Experimentation, Measurement, Performance

Keywords:
3D virtual environments, Telepresence, network applications, performance measurement, streaming media, videoconferencing, view synthesis