iSymphony: an adaptive interactive orchestral conducting system for digital audio and video streams

ABSTRACT

We present iSymphony, an interactive orchestral conducting system for digital audio and video that adaptively adjusts to the user's conducting style. Using a digital baton, users may control the tempo, volume, and instrument emphasis of a digital audio and video recording of an orchestra. The system adaptively recognizes three gesture profiles: the four-beat neutral-legato pattern, an up-down pattern, and random gestures. The system uses an audio time-stretching algorithm we developed that allows the playback speed of a digital audio recording to be arbitrarily adjusted without changing its pitch. iSymphony is an example of how computers can enable more people to experience an interaction style normally limited to a few people (conductors), and is installed as part of the It's Artastic! exhibit at the Betty Brinn Children's Museum in Milwaukee, USA.

REFERENCES

• 1: Jan Borchers , Frank Buschmann, A Pattern Approach to Interaction Design, John Wiley & Sons, Inc., New York, NY, 2001
• 2: Borchers, J., Lee, E., Samminger, W., and Mühlhäuser, M. Personal Orchestra: A real-time audio/video system for interactive conducting. ACM Multimedia Journal Special Issue on Multimedia Software Engineering 9(5), 2004, 458--465, Errata published in next issue.
• 3: Grüll, I. conga: A conducting gesture analysis framework. Diploma thesis, University of Ulm, 2005.
• 4: Eric Lee , Thorsten Karrer , Jan Borchers, Toward a Framework for Interactive Systems to Conduct Digital Audio and Video Streams, Computer Music Journal, v.30 n.1, p.21-36, March 2006 [doi>10.1162/014892606776021317]
• 5: Eric Lee , Teresa Marrin Nakra , Jan Borchers, You're the conductor: a realistic interactive conducting system for children, Proceedings of the 2004 conference on New interfaces for musical expression, p.68-73, June 03-05, 2004, Hamamatsu, Shizuoka, Japan

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Artificial, augmented, and virtual realities

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Audio input/output; Evaluation/methodology
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology; User-centered design

Keywords:
adaptive gesture recognition, conducting, exhibits, gestures, music interfaces