3d augmented mirror: a multimodal interface for string instrument learning and teaching with gesture support

ABSTRACT

Multimodal interfaces can open up new possibilities for music education, where the traditional model of teaching is based predominantly on verbal feedback. This paper explores the development and use of multimodal interfaces in novel tools to support music practice training. The design of multimodal interfaces for music education presents a challenge in several respects. One is the integration of multimodal technology into the music learning process. The other is the technological development, where we present a solution that aims to support string practice training with visual and auditory feedback. Building on the traditional function of a physical mirror as a teaching aid, we describe the concept and development of an "augmented mirror" using 3D motion capture technology.

REFERENCES

• 1: Ng, K., Ong, B., Khan, A., Bellini, P., Nesi, P., Mitolo, N. Technology Enhanced Learning with Interactive Multimedia Environment for Music Edutainment. In Proceedings of the EDEN 2006 Annual Conference -- E-Competences for Life, Employment and Innovation, "E" is more! E-learning Enabling Education in Evolving Europe, European Distance and E-Learning Network, Vienna, Austria, 2006, 182--187.
• 2: Ng, K. Creative Multimedia Interfaces and Collaborative Technology Enhanced Environment for Music Playing and Learning. In Proceedings of The Reflective Conservatoire: Sorcerers and Apprentices Conference, London, UK, 2006
• 3: West, T., and Rostvall, A-L. A Study of Interaction and Learning in Instrumental Teaching. International Journal of Music Education. 40,1, 2003 16--27.
• 4: Frederic Bevilacqua , Fabrice Guédy , Norbert Schnell , Emmanuel Fléty , Nicolas Leroy, Wireless sensor interface and gesture-follower for music pedagogy, Proceedings of the 7th international conference on New interfaces for musical expression, June 06-10, 2007, New York, New York [doi>10.1145/1279740.1279762]
• 5: Kramer, G. (ed.), Auditory Display: Sonification, Audification and Auditory Interfaces. Proceedings of the First International Conference on Auditory Display (ICAD) 1992
• 6: Styns, F., van Noorden, L., Leman, M. Some basic observations on how people move on music and how they relate music to movement. Second International Conference on Music and Gesture, Manchester, UK. 2006.
• 7: Effenberg, A. Zum Potential Komplexer Akustischer Bewegungsinformationen für die Technikansteuerung, In Leistungssport 5, 19--25.
• 8: Hermann, T., Höner, O., Ritter, H. AcouMotion -- An Interactive Sonification System for Acoustic Motion Control. In Gibet, S., Courty, N., and Kamp, J.-F., (Eds.):Gesture In Human-Computer Interaction and Simulation -- 6th International Gesture Workshop, (GW'2005), Berder Island, France, 2005, Series LNCS/LNAI 3881, Springer, 312--323.
• 9: Väljamäe, A., Larsson, P., Västfjäl, D., Kleiner, M. Travelling Without Moving: Auditory Scene Cues for Translational Self-Motion. In Proceedings of the International Conference on Auditory Display (ICAD'05). Limerick, Ireland, 2005.
• 10: Mora, J., Won-Sook, L., Comeau, G., Shirmohammadi, S., El Saddik., A. Assisted Piano Pedagogy through 3D Visualization of Piano Playing. In Proceedings of the IEEE International Workshop on Haptic Audio Visual Environments and Their Applications (HAVE' 2006), Ottawa, Canada, 2006.
• 11: Sturm, H., Rabbath F. The Art of the Bow DVD, 2006
• 12: Shan, G. Visentin, P. A Quantitative Three-Dimensional Analysis of Arm Kinematics in Violin Performance. In Medical Problems of Performing Artists, 18, 1. 2003, 3--10.
• 13: Sam Ferguson, Learning musical instrument skills through interactive sonification, Proceedings of the 2006 conference on New interfaces for musical expression, June 04-08, 2006, Paris, France
• 14: Verfaille, V., Quek., O, Wanderley M. M. Sonification of Musicians' Ancillary Gestures, in Proceedings of the International Conference on Auditory Display (ICAD'06), London, UK, 2006
• 15: Bevilaqua, F., Ridenour, J. and Cuccia. D. 3D Motion Capture Data: Motion Analysis and Mapping to Music. In Proceedings of the Symposium on Sensing and Input for Media Centric Systems (SIMS). Santa Barbara, USA. 2002
• 16: Rolland, P. The Teaching of Action in String Playing. Illinois String Research Associates. 1974
• 17: Gerle, R. The Art of Bowing Practice: The Expressive Bow Technique. Stainer & Bell, 1991
• 18: Winstein, C. Knowledge of Results and Motor Learning - Implications for Physical Therapy. Physical Therapy 71, 2. 1991, 140--149.
• 19: Pierfrancesco Bellini , Paolo Nesi , Giorgio Zoia, Symbolic Music Representation in MPEG, IEEE MultiMedia, v.12 n.4, p.42-49, October 2005 [doi>10.1109/MMUL.2005.82]

INDEX TERMS

Primary Classification:
  K. Computing Milieux
  K.3 COMPUTERS AND EDUCATION
      K.3.1 Computer Uses in Education
          Subjects: Computer-assisted instruction (CAI)

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Auditory (non-speech) feedback

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
            Subjects: Motion; 3D/stereo scene analysis
  I.5 PATTERN RECOGNITION
      I.5.5 Implementation
          Subjects: Interactive systems

  J. Computer Applications
  J.5 ARTS AND HUMANITIES
      Subjects: Performing arts (e.g., dance, music)

General Terms:
Design, Human Factors

Keywords:
3d, education, feedback, gesture, interface, motion capture, multimodal, music, sonification, visualisation, visualization