Wireless sensor interface and gesture-follower for music pedagogy

ABSTRACT

We present in this paper a complete gestural interface built to support music pedagogy. The development of this prototype concerned both hardware and software components: a small wireless sensor interface including accelerometers and gyroscopes, and an analysis system enabling gesture following and recognition. A first set of experiments was conducted with teenagers in a music theory class. The preliminary results were encouraging concerning the suitability of these developments in music education.

REFERENCES

• 1: Ryan Aylward , Joseph A. Paradiso, Sensemble: a wireless, compact, multi-user sensor system for interactive dance, Proceedings of the 2006 conference on New interfaces for musical expression, June 04-08, 2006, Paris, France
• 2: Frédéric Bevilacqua , Rémy Müller , Norbert Schnell, MnM: a Max/MSP mapping toolbox, Proceedings of the 2005 conference on New interfaces for musical expression, May 26-28, 2005, Vancouver, Canada
• 3: Frédéric Bevilacqua , Nicolas Rasamimanana , Emmanuel Fléty , Serge Lemouton , Florence Baschet, The augmented violin project: research, composition and performance report, Proceedings of the 2006 conference on New interfaces for musical expression, June 04-08, 2006, Paris, France
• 4: Ari Y. Benbasat , Joseph A. Paradiso, An Inertial Measurement Framework for Gesture Recognition and Applications, Revised Papers from the International Gesture Workshop on Gesture and Sign Languages in Human-Computer Interaction, p.9-20, April 18-20, 2001
• 5: Jan Borchers , Aristotelis Hadjakos , Max Mühlhäuser, MICON a music stand for interactive conducting, Proceedings of the 2006 conference on New interfaces for musical expression, June 04-08, 2006, Paris, France
• 6: Thierry Coduys , Cyrille Henry , Arshia Cont, TOASTER and KROONDE: high-resolution and high-speed real-time sensor interfaces, Proceedings of the 2004 conference on New interfaces for musical expression, p.205-206, June 03-05, 2004, Hamamatsu, Shizuoka, Japan
• 7: Sam Ferguson, Learning musical instrument skills through interactive sonification, Proceedings of the 2006 conference on New interfaces for musical expression, June 04-08, 2006, Paris, France
• 8: Emmanuel Fléty, The Wise Box: a multi-performer wireless sensor interface using WiFi and OSC, Proceedings of the 2005 conference on New interfaces for musical expression, May 26-28, 2005, Vancouver, Canada
• 9: Guédy, F, "Le traitement du son en pédagogie musicale", L'Inouï, Vol. 2, Ircam - Editions Léo Scheer.
• 10: Puig V., Guédy F., Fingerhut M., Serrière F., Bresson J., Zeller O. "Musique Lab 2: A Three Level Approach for Music Education at School", Proc. of the International Computer Music Conference (ICMC 2005), Barcelona, Spain, 2005.
• 11: Iazzetta, F., "Meaning in Music Gesture", Trends in gestural Control of Music, Marc Battier & Marcelo M. Wanderley (Eds.) Paris-IRCAM - Centre Pompidou - CD-ROM, 2000.
• 12: Kolesnik P. and Wanderley M. M., "Recognition, Analysis and Performance with Expressive Conducting Gestures.", Proc. of the 2004 International Computer Music Conference (ICMC 2004), Miami, USA, 2004.
• 13: Eric Lee , Ingo Grüll , Henning Kiel , Jan Borchers, conga: a framework for adaptive conducting gesture analysis, Proceedings of the 2006 conference on New interfaces for musical expression, June 04-08, 2006, Paris, France
• 14: Eric Lee , Thorsten Karrer , Jan Borchers, Toward a Framework for Interactive Systems to Conduct Digital Audio and Video Streams, Computer Music Journal, v.30 n.1, p.21-36, March 2006 [doi>10.1162/014892606776021317]
• 15: Merrill, D. and Paradiso, J. A Personalization, "Expressivity, and Learnability of an Implicit Mapping Strategy for Physical Interfaces", Proc. of CHI 2005 Conference on Human Factors in Computing Systems, Extended Abstracts, ACM Press, Portland, OR, April 2--7, 2005, pp. 2152--2161.
• 16: Bob Pritchard , Sidney Fels, GRASSP: gesturally-realized audio, speech and song performance, Proceedings of the 2006 conference on New interfaces for musical expression, June 04-08, 2006, Paris, France
• 17: Schnell N., Schwarz, D. "Gabor, Multi-Representation Real-Time Analysis/Synthesis", COST-G6 Conference on Digital Audio Effects (DAFx), Madrid, Spain, 2005.
• 18: Schnell, N., Borghesi, R., Schwarz D., Bevilacqua, F. Müller, R. "FTM --- Complex data structures for Max", International Computer Music Conference (ICMC), Barcelona, Spain, 2005.
• 19: Rabiner L. R., Juang B., Tutorial on hidden Markov models and selected applications in speech recognition, Proc. of the IEEE, vol. 77, no. 2, pp. 257--285, 1989.
• 20: http://ecomote.net/
• 21: http://www.infusionsystems.com/
• 22: http://www.glui.de/
• 23: http://www.i-maestro.org/
• 24: http://www.maxstream.net/
• 25: http://www.opensoundcontrol.org/
• 26: http://www.sparkfun.com/
• 27: http://www.toysymphony.net/
• 28: http://www.xbow.com

INDEX TERMS

Keywords:
gesture recognition, gesture-follower, music pedagogy, technology-enhanced learning, wireless interface