Visual analysis of fingering for pedagogical violin transcription

ABSTRACT

Automatic music transcription, in spite of decades of research, remains a challenging research problem. The traditional audio-only approach has yet to achieve a satisfactory performance for any computer-aided pedagogical system. Inspired by the high correlation between violin playing techniques (fingering, bowing) and the played acoustic notes, this paper presents a first attempt in visual analysis of violin fingering to compensate for the difficulties in audio-only music transcription. This is achieved by a robust multiple finger tracking algorithm and a string detection method that extract press, release, and fingertip position from the fingering video and automatically translate the fingering information into the played acoustic note, i.e., onset, offset, and pitches. Experimental results reveal high correctness in multiple finger tracking and string detection, thus paving the way for an improved audio-visual violin transcription system.

REFERENCES

• 1: Chika Oshima , Kazushi Nishimoto , Masami Suzuki, Family ensemble: a collaborative musical edutainment system for children and parents, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027662]
• 2: Jun Yin , Ye Wang , David Hsu, Digital violin tutor: an integrated system for beginning violin learners, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101353]
• 3: Collins N., A comparison of sound onset detection algorithms with emphasis on psycho-acoustically motivated detection functions, Journal of the Audio Engineering Society, 2005.
• 4: Klapuri A., Automatic music transcription as we know it today, Journal of New Music Research, 33(3), pp:269--282, 2004.
• 5: Christian von Hardenberg , François Bérard, Bare-hand human-computer interaction, Proceedings of the 2001 workshop on Perceptive user interfaces, November 15-16, 2001, Orlando, Florida [doi>10.1145/971478.971513]
• 6: Ying Wu , John Lin , Thomas S. Huang, Analyzing and Capturing Articulated Hand Motion in Image Sequences, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.27 n.12, p.1910-1922, December 2005 [doi>10.1109/TPAMI.2005.233]
• 7: Anne-Marie Burns , Marcelo M. Wanderley, Visual methods for the retrieval of guitarist fingering, Proceedings of the 2006 conference on New interfaces for musical expression, June 04-08, 2006, Paris, France
• 8: Michael Isard , Andrew Blake, Contour Tracking by Stochastic Propagation of Conditional Density, Proceedings of the 4th European Conference on Computer Vision-Volume I, p.343-356, April 15-18, 1996
• 9: Blake A. and Isard M., Active contours, Springer, 1998.
• 10: Hough. P. V. C., Method and means for recognizing complex patterns, U.S. Patent, 3.069.654, 1962.
• 11: Molteno T. C. A., Tufillaro N. B., An experimental investigation into the dynamics of a string, American Journal of Physics, 72(9), pp:1157--1169, 2004.
• 12: Bachmann A., An encyclopedia of the violin, Da Capo Press, 2005.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.5 Sound and Music Computing
          Subjects: Systems

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.5 Sound and Music Computing
          Subjects: Methodologies and techniques

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
            Subjects: Shape; Motion; Video analysis
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis
          Subjects: Tracking; Motion

General Terms:
Algorithms, Design, Experimentation, Human Factors

Keywords:
automatic note inference, multiple finger tracking, music transcription, violin fingering chart