Changing timbre and phrase in existing musical performances as you like: manipulations of single part using harmonic and inharmonic models

ABSTRACT

This paper presents a new music manipulation method that can change the timbre and phrases of an existing instrumental performance in a polyphonic sound mixture. This method consists of three primitive functions: 1) extracting and analyzing of a single instrumental part from polyphonic music signals, 2) mixing the instrument timbre with another, and 3) rendering a new phrase expression for another given score. The resulting customized part is re-mixed with the remaining parts of the original performance to generate new polyphonic music signals. A single instrumental part is extracted by using an integrated tone model that consists of harmonic and inharmonic tone models with the aid of the score of the single instrumental part. The extraction incorporates a residual model for the single instrumental part in order to avoid crosstalk between instrumental parts. The extracted model parameters are classified into their averages and deviations. The former is treated as instrument timbre and is customized by mixing, while the latter is treated as phrase expression and is customized by rendering. We evaluated our method in three experiments. The first experiment focused on introduction of the residual model, and it showed that the model parameters are estimated more accurately by 35.0 points. The second focused on timbral customization, and it showed that our method is more robust by 42.9 points in spectral distance compared with a conventional sound analysis-synthesis method, STRAIGHT. The third focused on the acoustic fidelity of customizing performance, and it showed that rendering phrase expression according to the note sequence leads to more accurate performance by 9.2 points in spectral distance in comparison with a rendering method that ignores the note sequence.

REFERENCES

• 1: K. Yoshii, M.Goto, K.Komatani, T. Ogata, and H. G. Okuno. Drumix: An audio player with real-time drum-part rearrangement functions for active music listening. The Journal of Information Processing Society of Japan, 48(3):1229--1239, 2007.
• 2: K. Itoyama, M. Goto, K. Komatani, T. Ogata, and H. G. Okuno. Integration and adaptation of harmonic and inharmonic models for separating polyphonic musical signals. In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing, pages 57--60, 2007.
• 3: E. Lindemann. Music synthesis with reconstructive phrase modeling. Signal Processing Magazine, IEEE, 24(2):80--91, March 2007.
• 4: A.P. Klapuri. Multiple fundamental frequency estimation based on harmonicity and spectral smoothness. IEEE Transactions on Speech and Audio Processing, 11(6):804--816, Nov. 2003.
• 5: T. Kitahara, M. Goto, K. Komatani, T. Ogata, and H.G. Okuno. Instrogram: A new musical instrument recognition technique without using onset detection nor f0 estimation. In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 5, pages 229--232, May 2006.
• 6: M. Goto and Y. Muraoka. Beat tracking based on multiple-agent architecture a real-time beat tracking system for audio signals. In In Proc. Second International Conference on Multiagent Systems, pages 103--110, 1996.
• 7: A. Eronen. Comparison of features for musical instrument recognition. In Proc. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, pages 19--22, 2001.
• 8: H. Kawahara. STRAIGHT, exploration of the other aspect of vocoder: Perceptually isomorphic decomposition of speech sounds. Acoustic Science and Technology, 27(6):349--353, 2006.
• 9: M. Slaney, M. Covell, and B. Lassiter. Automatic audio morphing. In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing, pages 1001--1004, 1996.
• 10: H. Kameoka, T. Nishimoto, and S. Sagayama. A multipitch analyzer based on harmonic temporal structured clustering. IEEE Transactions on Audio, Speech and Language Processing, 15(3):982--994, 2007.
• 11: E. Tellman, L. Haken, and B. Holloway. Timbre morphing of sounds with unequal number of features. J. Audio Eng. Soc., 43(9):678--689, 1995.
• 12: R.J. McAulay and T.F. Quatieri. Pitch estimation and voicing d etection based on a sinusoidal speech model. In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing, pages 249--252 vol.1, Apr 1990.
• 13: G. Widmer. Modeling the rational basis of musical expression. Computer Music Journal, 19(2):76--96, 1995.
• 14: T. Suzuki. A case based approach to the generation of musical expression. In Proc. International Joint Conferences on Artificial Intelligence, pages 642--648, 1999.
• 15: J. L. Arcos, R. L. de Mántaras, and X. Serra. Saxex : a case-based reasoning system for generating expressive musical performances. In Proc. International Computer Music Conference, pages 329--336, 1997.
• 16: S. Canazza, G. De Poli, C. Drioli, A. Roda, and A. Vidolin. Modeling and control of expressiveness in music performance. Proceedings of the IEEE, 92(4):686--701, Apr 2004.
• 17: M. Casey and A. Westner. Separation of mixed audio sources by independent subspace analysis. In Proc. International Computer Music Conference, pages 154--161, 2000.
• 18: H. Fletcher, E. Blackham, and R. Stratton. Quality of piano. tones. The Journal of the Acoustical Society of America, 34(6):749--761, 1962.
• 19: N. H. Fletcher and T. D. Rossing. The Physics of Musical Instruments. Springer, second edition, 1997.
• 20: T. Takahashi, H. Kawahara, and T. Irino. Evaluation of iterative analysis-by-synthesis speech sounds using STRAIGHT. In Proc. of Autumn Meeting of Acoust. Soc. Japan, pages 289--290, 2007. (in Japanese).
• 21: J. M. Grey. Multidimensional perceptual scaling of musical timbres. The Journal of the Acoustical Society of America, 61(5):1270--1277, 1977.
• 22: J. Marozeau, A. Cheveigne, S. McAdams, and S. Winsberg. The dependency of timbre on fundamental frequency. The Journal of the Acoustical Society of America, 114(5):2946--2957, 2003.
• 23: T. Abe, K. Itoyama, K. Yoshii, K. Komatani, T. Ogata, and H. G. Okuno. Analysis-and-manipulation approach to pitch and duration of musical instrument sounds without distroting timbral characteristics. In Proc. Digital Audio Effects, pages 249--256, 2008.
• 24: R. McAulay and T. Quatieri. Speech analysis/synthesis based on a sinusoidal representation. IEEE Transactions on Acoustics, Speech, & Signal Processing, 34(4):744--754, 1986.
• 25: M. Portnoff. Implementation of the digital phase vocoder using the fast fourier transform. IEEE Transactions on Acoustics, Speech, & Signal Processing, 24(3):243--248, 1976.
• 26: M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka. RWC music database: Popular, classical, and jazz music databases. In Proc. International Symposium on Music Information Retrieval, pages 287--288, October 2002.
• 27: M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka. RWC music database: Music genre database and musical instrument sound database. In Proc. International Symposium on Music Information Retrieval, pages 229--230, October 2003.
• 28: R. D. Patterson. Auditory filter shapes derived with noise stimuli. The Journal of the Acoustical Society of America, 59(3):640--654, 1976.
• 29: T. Yoshioka, T. Nakatani, and M. Miyoshi. Integrated speech enhancement method using noise suppression and dereverberation. IEEE Transactions on Audio, Speech and Language Processing, 17(2):231--246, Feb. 2009.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.5 Sound and Music Computing
          Subjects: Signal analysis, synthesis, and processing

General Terms:
Algorithms, Design

Keywords:
music manipulation, performance rendering, signal processing, sound source extraction, timbre mixing