Sound/tracks: real-time synaesthetic sonification and visualisation of passing landscapes

ABSTRACT

When travelling on a train, many people enjoy looking out of the window at the landscape passing by. We present sound/tracks, an application that translates the perceived movement of the scenery and other visual impressions, such as passing trains, into music. The continuously changing view outside the window is captured with a camera and translated into MIDI events that are replayed instantaneously. This allows for a reflection of the visual impression, adding a sound dimension to the visual experience and deepening the state of contemplation. The application is intended to be run on both mobile phones (with built-in camera) and on laptops (with a connected Web-cam). We propose and discuss different approaches to translating the video signal into an audio stream, present different application scenarios, and introduce a method to visualise the dynamics of complete train journeys by "re-transcribing" the captured video frames used to generate the music.

REFERENCES

• 1: W. G. Collier and T. L. Hubbard. Musical scales and brightness evaluations: Effects of pitch, direction, and scale mode. Musicae Scientiae, 8:151--173 2004.
• 2: Darcee L. Datteri and Jeffrey N. Howard. The sound of color. In Proceedings of the 8th International Conference on Music Perception & Cognition (ICMPC8), Evanston, IL, USA, August 3-7 2004.
• 3: Michel Gondry. Music video for "The Chemical Brothers - Star Guitar", 2002.
• 4: Lauri Gröhn. Sound of Paintings. URL: http://www.synestesia.fi/
• 5: Kazuhiro Jo and Norihisa Nagano. Monalisa: "See the sound, hear the image". In Proceedings of the 8th International Conference New Interface for Musical Expression. NIME '08, pages 315--318, Genova, Italy, June 5-7 2008.
• 6: C. L. Krumhansl and E. J. Kessler. Tracing the dynamic changes in perceived tonal organization in a spatial representation of musical keys. Psychological Review, 89:334--368, 1982.
• 7: L. E. Marks. On associations of light and sound: The mediation of brightness, pitch, and loudness. American Journal of Psychology, 87 (1-2):173--188, 1974.
• 8: Daphne Maurer, Thanujeni Pathman, and Catherine J. Mondloch. The shape of boubas: sound-shape correspondences in toddlers and adults. Developmental Science, 9:3:316--322, 2006.
• 9: H. Purwins. Profiles of Pitch Classes Circularity of Relative Pitch and Key - Experiments, Models, Computational Music Analysis, and Perspectives. PhD thesis, TU Berlin, 2005.
• 10: Martin Reinhart and Virgil Widrich. tx-transform. URL: http://www.tx-transform.com/
• 11: Elena Rusconi, Bonnie Kwan, Bruno L. Giordano, Carlo Umiltà, and Brian Butterworth. Spatial representation of pitch height: the SMARC effect. Cognition, 99:2:113--129, 2006.
• 12: Atau Tanaka. Bondage. URL: http://www.xmira.com/atau/bondage/, 2004.
• 13: Wikipedia. Alexander Scriabin. URL: http://en.wikipedia.org/wiki/Alexander_Scriabin

INDEX TERMS

Primary Classification:
  J. Computer Applications
  J.5 ARTS AND HUMANITIES
      Subjects: Performing arts (e.g., dance, music)

General Terms:
Algorithms

Keywords:
mobile music generation, sonification, train journey