Multimedia signal processing for behavioral quantification in neuroscience

ABSTRACT

While there have been great advances in quantification of the genotype of organisms, including full genomes for many species, the quantification of phenotype is at a comparatively primitive stage. Part of the reason is technical difficulty: the phenotype covers a wide range of characteristics, ranging from static morphological features, to dynamic behavior. The latter poses challenges that are in the area of multimedia signal processing. Automated analysis of video and audio recordings of animal and human behavior is a growing area of research, ranging from the behavioral phenotyping of genetically modified mice or drosophila to the study of song learning in birds and speech acquisition in human infants. This paper reviews recent advances and identifies key problems for a range of behavior experiments that use audio and video recording. This research area offers both research challenges and an application domain for advanced multimedia signal processing. There are a number of MMSP tools that now exist which are directly relevant for behavioral quantification, such as speech recognition, video analysis and more recently, wired and wireless sensor networks for surveillance. The research challenge is to adapt these tools and to develop new ones required for studying human and animal behavior in a high throughput manner while minimizing human intervention. In contrast with consumer applications, in the research arena there is less of a penalty for computational complexity, so that algorithmic quality can be maximized through the utilization of larger computational resources that are available to the biomedical researcher.

REFERENCES

• 1: Archer, J., Tests for emotionality in rats and mice: a review. Anim Behav, 1973. 21: p. 205--235.
• 2: Benjamini, Y., et al., SEE Software Home Page. 2006.
• 3: Bolivar, V., Cook, M. and Flaherty, L., List of transgenic and knockout mice: behavioral profiles. MAMM Genome, 2000. 11: p. 260--274.
• 4: Chen, S., et al., Fighting fruit flies: A model system for the study of aggression. Proceedings Of The National Academy Of Sciences Of The United States Of America, 2002. 99(8): p. 5664--5668.
• 5: Clark, C.W., Marler, P. and Beeman, K., Quantitative-Analysis of Animal Vocal Phonology - an Application to Swamp Sparrow Song. Ethology, 1987. 76(2): p. 101--115.
• 6: Cleveland, W.S., Robust locally weighted regression and smoothing scatterplots. J Am Stat Assoc, 1977. 74: p. 829--836.
• 7: Crabbe, J.C., Wahlsten, D. and Dudek, B.C., Genetics of mouse behavior: interactions with laboratory environment. Science, 1999. 284: p. 1670--1672.
• 8: Deregnaucourt, S., et al., How sleep affects the developmental learning of bird song. Nature, 2005. 433(7027): p. 710--716.
• 9: Drai, D. and Golani, I., SEE: a tool for the visualization and analysis of rodent exploratory behavior. Neuroscience And Biobehavioral Reviews, 2001. 25(5): p. 409--426.
• 10: Everitt, B.S., Finite Mixture Distributions. 1981, London: Chapman & Hall.
• 11: Fan, J. and Gijbels, I., Local polynomial modeling and its applications. 1996, London: Chapman & Hall.
• 12: Fee, M.S., Shraiman, B., Pesaran, B. and Mitra, P.P., The role of nonlinear dynamics of the syrinx in the vocalizations of a songbird. Nature, 1998. 395(6697): p. 67--71.
• 13: Fry, S.N., Sayaman, R. and Dickinson, M.H., The aerodynamics of hovering flight in Drosophila. Journal Of Experimental Biology, 2005. 208(12): p. 2303--2318.
• 14: Frye, M.A. and Dickinson, M.H., Closing the loop between neurobiology and flight behavior in Drosophila. Current Opinion In Neurobiology, 2004. 14(6): p. 729--736.
• 15: Greenspan, R.J. and Ferveur, J.F., Courtship in Drosophila. Annual Review Of Genetics, 2000. 34: p. 205--232.
• 16: Greenspan, R.J., Tononi, G., Cirelli, C. and Shaw, P.J., Sleep and the fruit fly. Trends In Neurosciences, 2001. 24(3): p. 142--145.
• 17: Hen, I., et al., The dynamics of spatial behavior: how can robust smoothing techniques help? Journal of Neuroscience Methods, 2004. 133: p. 161--172.
• 18: Ho, C.E., Pesaran, B., Fee, M.S. and Mitra, P.P., Characterization of the structure and variability of zebra finch song elements. Proceedings of the Joint Symposium on Neural Computation, 1998. 5: p. 76--83.
• 19: Immelmann, K., Song development in the zebra finch and in other estrilded finches. Bird Vocalization (Ed. by R. A. Hinde), 1969: p. 61--74.
• 20: Iverson, P., et al., A perceptual interference account of acquisition difficulties for non-native phonemes. Cognition, 2003. 87: p. B47--B57.
• 21: Kafkafi, N., et al., Genotype-environment interactions in mouse behavior: A way out of the problem. Proceedings of the National Academy of Sciences, 2005. 102(12): p. 4619--4624.
• 22: Kent, R.D. and Murray, A.D., Acoustic features of infant vocalic utterances at 3, 6 and 9 months. Journal of the Acoustical Society of America, 1982. 72: p. 353--365.
• 23: Knoppien, P., van der Pers, J.N.C. and van Delden, W., Quantification of locomotion and the effect of food deprivation on locomotor activity in Drosophila. Journal of Insect Behavior, 2000. 13(1): p. 27--43.
• 24: Konishi, M., The role of auditory feedback in the control of vocalization in the white-crowned sparrow. Zeitschrift für Tierpsychologie, 1965. 22: p. 770--783.
• 25: Koopmans-Van Beinum, F. and Van Der Stelt, J., Early stages in the development of speech movements, in Precursors of Early Speech, B. Lindblom, e.Z., R., Editor. 1986, Stockton Press: New York. p. 37--49.
• 26: Koopmans-van Beinum, F.J. and van der Stelt, J.M., Early speech development in childs acquiring Dutch mastering general basic elements, in The Acquisition of Dutch, Gillis, S.e.d.H., A., Editor. 1998, Benjamins: Amsterdam/Philadelphia. p. 101--425.
• 27: Kravitz, E.A. and Huber, R., Aggression in invertebrates. Current Opinion In Neurobiology, 2003. 13(6): p. 736--743.
• 28: Kroodsma, D.E., Learning and the ontogeny of sound signals in birds. Acoustic Communication in Birds (Ed. by D. E. Kroodsma & E. H. Miller), 1982: p. 11--23.
• 29: Kuhl, P.K. and Meltzoff, A.N., Infant vocalizations in response to speech: vocal imitation and developmental change. Journal of the Acoustical Society of America, 1996. 100: p. 2425--2438.
• 30: Lipkind, D., et al., New replicable anxiety-related measures of wall vs. center behavior of mice in the open field. Jour. Appl. Physiol., 2004. 97(1): p. 347--359.
• 31: Liu, G., et al., Distinct memory traces for two visual features in the Drosophila brain. Nature, 2006. 439(7076): p. 551--556.
• 32: Margulies, C., Tully, T. and Dubnau, J., Deconstructing memory in Drosophila. Current Biology, 2005. 15(17): p. R700--R713.
• 33: Marler, P. and Tamura, M., Culturally transmitted patterns of vocal behavior in sparrows. Science, 1964. 146: p. 1483-- 1486.
• 34: Martin, J.R., Locomotor activity: a complex behavioural trait to unravel. Behavioural Processes, 2003. 64(2): p. 145--160.
• 35: Martin, J.R., A portrait of locomotor behaviour in Drosophila determined by a video-tracking paradigm. Behavioural Processes, 2004. 67(2): p. 207--219.
• 36: Matyear, C.L., MacNeilage, P.F. and Davis, B.L., Nasalization of vowels in nasal environments in babbling: evidence for frame dominance. Phonetica, 1998. 55: p. 1--17.
• 37: McGuire, S.E., Deshazer, M. and Davis, R.L., Thirty years of olfactory learning and memory research in Drosophila melanogaster. Progress In Neurobiology, 2005. 76(5): p. 328--347.
• 38: Mehren, J.E., Ejima, A. and Griffith, L.C., Unconventional sex: fresh approaches to courtship learning. Current Opinion In Neurobiology, 2004. 14(6): p. 745--750.
• 39: Nottebohm, F. and Nottebohm, M.E., Relationship between Song Repertoire and Age in Canary, Serinus-Canarius. Zeitschrift Fur Tierpsychologie-Journal of Comparative Ethology, 1978. 46(3): p. 298--305.
• 40: Scharff, C. and Nottebohm, F., A Comparative-Study of the Behavioral Deficits Following Lesions of Various Parts of the Zebra Finch Song System - Implications for Vocal Learning. Journal of Neuroscience, 1991. 11(9): p. 2896--2913.
• 41: Serkhane, J.E., Un bébé androïde vocalisant: Etude et modélisation des mécanismes d'exploration vocale et d'imitation orofaciale dans le developpement de la parole. 2005, Institut Polytechnique de Grenoble: Grenoble (France). p. 263.
• 42: Shaw, P.J., Cirelli, C., Greenspan, R.J. and Tononi, G., Correlates of sleep and waking in Drosophila melanogaster. Science, 2000. 287(5459): p. 1834--1837.
• 43: Sokolowski, M.B., Foraging Strategies Of Drosophila-Melanogaster - A Chromosomal Analysis. Behavior Genetics, 1980. 10(3): p. 291--302.
• 44: Tchernichovski, O., et al., Studying the song development process rationale and methods. Behavioral Neurobiology of Birdsong, 2004. 1016: p. 348--363.
• 45: Tchernichovski, O., Mitra, P.P., Lints, T. and Nottebohm, F., Dynamics of the vocal imitation process: How a zebra finch learns its song. Science, 2001. 291(5513): p. 2564--2569.
• 46: Tchernichovski, O. and Nottebohm, F., Social inhibition of song imitation among sibling male zebra finches. Proceedings of the National Academy of Sciences of the United States of America, 1998. 95(15): p. 8951--8956.
• 47: Tchernichovski, O., et al., A procedure for an automated measurement of song similarity. Animal Behaviour, 2000. 59: p. 1167--1176.
• 48: Thomson, D.J., Quadratic-Inverse Spectrum Estimates - Applications to Paleoclimatology. Philosophical Transactions of the Royal Society of London Series a-Mathematical Physical and Engineering Sciences, 1990. 332(1627): p. 539--597.
• 49: Thomson, D.J., Non-stationary fluctuations in stationary time series. Proceedings of the International Society of Optical Engineering, 1993. 2027: p. 236--244.
• 50: Thorpe, W.H., The process of song-learning in the chaffinch as studied by means of the sound spectrograph. Nature, 1954. 173: p. 465.
• 51: Thorpe, W.H., The learning of song patterns by birds, with special reference to the song of the chaffinch, Fringella coelebs. Nature, 1958. 100: p. 535--570.
• 52: van der Stelt, J.M. and Koopmans-van Beinum, F.J., The onset of babbling related to gross motor development, in Precursors of Early Speech, B. Lindblom, e.Z., R., Editor. 1986, Stockton Press: New York. p. 163--173.
• 53: van der Stelt, J.M., Wempe, T.G. and Pols, L.C.W. Progression in vowel production: comparing deaf and hearing children. in Proceedings. 2003. Amsterdam (Netherlands): University of Amsterdam.
• 54: van Swinderen, B., The remote roots of consciousness in fruit-fly selective attention? Bioessays, 2005. 27(3): p. 321--330.
• 55: van Swinderen, B. and Andretic, R., Arousal in Drosophila. Behavioural Processes, 2003. 64(2): p. 133--144.

INDEX TERMS

Primary Classification:
  J. Computer Applications
  J.3 LIFE AND MEDICAL SCIENCES

Additional Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.0 General

  J. Computer Applications
  J.4 SOCIAL AND BEHAVIORAL SCIENCES

General Terms:
Algorithms, Experimentation, Measurement

Keywords:
audio, behavior, birdsong, human, infant, locomotion, mouse, multimedia, neuroscience, phenotype, signal processing, video, vocal development, zebra finch