Perceiving Systems, Computer Vision

Detecting synchrony in degraded audio-visual streams

2011

Article

ps


Even 8–10 week old infants, when presented with two dynamic faces and a speech stream, look significantly longer at the ‘correct’ talking person (Patterson & Werker, 2003). This is true even though their reduced visual acuity prevents them from utilizing high spatial frequencies. Computational analyses in the field of audio/video synchrony and automatic speaker detection (e.g. Hershey & Movellan, 2000), in contrast, usually depend on high-resolution images. Therefore, the correlation mechanisms found in these computational studies are not directly applicable to the processes through which we learn to integrate the modalities of speech and vision. In this work, we investigated the correlation between speech signals and degraded video signals. We found a high correlation persisting even with high image degradation, resembling the low visual acuity of young infants. Additionally (in a fashion similar to Graf et al., 2002) we explored which parts of the face correlate with the audio in the degraded video sequences. Perfect synchrony and small offsets in the audio were used while finding the correlation, thereby detecting visual events preceding and following audio events. In order to achieve a sufficiently high temporal resolution, high-speed video sequences (500 frames per second) of talking people were used. This is a temporal resolution unachieved in previous studies and has allowed us to capture very subtle and short visual events. We believe that the results of this study might be interesting not only to vision researchers, but, by revealing subtle effects on a very fine timescale, also to people working in computer graphics and the generation and animation of artificial faces.

Author(s): Keshav Dhandhania and Jonas Wulff and Pawan Sinha
Journal: Journal of Vision
Volume: 11
Number (issue): 11
Pages: 800--800
Year: 2011
Month: September
Publisher: ARVO

Department(s): Perceiving Systems
Bibtex Type: Article (article)
Paper Type: Abstract

DOI: doi: 10.1167/11.11.800
URL: http://www.journalofvision.org/content/11/11/800.meeting_abstract

BibTex

@article{Dhandhania:2011:JoV,
  title = {Detecting synchrony in degraded audio-visual streams},
  author = {Dhandhania, Keshav and Wulff, Jonas and Sinha, Pawan},
  journal = {Journal of Vision},
  volume = {11},
  number = {11},
  pages = {800--800},
  publisher = {ARVO},
  month = sep,
  year = {2011},
  doi = {doi: 10.1167/11.11.800},
  url = {http://www.journalofvision.org/content/11/11/800.meeting_abstract},
  month_numeric = {9}
}