Auditory visual prominence From intelligibility to behavior
2009 (English)In: Journal on Multimodal User Interfaces, ISSN 1783-7677, E-ISSN 1783-8738, Vol. 3, no 4, 299-309 p.Article in journal (Refereed) Published
Auditory prominence is defined as when an acoustic segment is made salient in its context. Prominence is one of the prosodic functions that has been shown to be strongly correlated with facial movements. In this work, we investigate the effects of facial prominence cues, in terms of gestures, when synthesized on animated talking heads. In the first study, a speech intelligibility experiment is conducted, speech quality is acoustically degraded and the fundamental frequency is removed from the signal, then the speech is presented to 12 subjects through a lip synchronized talking head carrying head-nods and eyebrows raise gestures, which are synchronized with the auditory prominence. The experiment shows that presenting prominence as facial gestures significantly increases speech intelligibility compared to when these gestures are randomly added to speech. We also present a follow-up study examining the perception of the behavior of the talking heads when gestures are added over pitch accents. Using eye-gaze tracking technology and questionnaires on 10 moderately hearing impaired subjects, the results of the gaze data show that users look at the face in a similar fashion to when they look at a natural face when gestures are coupled with pitch accents opposed to when the face carries no gestures. From the questionnaires, the results also show that these gestures significantly increase the naturalness and the understanding of the talking head.
Place, publisher, year, edition, pages
2009. Vol. 3, no 4, 299-309 p.
Prominence, Visual prosody, Gesture, ECA, Eye gaze, Head nod, eyebrows
Computer Science Language Technology (Computational Linguistics)
IdentifiersURN: urn:nbn:se:kth:diva-52119DOI: 10.1007/ s12193-010-0054-0ISI: 000208480400005ScopusID: 2-s2.0-78649632880OAI: oai:DiVA.org:kth-52119DiVA: diva2:465414
FunderSwedish Research Council, 2005-3488
QC 201409262011-12-142011-12-142014-09-26Bibliographically approved