Auditory vs. audiovisual prominence ratings of speech involving spontaneously produced head movements
2022 (English)In: Proceedings of the 11th International Conference on Speech Prosody, Speech Prosody 2022, International Speech Communication Association , 2022, p. 352-356Conference paper, Published paper (Refereed)
Abstract [en]
Visual information can be integrated in prominence perception, but most available evidence stems from controlled experimental settings, often involving synthetic stimuli. The present study provides evidence from spontaneously produced head gestures that occurred in Swedish television news readings. Sixteen short clips (containing 218 words in total) were rated for word prominence by 85 adult volunteers in a between-subjects design (44 in an audio-visual vs. 41 in an audio-only condition) using a web-based rating task. As an initial test of overall rating behavior, average prominence across all 218 words was compared between the two conditions, revealing no significant difference. In a second step, we compared normalized prominence ratings between the two conditions for all 218 words individually. These results displayed significant (or near significant, p<.08) differences for 28 out of 218 words, with higher ratings in either the audiovisual (13 words) or the audio-only-condition (15 words). A detailed examination revealed that the presence of head movements (previously annotated) can boost prominence ratings in the audiovisual condition, while words with low prominence tend to be rated slightly higher in the audio-only condition. The study suggests that visual prominence signals are integrated in speech processing even in a relatively uncontrolled, naturalistic setting.
Place, publisher, year, edition, pages
International Speech Communication Association , 2022. p. 352-356
Keywords [en]
beat gesture, head movement, multimodality, pitch accent, prominence perception, visual prosody
National Category
General Language Studies and Linguistics
Identifiers
URN: urn:nbn:se:kth:diva-335753DOI: 10.21437/SpeechProsody.2022-72Scopus ID: 2-s2.0-85147200055OAI: oai:DiVA.org:kth-335753DiVA, id: diva2:1795813
Conference
11th International Conference on Speech Prosody, Speech Prosody 2022, Lisbon, Portugal, May 23 2022 - May 26 2022
Note
QC 20230911
2023-09-112023-09-112023-09-11Bibliographically approved