kth.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Integrated Speech and Gesture Synthesis
KTH, School of Electrical Engineering and Computer Science (EECS), Intelligent systems, Speech, Music and Hearing, TMH.
KTH, School of Electrical Engineering and Computer Science (EECS), Intelligent systems, Speech, Music and Hearing, TMH.ORCID iD: 0000-0002-7801-7617
KTH, School of Electrical Engineering and Computer Science (EECS), Intelligent systems, Speech, Music and Hearing, TMH.ORCID iD: 0000-0002-0397-6442
KTH, School of Electrical Engineering and Computer Science (EECS), Intelligent systems, Speech, Music and Hearing, TMH.ORCID iD: 0000-0003-1399-6604
Show others and affiliations
2021 (English)In: ICMI 2021 - Proceedings of the 2021 International Conference on Multimodal Interaction, Association for Computing Machinery (ACM) , 2021, p. 177-185Conference paper, Published paper (Refereed)
Abstract [en]

Text-to-speech and co-speech gesture synthesis have until now been treated as separate areas by two different research communities, and applications merely stack the two technologies using a simple system-level pipeline. This can lead to modeling inefficiencies and may introduce inconsistencies that limit the achievable naturalness. We propose to instead synthesize the two modalities in a single model, a new problem we call integrated speech and gesture synthesis (ISG). We also propose a set of models modified from state-of-the-art neural speech-synthesis engines to achieve this goal. We evaluate the models in three carefully-designed user studies, two of which evaluate the synthesized speech and gesture in isolation, plus a combined study that evaluates the models like they will be used in real-world applications - speech and gesture presented together. The results show that participants rate one of the proposed integrated synthesis models as being as good as the state-of-the-art pipeline system we compare against, in all three tests. The model is able to achieve this with faster synthesis time and greatly reduced parameter count compared to the pipeline system, illustrating some of the potential benefits of treating speech and gesture synthesis together as a single, unified problem.

Place, publisher, year, edition, pages
Association for Computing Machinery (ACM) , 2021. p. 177-185
Keywords [en]
gesture generation, neural networks, speech synthesis, Piping systems, Neural-networks, Pipeline systems, Research applications, Research communities, Simple system, Single models, State of the art, System level pipelines, Text to speech, Water pipelines
National Category
Computer Sciences
Identifiers
URN: urn:nbn:se:kth:diva-313183DOI: 10.1145/3462244.3479914Scopus ID: 2-s2.0-85118992736OAI: oai:DiVA.org:kth-313183DiVA, id: diva2:1663462
Conference
ICMI '21: International Conference on Multimodal Interaction, Montréal, QC, Canada, October 18-22, 2021
Note

Part of proceedings ISBN 9781450384810

QC 20220602

Available from: 2022-06-02 Created: 2022-06-02 Last updated: 2022-06-25Bibliographically approved

Open Access in DiVA

fulltext(1680 kB)97 downloads
File information
File name FULLTEXT01.pdfFile size 1680 kBChecksum SHA-512
975d47d3ee53cc452d48d8298406eab73bdbe320718468cf3eb9a0ecca57bcb200b0d937a7ba3a6b3a7de8bf0e152a035e7f66415d5744098ca5bf086d62046c
Type fulltextMimetype application/pdf

Other links

Publisher's full textScopus

Authority records

Wang, SiyangAlexanderson, SimonGustafsson, JoakimBeskow, JonasHenter, Gustav EjeSzékely, Éva

Search in DiVA

By author/editor
Wang, SiyangAlexanderson, SimonGustafsson, JoakimBeskow, JonasHenter, Gustav EjeSzékely, Éva
By organisation
Speech, Music and Hearing, TMH
Computer Sciences

Search outside of DiVA

GoogleGoogle Scholar
Total: 97 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 145 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf