kth.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Acoustic-to-Articulatory Mapping With Joint Optimization of Deep Speech Enhancement and Articulatory Inversion Models
Norwegian Univ Sci & Technol, Signal Proc, N-7491 Trondheim, Norway..
KTH, School of Electrical Engineering and Computer Science (EECS), Intelligent systems, Speech, Music and Hearing, TMH. Norwegian Univ Sci & Technol, Signal Proc, N-7491 Trondheim, Norway.ORCID iD: 0000-0002-3323-5311
Norwegian Univ Sci & Technol, Signal Proc, N-7491 Trondheim, Norway..
Norwegian Univ Sci & Technol, Signal Proc, N-7491 Trondheim, Norway..
2022 (English)In: IEEE/ACM transactions on audio, speech, and language processing, ISSN 2329-9290, Vol. 30, p. 135-147Article in journal (Refereed) Published
Abstract [en]

We investigate the problem of speaker independent acoustic-to-articulatory inversion (AAI) in noisy conditions within the deep neural network (DNN) framework. In contrast with recent results in the literature, we argue that a DNN vector-to-vector regression front-end for speech enhancement (DNN-SE) can play a key role in AAI when used to enhance spectral features prior to AAI back-end processing. We experimented with single- and multi-task training strategies for the DNN-SE block finding the latter to be beneficial to AAI. Furthermore, we show that coupling DNN-SE producing enhanced speech features with an AAI trained on clean speech outperforms a multi-condition AAI (AAI-MC) when tested on noisy speech. We observe a 15% relative improvement in the Pearson's correlation coefficient (PCC) between our system and AAI-MC at 0 dB signal-to-noise ratio on the Haskins corpus. Our approach also compares favourably against using a conventional DSP approach to speech enhancement (MMSE with IMCRA) in the front-end. Finally, we demonstrate the utility of articulatory inversion in a downstream speech application. We report significant WER improvements on an automatic speech recognition task in mismatched conditions based on the Wall Street Journal corpus (WSJ) when leveraging articulatory information estimated by AAI-MC system over spectral-alone speech features.

Place, publisher, year, edition, pages
Institute of Electrical and Electronics Engineers (IEEE) , 2022. Vol. 30, p. 135-147
Keywords [en]
Noise measurement, Speech enhancement, Task analysis, Mel frequency cepstral coefficient, Training, Hidden Markov models, Deep learning, Deep neural network, acoustic-to-articulatory inversion, multi-task training, speaker independent models
National Category
Natural Language Processing
Identifiers
URN: urn:nbn:se:kth:diva-307335DOI: 10.1109/TASLP.2021.3133218ISI: 000735507400007Scopus ID: 2-s2.0-85121342065OAI: oai:DiVA.org:kth-307335DiVA, id: diva2:1631339
Note

QC 20220124

Available from: 2022-01-24 Created: 2022-01-24 Last updated: 2025-02-07Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopus

Authority records

Salvi, Giampiero

Search in DiVA

By author/editor
Salvi, Giampiero
By organisation
Speech, Music and Hearing, TMH
Natural Language Processing

Search outside of DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 138 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf