Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
ANIMATION OF TALKING AGENTS
KTH, Superseded Departments, Speech, Music and Hearing.ORCID iD: 0000-0003-1399-6604
1997 (English)In: Proceedings of International Conference on Auditory-Visual Speech Processing / [ed] Benoït, C & Campbell, R, Rhodos, Greece, 1997, 149-152 p.Conference paper, Published paper (Other academic)
Abstract [en]

It is envisioned that autonomous software agents that cancommunicate using speech and gesture will soon be oneverybody’s computer screen. This paper describes anarchitecture that can be used to design and animate characterscapable of lip-synchronised synthetic speech as well as bodygestures, for use in for example spoken dialogue systems. Ageneral scheme for computationally efficient parametricdeformation of facial surfaces is presented, as well as techniques for generation of bimodal speech, facial expressionsand body gestures in a spoken dialogue system. Resultsindicating that an animated cartoon-like character can be asignificant contribution to speech intelligibility, are also reported.

Place, publisher, year, edition, pages
Rhodos, Greece, 1997. 149-152 p.
National Category
Natural Sciences
Identifiers
URN: urn:nbn:se:kth:diva-12709OAI: oai:DiVA.org:kth-12709DiVA: diva2:318253
Conference
International Conference on Auditory-Visual Speech Processing
Note
QC 20100507Available from: 2010-05-07 Created: 2010-05-07 Last updated: 2010-05-11Bibliographically approved
In thesis
1. Talking Heads - Models and Applications for Multimodal Speech Synthesis
Open this publication in new window or tab >>Talking Heads - Models and Applications for Multimodal Speech Synthesis
2003 (English)Doctoral thesis, comprehensive summary (Other scientific)
Abstract [en]

This thesis presents work in the area of computer-animatedtalking heads. A system for multimodal speech synthesis hasbeen developed, capable of generating audiovisual speechanimations from arbitrary text, using parametrically controlled3D models of the face and head. A speech-specific directparameterisation of the movement of the visible articulators(lips, tongue and jaw) is suggested, along with a flexiblescheme for parameterising facial surface deformations based onwell-defined articulatory targets.

To improve the realism and validity of facial and intra-oralspeech movements, measurements from real speakers have beenincorporated from several types of static and dynamic datasources. These include ultrasound measurements of tonguesurface shape, dynamic optical motion tracking of face pointsin 3D, as well as electromagnetic articulography (EMA)providing dynamic tongue movement data in 2D. Ultrasound dataare used to estimate target configurations for a complex tonguemodel for a number of sustained articulations. Simultaneousoptical and electromagnetic measurements are performed and thedata are used to resynthesise facial and intra-oralarticulation in the model. A robust resynthesis procedure,capable of animating facial geometries that differ in shapefrom the measured subject, is described.

To drive articulation from symbolic (phonetic) input, forexample in the context of a text-to-speech system, bothrule-based and data-driven articulatory control models havebeen developed. The rule-based model effectively handlesforward and backward coarticulation by targetunder-specification, while the data-driven model uses ANNs toestimate articulatory parameter trajectories, trained ontrajectories resynthesised from optical measurements. Thearticulatory control models are evaluated and compared againstother data-driven models trained on the same data. Experimentswith ANNs for driving the articulation of a talking headdirectly from acoustic speech input are also reported.

A flexible strategy for generation of non-verbal facialgestures is presented. It is based on a gesture libraryorganised by communicative function, where each function hasmultiple alternative realisations. The gestures can be used tosignal e.g. turn-taking, back-channelling and prominence whenthe talking head is employed as output channel in a spokendialogue system. A device independent XML-based formalism fornon-verbal and verbal output in multimodal dialogue systems isproposed, and it is described how the output specification isinterpreted in the context of a talking head and converted intofacial animation using the gesture library.

Through a series of audiovisual perceptual experiments withnoise-degraded audio, it is demonstrated that the animatedtalking head provides significantly increased intelligibilityover the audio-only case, in some cases not significantly belowthat provided by a natural face.

Finally, several projects and applications are presented,where the described talking head technology has beensuccessfully employed. Four different multimodal spokendialogue systems are outlined, and the role of the talkingheads in each of the systems is discussed. A telecommunicationapplication where the talking head functions as an aid forhearing-impaired users is also described, as well as a speechtraining application where talking heads and languagetechnology are used with the purpose of improving speechproduction in profoundly deaf children.

Place, publisher, year, edition, pages
Institutionen för talöverföring och musikakustik, 2003. viii, 63 p.
Series
Trita-TMH, 2003:7
Keyword
Talking heads, facial animation, speech synthesis, coarticulation, intelligibility, embodied conversational agents
Identifiers
urn:nbn:se:kth:diva-3561 (URN)91-7283-536-2 (ISBN)
Public defence
2003-06-11, 00:00 (English)
Note
QC 20100506Available from: 2003-06-26 Created: 2003-06-26 Last updated: 2010-05-11Bibliographically approved

Open Access in DiVA

No full text

Authority records BETA

Beskow, Jonas

Search in DiVA

By author/editor
Beskow, Jonas
By organisation
Speech, Music and Hearing
Natural Sciences

Search outside of DiVA

GoogleGoogle Scholar

urn-nbn

Altmetric score

urn-nbn
Total: 161 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf