A system for rule based audiovisual text-to-speech synthesishas been created. The system is based on the KTHtext-to-speech system which has been complementedwith a three-dimensional parameterized model of a humanface. The face can be animated in real time, synchronizedwith the auditory speech. The facial model iscontrolled by the same synthesis software as the auditoryspeech synthesizer. A set of rules that takes coarticulationinto account has been developed. The audiovisualtext-to-speech system has also been incorporated into aspoken man-machine dialogue system that is being developedat the department.

It is envisioned that autonomous software agents that cancommunicate using speech and gesture will soon be oneverybody’s computer screen. This paper describes anarchitecture that can be used to design and animate characterscapable of lip-synchronised synthetic speech as well as bodygestures, for use in for example spoken dialogue systems. Ageneral scheme for computationally efficient parametricdeformation of facial surfaces is presented, as well as techniques for generation of bimodal speech, facial expressionsand body gestures in a spoken dialogue system. Resultsindicating that an animated cartoon-like character can be asignificant contribution to speech intelligibility, are also reported.

We report on our recent facial animation work to improve the realism and accuracy of visual speech synthesis. The general approach is to use both staticand dynamic observations of natural speech to guidethe facial modeling. One current goal is to model the internal articulators of a highly realistic palate, teeth, and an improved tongue. Because our talkinghead can be made transparent, we can provide ananatomically valid and pedagogically useful displaythat can be used in speech training of children withhearing loss [1]. High-resolution models of palateand teeth [2] were reduced to a relatively smallnumber of polygons for real-time animation [3]. Forthe improved tongue, we are using 3D ultrasound data and electropalatography (EPG) [4] with errorminimization algorithms to educate our parametricB-spline based tongue model to simulate realisticspeech. In addition, a high-speed algorithm has beendeveloped for detection and correction of collisions, to prevent the tongue from protruding through the palate and teeth, and to enable the real-time displayof synthetic EPG patterns.

This paper presents an initial implementation and evaluation  of  a  system  that  synthesizes  visualspeech  directly  from  the  acoustic waveform. Anartificial  neural  network  (ANN)  was  trained  tomap  the  cepstral  coefficients  of  an  individual’snatural  speech  to  the  control  parameters  of  ananimated  synthetic  talking  head. We  trained  ontwo data sets; one was a set of 400 words spokenin  isolation  by  a  single  speaker  and  the  other  a subset  of  extemporaneous  speech  from  10different speakers. The system showed learning inboth cases. A perceptual evaluation test indicatedthat the system’s generalization  to new words bythe  same  speaker  provides  significant  visible information, but significantly below that given bya text-to-speech algorithm.

We present a formalism for specifying verbal and non-verbal output from a multimodal dialogue system. The output specification is XML-based and provides information about communicative functions of the output, without detailing the realisation of these functions. The aim is to let dialogue systems generate the same output for a wide variety of output devices and modalities. The formalism was developed and implemented in the multimodal spoken dialogue system AdApt. We also describe how facial gestures in the 3D-animated talking head used within this system are controlled through the formalism.

Simultaneous measurements of tongue and facial motion,using a combination of electromagnetic articulography(EMA) and optical motion tracking, are analysed to improvethe articulation of an animated talking head and toinvestigate the correlation between facial and vocal tractmovement. The recorded material consists of VCV andCVC words and 270 short everyday sentences spoken byone Swedish subject. The recorded articulatory movementsare re-synthesised by a parametrically controlled 3D modelof the face and tongue, using a procedure involvingminimisation of the error between measurement and model.Using linear estimators, tongue data is predicted from theface and vice versa, and the correlation betweenmeasurement and prediction is computed.

This paper deals with the problem of modelling the dynamics of articulation for a parameterised talkinghead based on phonetic input. Four different models are implemented and trained to reproduce the articulatorypatterns of a real speaker, based on a corpus of optical measurements. Two of the models, (“Cohen-Massaro”and “O¨ hman”) are based on coarticulation models from speech production theory and two are based on artificialneural networks, one of which is specially intended for streaming real-time applications. The different models areevaluated through comparison between predicted and measured trajectories, which shows that the Cohen-Massaromodel produces trajectories that best matches the measurements. A perceptual intelligibility experiment is alsocarried out, where the four data-driven models are compared against a rule-based model as well as an audio-alonecondition. Results show that all models give significantly increased speech intelligibility over the audio-alone case,with the rule-based model yielding highest intelligibility score.