Animations produced by generative models are often evaluated using objective quantitative metrics that do not fully capture perceptual effects in immersive virtual environments. To address this gap, we present a preliminary perceptual evaluation of generative models for animation synthesis, conducted via a VR-based user study (N = 48). Our investigation specifically focuses on animation congruency—ensuring that generated facial expressions and body gestures are both congruent with and synchronized to driving speech. We evaluated two state-of-the-art methods: a speech-driven full-body animation model and a video-driven full-body reconstruction model, assessing their capability to produce congruent facial expressions and body gestures. Our results demonstrate a strong user preference for combined facial and body animations, highlighting that congruent multimodal animations significantly enhance perceived realism compared to animations featuring only a single modality. By incorporating VR-based perceptual feedback into training pipelines, our approach provides a foundation for developing more engaging and responsive virtual characters.

Introduction: Social interactions incorporate various nonverbal signals to convey emotions alongside speech, including facial expressions and body gestures. Generative models have demonstrated promising results in creating full-body nonverbal animations synchronized with speech; however, evaluations using statistical metrics in 2D settings fail to fully capture user-perceived emotions, limiting our understanding of the effectiveness of these models. Methods: To address this, we evaluate emotional 3D animation generative models within an immersive Virtual Reality (VR) environment, emphasizing user—centric metrics-emotional arousal realism, naturalness, enjoyment, diversity, and interaction quality—in a real-time human-agent interaction scenario. Through a user study (N = 48), we systematically examine perceived emotional quality for three state-of-the-art speech-driven 3D animation methods across two specific emotions: happiness (high arousal) and neutral (mid arousal). Additionally, we compare these generative models against real human expressions obtained via a reconstruction-based method to assess both their strengths and limitations and how closely they replicate real human facial and body expressions. Results: Our results demonstrate that methods explicitly modeling emotions lead to higher recognition accuracy compared to those focusing solely on speech-driven synchrony. Users rated the realism and naturalness of happy animations significantly higher than those of neutral animations, highlighting the limitations of current generative models in handling subtle emotional states. Discussion: Generative models underperformed compared to reconstruction-based methods in facial expression quality, and all methods received relatively low ratings for animation enjoyment and interaction quality, emphasizing the importance of incorporating user-centric evaluations into generative model development. Finally, participants positively recognized animation diversity across all generative models.

Embodied social agents are entities, either physical or digital, that can communicate with humans or even other agents, often expressing ideas, thoughts, or emotions. In this study, participants were given the task of collecting a coffee cup standing on a table behind a free-standing conversational group of two embodied virtual agents in a Virtual Reality (VR) environment. Participants were presented with a social dilemma, since they had to choose whether to walk between the agents, violating their o-space, or through a narrower gap between the agents and the table. In each of the six conditions the agents expressed different interpersonal attitudes defined along Friendly/Hostile and Submissive/Dominant axes, shown through different verbal scripts and non-verbal behaviours. The 32 participants in this within-group user study walked around the agent group in 62.9% of the trials. In the four conditions where the participants were asked by one of the agents to wait, they were significantly more likely to walk around the agents rather than between them. Of those conditions, participants violated the agents’ o-space the most in the conditions in which agents’ attitudes were perceived to be the most Hostile and Dominant. Participants liked the Friendly agents the most and the Hostile agents the least. We discuss these findings in addition to their implications for the design of socially interactive agents.

This cross-cultural study (N=108) examines how cultural differences between Japan and Sweden influence participants social behaviors and perceptions when joining a free-standing group of two agents. Agents within the group, embodied as humans, robots, and virtual characters, respectively, use three distinct behaviors, varying with respect to politeness strategy, to request the participant to join on a specific side and position in the group. The experimental results showed that Japanese participants, from a culture characterized by higher power distance, masculinity, uncertainty avoidance, long-term orientation, restraint, and collectivism, were more likely to comply with the agent's request regarding the joining position, compared to Swedish participants. This trend was even more pronounced when comparing different types of embodiment: Japanese participants more strictly complied with human agents than with non-human agents. Additionally, Japanese females and Swedish males adhered more to social norms by avoiding walking between group members (i.e. through the group's o-space) when joining. Second, cultural differences also significantly impacted the perception of agents' politeness behaviors, while the effect of embodiment on feelings of friendliness and closeness varied depending on the culture. We reflect on our results as a basis for highlighting key challenges involved in the design of culturally adapted agents and their behaviors toward enhancing the localization of human-agent interaction.

Human footsteps play a significant role in everyday life, allowing individuals to discern the emotions, gender, and intentions of others solely from the sound of their footsteps. However, the influence of footstep sounds made when walking on different floor types in virtual reality (VR) environments when joining conversational groups remains unclear. In this paper, we present a controlled study (N=50) to assess the impact of five different floor types, associated with specific footstep sounds and visuals, on the persuasiveness of Embodied Conversational Agents (ECAs) when inviting participants to join a free-standing conversational group. We analyze routes taken by participants and the positions at which they join the group, which may be compliant or not with the agent’s request when approaching the group while walking on different virtual floor types. Our findings reveal that the type of floor being walked upon, defined by footstep sounds and visual appearance, significantly impacts the persuasiveness of ECAs and the trajectories taken by participants to join the group. Participants took longer paths and joined the group in the presence of more pleasant footstep sounds. Further, they tended to adhere to social norms by avoiding walking through the group’s center.

Existing methods for synthesizing 3D human gestures from speech have shown promising results but they do not explicitly model the impact of emotions on the generated gestures. Instead these methods directly output animations from speech without control over the expressed emotion. To address this limitation we present AMUSE an emotional speech-driven body animation model based on latent diffusion. Our observation is that content (i.e. gestures related to speech rhythm and word utterances) emotion and personal style are separable. To account for this AMUSE maps the driving audio to three disentangled latent vectors: one for content one for emotion and one for personal style. A latent diffusion model trained to generate gesture motion sequences is then conditioned on these latent vectors. Once trained AMUSE synthesizes 3D human gestures directly from speech with control over the expressed emotions and style by combining the content from the driving speech with the emotion and style of another speech sequence. Randomly sampling the noise of the diffusion model further generates variations of the gesture with the same emotional expressivity. Qualitative quantitative and perceptual evaluations demonstrate that AMUSE outputs realistic gesture sequences. Compared to the state of the art the generated gestures are better synchronized with the speech content and better represent the emotion expressed by the input speech. Our code is available at amuse.is.tue.mpg.de.

Politeness is a crucial aspect of human social interactions. While the infuence of politeness is well understood in human groups, it remains underexplored in group interactions with robots. Therefore, in this paper, we conduct an initial exploration into the infuence of the presence of humanoid robots on their persuasiveness and perceived politeness in small groups. We conducted a user study (N = 119) with co-present and remote robots that invited participants to join the group using six politeness behaviors derived from Brown and Levinson’s politeness theory. It requests participants to join them at the furthest side of the group, even though a closer side is also available to them, but would ignore the robot’s request. The results show that co-present robots are perceived to be less persuasive than remote ones. However, co-presence enhances the clarity of the robot’s requests and the perceived freedom of action while decreasing the perceived friendliness and ofensiveness. 

Creativity is an important skill that is known to plummet in children when they start school education that limits their freedom of expression and their imagination. On the other hand, research has shown that integrating social robots into educational settings has the potential to maximize children’s learning outcomes. Therefore, our aim in this work was to investigate stimulating children’s creativity through child-robot interactions. We fine-tuned a Large Language Model (LLM) to exhibit creative behavior and non-creative behavior in a robot and conducted two studies with children to evaluate the viability of our methods in fostering children’s creativity skills. We evaluated creativity in terms of four metrics: fluency, flexibility, elaboration, and originality. We first conducted a study as a storytelling interaction between a child and a wizard-ed social robot in one of two conditions: creative versus non-creative with 38 children. We investigated whether interacting with a creative social robot will elicit more creativity from children. However, we did not find a significant effect of the robot’s creativity on children’s creative abilities. Second, in an attempt to increase the possibility for the robot to have an impact on children’s creativity and to increase the fluidity of the interaction, we produced two models that allow a social agent to autonomously engage with a human in a storytelling context in a creative manner and a non-creative manner respectively. Finally, we conducted another study to evaluate our models by deploying them on a social robot and evaluating them with 103 children. Our results show that children who interacted with the creative autonomous robot were more creative than children who interacted with the non-creative autonomous robot in terms of the fluency, the flexibility, and the elaboration aspects of creativity. The results highlight the difference in children’s learning performance when inetracting with a robot operated at different autonomy levels (Wizard of Oz versus autonoumous). Furthermore, they emphasize on the impact of designing adequate robot’s behaviors on children’s corresponding learning gains in child-robot interactions.

Politeness and embodiment are pivotal elements in Human-Agent Interactions. While many previous works advocate the positive role of embodiment in enhancing Human-Agent Interactions, it remains unclear how embodiment and politeness affect individuals joining groups. In this paper, we explore how polite behaviors (verbal and nonverbal) exhibited by three distinct embodiments (humans, robots, and virtual characters) influence individuals' decisions to join a group of two agents in a controlled experiment (N=54). We assessed agent effectiveness regarding persuasiveness, perceived politeness, and participants' trajectories when joining the group. We found that embodiment does not significantly impact agent persuasiveness and perceived politeness, but polite behaviors do. Direct and explicit politeness strategies have a higher success rate in persuading participants to join at the furthest side. Lastly, participants adhered to social norms when joining at the furthest side, maintained a greater physical distance from humans, chose longer paths, and walked faster when interacting with humans.

Politeness is critical for shaping human-human interactions and therefore seems an important consideration in human interactions with Embodied Conversational Agents (ECAs). However, the impact of artificially-generated politeness behaviors on humans in Virtual Environments (VE) is not clear. We explore the impact of immersiveness on the perceived politeness and consequent persuasive abilities of ECAs in a small group context. A user study with two main conditions, immersive and non-immersive, was conducted with 66 participants. In the immersive condition, participants were fully immersed in virtual reality (HMD, walking freely), while in the non-immersive condition, participants used a desktop computer interface (screen display, mouse and keyboard control). In both conditions, the primary agent in a group of two ECAs invited participants to join the group using six politeness behaviors derived from Brown and Levinson's politeness theory. While the results of the study did not indicate any significant differences between the immersive and non-immersive conditions in terms of persuasiveness and offensiveness, in the immersive condition, participants perceived the ECAs as less friendly and found their requests to be less clear. On the other hand, participants in the immersive condition reported a greater sense of freedom. Furthermore, the non-immersive condition showed higher adherence to social norms compared to the immersive condition. These findings emphasize the significance of examining immersiveness on the persuasiveness of ECAs and their perceived politeness and social adherence by humans in human-agent interactions within small groups.