Animations produced by generative models are often evaluated using objective quantitative metrics that do not fully capture perceptual effects in immersive virtual environments. To address this gap, we present a preliminary perceptual evaluation of generative models for animation synthesis, conducted via a VR-based user study (N = 48). Our investigation specifically focuses on animation congruency—ensuring that generated facial expressions and body gestures are both congruent with and synchronized to driving speech. We evaluated two state-of-the-art methods: a speech-driven full-body animation model and a video-driven full-body reconstruction model, assessing their capability to produce congruent facial expressions and body gestures. Our results demonstrate a strong user preference for combined facial and body animations, highlighting that congruent multimodal animations significantly enhance perceived realism compared to animations featuring only a single modality. By incorporating VR-based perceptual feedback into training pipelines, our approach provides a foundation for developing more engaging and responsive virtual characters.

This cross-cultural study (N=108) examines how cultural differences between Japan and Sweden influence participants social behaviors and perceptions when joining a free-standing group of two agents. Agents within the group, embodied as humans, robots, and virtual characters, respectively, use three distinct behaviors, varying with respect to politeness strategy, to request the participant to join on a specific side and position in the group. The experimental results showed that Japanese participants, from a culture characterized by higher power distance, masculinity, uncertainty avoidance, long-term orientation, restraint, and collectivism, were more likely to comply with the agent's request regarding the joining position, compared to Swedish participants. This trend was even more pronounced when comparing different types of embodiment: Japanese participants more strictly complied with human agents than with non-human agents. Additionally, Japanese females and Swedish males adhered more to social norms by avoiding walking between group members (i.e. through the group's o-space) when joining. Second, cultural differences also significantly impacted the perception of agents' politeness behaviors, while the effect of embodiment on feelings of friendliness and closeness varied depending on the culture. We reflect on our results as a basis for highlighting key challenges involved in the design of culturally adapted agents and their behaviors toward enhancing the localization of human-agent interaction.

Existing methods for synthesizing 3D human gestures from speech have shown promising results but they do not explicitly model the impact of emotions on the generated gestures. Instead these methods directly output animations from speech without control over the expressed emotion. To address this limitation we present AMUSE an emotional speech-driven body animation model based on latent diffusion. Our observation is that content (i.e. gestures related to speech rhythm and word utterances) emotion and personal style are separable. To account for this AMUSE maps the driving audio to three disentangled latent vectors: one for content one for emotion and one for personal style. A latent diffusion model trained to generate gesture motion sequences is then conditioned on these latent vectors. Once trained AMUSE synthesizes 3D human gestures directly from speech with control over the expressed emotions and style by combining the content from the driving speech with the emotion and style of another speech sequence. Randomly sampling the noise of the diffusion model further generates variations of the gesture with the same emotional expressivity. Qualitative quantitative and perceptual evaluations demonstrate that AMUSE outputs realistic gesture sequences. Compared to the state of the art the generated gestures are better synchronized with the speech content and better represent the emotion expressed by the input speech. Our code is available at amuse.is.tue.mpg.de.

Politeness is a crucial aspect of human social interactions. While the infuence of politeness is well understood in human groups, it remains underexplored in group interactions with robots. Therefore, in this paper, we conduct an initial exploration into the infuence of the presence of humanoid robots on their persuasiveness and perceived politeness in small groups. We conducted a user study (N = 119) with co-present and remote robots that invited participants to join the group using six politeness behaviors derived from Brown and Levinson’s politeness theory. It requests participants to join them at the furthest side of the group, even though a closer side is also available to them, but would ignore the robot’s request. The results show that co-present robots are perceived to be less persuasive than remote ones. However, co-presence enhances the clarity of the robot’s requests and the perceived freedom of action while decreasing the perceived friendliness and ofensiveness. 

Creativity is an important skill that is known to plummet in children when they start school education that limits their freedom of expression and their imagination. On the other hand, research has shown that integrating social robots into educational settings has the potential to maximize children’s learning outcomes. Therefore, our aim in this work was to investigate stimulating children’s creativity through child-robot interactions. We fine-tuned a Large Language Model (LLM) to exhibit creative behavior and non-creative behavior in a robot and conducted two studies with children to evaluate the viability of our methods in fostering children’s creativity skills. We evaluated creativity in terms of four metrics: fluency, flexibility, elaboration, and originality. We first conducted a study as a storytelling interaction between a child and a wizard-ed social robot in one of two conditions: creative versus non-creative with 38 children. We investigated whether interacting with a creative social robot will elicit more creativity from children. However, we did not find a significant effect of the robot’s creativity on children’s creative abilities. Second, in an attempt to increase the possibility for the robot to have an impact on children’s creativity and to increase the fluidity of the interaction, we produced two models that allow a social agent to autonomously engage with a human in a storytelling context in a creative manner and a non-creative manner respectively. Finally, we conducted another study to evaluate our models by deploying them on a social robot and evaluating them with 103 children. Our results show that children who interacted with the creative autonomous robot were more creative than children who interacted with the non-creative autonomous robot in terms of the fluency, the flexibility, and the elaboration aspects of creativity. The results highlight the difference in children’s learning performance when inetracting with a robot operated at different autonomy levels (Wizard of Oz versus autonoumous). Furthermore, they emphasize on the impact of designing adequate robot’s behaviors on children’s corresponding learning gains in child-robot interactions.

Politeness and embodiment are pivotal elements in Human-Agent Interactions. While many previous works advocate the positive role of embodiment in enhancing Human-Agent Interactions, it remains unclear how embodiment and politeness affect individuals joining groups. In this paper, we explore how polite behaviors (verbal and nonverbal) exhibited by three distinct embodiments (humans, robots, and virtual characters) influence individuals' decisions to join a group of two agents in a controlled experiment (N=54). We assessed agent effectiveness regarding persuasiveness, perceived politeness, and participants' trajectories when joining the group. We found that embodiment does not significantly impact agent persuasiveness and perceived politeness, but polite behaviors do. Direct and explicit politeness strategies have a higher success rate in persuading participants to join at the furthest side. Lastly, participants adhered to social norms when joining at the furthest side, maintained a greater physical distance from humans, chose longer paths, and walked faster when interacting with humans.

Politeness is critical for shaping human-human interactions and therefore seems an important consideration in human interactions with Embodied Conversational Agents (ECAs). However, the impact of artificially-generated politeness behaviors on humans in Virtual Environments (VE) is not clear. We explore the impact of immersiveness on the perceived politeness and consequent persuasive abilities of ECAs in a small group context. A user study with two main conditions, immersive and non-immersive, was conducted with 66 participants. In the immersive condition, participants were fully immersed in virtual reality (HMD, walking freely), while in the non-immersive condition, participants used a desktop computer interface (screen display, mouse and keyboard control). In both conditions, the primary agent in a group of two ECAs invited participants to join the group using six politeness behaviors derived from Brown and Levinson's politeness theory. While the results of the study did not indicate any significant differences between the immersive and non-immersive conditions in terms of persuasiveness and offensiveness, in the immersive condition, participants perceived the ECAs as less friendly and found their requests to be less clear. On the other hand, participants in the immersive condition reported a greater sense of freedom. Furthermore, the non-immersive condition showed higher adherence to social norms compared to the immersive condition. These findings emphasize the significance of examining immersiveness on the persuasiveness of ECAs and their perceived politeness and social adherence by humans in human-agent interactions within small groups.

Multimodal communication is essential in human interactions, as it allows for a more comprehensive and nuanced exchange of information and emotions. The use of multiple communication channels such as speech, body language, and gaze can enhance the clarity and richness of the communication, leading to better understanding and more effective social interactions. This paper investigates the importance of multimodal expressive communication, specifically voice, arm gestures, and gaze, in regulating human-agent interaction when joining a group of two virtual agents in a virtual reality environment. One of the virtual agents in the group uses politeness behaviors based on Brown and Levinson's politeness theory to invite participants to join the group at the side further to them, even though a closer side is available. The study finds that a combination of all modalities (verbal, gaze, arm gesture) is more effective in persuading participants to join the group at the farthest side, and arm gestures alone are more effective than gaze behavior although they are perceived to be less polite. Furthermore, although verbal-only communication can be as persuasive as other modalities, it can place a greater cognitive load on participants. This increased cognitive load may lead to delayed responses in comparison to other modalities. The findings give insight to designers of human-agent interaction systems about the use of multiple communication channels, particularly nonverbal behaviors such as arm gestures, to enhance the effectiveness of persuasive communication but they also need to balance this with other factors such as the impression and perceived politeness of virtual agents.

Politeness is at the core of the common set of behavioral norms that regulate human communication and is therefore of significant interest in the design of Human-Robot Interactions. In this paper, we investigate how the politeness behaviors of a humanoid robot impact human decisions about where to join a group of two robots. We also evaluate the resulting impact on the perception of the robot's politeness. In a study involving 59 participants, the main (Pepper) robot in the group invited participants to join using six politeness behaviors derived from Brown and Levinson's politeness theory. It requests participants to join the group at the furthest side of the group which involves more effort to reach than a closer side that is also available to the participant but would ignore the request of the robot. We evaluated the robot's effectiveness in terms of persuasiveness, politeness, and clarity. We found that more direct and explicit politeness strategies derived from the theory have a higher level of success in persuading participants to join at the furthest side of the group. We also evaluated participants' adherence to social norms i.e. not walking through the center, or \textit{o-space}, of the group when joining it. Our results showed that participants tended to adhere to social norms when joining at the furthest side by not walking through the center of the group of robots, even though they were informed that the robots were fully automated. 

While creativity has been previously studied in Child-Robot interaction, the effect of regulatory focus on creativity skills has not been investigated. This paper presents an exploratory study that, for the first time, uses the Regulatory Focus Theory to assess children's creativity skills in an educational context with a social robot. We investigated whether two key emotional regulation techniques, promotion (approach) and prevention (avoidance), stimulate creativity during a storytelling activity between a child and a robot. We conducted a between-subjects field study with 69 children between the ages of 7 and 9 years old, divided between two study conditions: (1) promotion, where a social robot primes children for action by eliciting positive emotional states, and (2) prevention, where a social robot primes children for avoidance by evoking a states related to security and safety associated with blockage-oriented behaviors. To assess changes in creativity as a response to the priming interaction, children were asked to tell stories to the robot before (pre-test) and after (post-test) the priming interaction. We measured creativity levels by analyzing the verbal content of the stories. We coded verbal expressions related to creativity variables, including fluency, flexibility, elaboration, and originality. Our results show that children in the promotion condition generated significantly more ideas, and their ideas were on average more original in the stories they created in the post-test rather than in the pre-test. We also modeled the process of creativity that emerges during storytelling in response to the robot's verbal behavior. This paper enriches the scientific understanding of creativity emergence in child-robot collaborative interactions.