Existing methods for human parsing into body parts and clothing often use fixed mask categories with broad labels that obscure fine-grained clothing types. Recent open-vocabulary segmentation approaches leverage pretrained text-to-image (T2I) diffusion model features for strong zero-shot transfer, but typically group entire humans into a single person category, failing to distinguish diverse clothing or detailed body parts. To address this, we propose Spectrum, a unified network for part-level pixel parsing (body parts and clothing) and instance-level grouping. While diffusion-based open-vocabulary models generalize well across tasks, their internal representations are not specialized for detailed human parsing. We observe that, unlike diffusion models with broad representations, image-driven 3D texture generators maintain faithful correspondence to input images, enabling stronger representations for parsing diverse clothing and body parts. Spectrum introduces a novel repurposing of an Image-to-Texture (I2Tx) diffusion model—obtained by fine-tuning a T2I model on 3D human texture maps—for improved alignment with body parts and clothing. From an input image, we extract human-part internal features via the I2Tx diffusion model and generate semantically valid masks aligned to diverse clothing categories through prompt-guided grounding. Once trained, Spectrum produces semantic segmentation maps for every visible body part and clothing category, ignoring standalone garments or irrelevant objects, for any number of humans in the scene. We conduct extensive cross-dataset experiments—separately assessing body parts, clothing parts, unseen clothing categories, and full-body masks—and demonstrate that Spectrum consistently outperforms baseline methods in prompt-based segmentation.

City Information Modelling (CIM) seeks to create digital tools for urban planners and designers with design toolbox and elements that derive from morphological research and to develop software for analysis, planning and design of cities that more closely corresponds to urban planning and design practices. The paper explores urban models and data structures from geoinformatics, architectural informatics, computer graphics and generative algorithms that are used to create interactive virtual cities. It focuses on the peculiarities of professional urban design practices and the computational models for digital cities to discuss conceptualization and programming of CIM as a digital tool. The paper aims to inspire discussions on linking data structure, design elements and drawing boards for development of a CIM software and establish a connection between data structures and suitability for the specific practices and deliverables of urban planners and designers. It is a conceptual paper for a software that is currently programmed, and further user testing will articulate practical applications and benefits.

Animations produced by generative models are often evaluated using objective quantitative metrics that do not fully capture perceptual effects in immersive virtual environments. To address this gap, we present a preliminary perceptual evaluation of generative models for animation synthesis, conducted via a VR-based user study (N = 48). Our investigation specifically focuses on animation congruency—ensuring that generated facial expressions and body gestures are both congruent with and synchronized to driving speech. We evaluated two state-of-the-art methods: a speech-driven full-body animation model and a video-driven full-body reconstruction model, assessing their capability to produce congruent facial expressions and body gestures. Our results demonstrate a strong user preference for combined facial and body animations, highlighting that congruent multimodal animations significantly enhance perceived realism compared to animations featuring only a single modality. By incorporating VR-based perceptual feedback into training pipelines, our approach provides a foundation for developing more engaging and responsive virtual characters.

Introduction: Social interactions incorporate various nonverbal signals to convey emotions alongside speech, including facial expressions and body gestures. Generative models have demonstrated promising results in creating full-body nonverbal animations synchronized with speech; however, evaluations using statistical metrics in 2D settings fail to fully capture user-perceived emotions, limiting our understanding of the effectiveness of these models. Methods: To address this, we evaluate emotional 3D animation generative models within an immersive Virtual Reality (VR) environment, emphasizing user—centric metrics-emotional arousal realism, naturalness, enjoyment, diversity, and interaction quality—in a real-time human-agent interaction scenario. Through a user study (N = 48), we systematically examine perceived emotional quality for three state-of-the-art speech-driven 3D animation methods across two specific emotions: happiness (high arousal) and neutral (mid arousal). Additionally, we compare these generative models against real human expressions obtained via a reconstruction-based method to assess both their strengths and limitations and how closely they replicate real human facial and body expressions. Results: Our results demonstrate that methods explicitly modeling emotions lead to higher recognition accuracy compared to those focusing solely on speech-driven synchrony. Users rated the realism and naturalness of happy animations significantly higher than those of neutral animations, highlighting the limitations of current generative models in handling subtle emotional states. Discussion: Generative models underperformed compared to reconstruction-based methods in facial expression quality, and all methods received relatively low ratings for animation enjoyment and interaction quality, emphasizing the importance of incorporating user-centric evaluations into generative model development. Finally, participants positively recognized animation diversity across all generative models.

Embodied social agents are entities, either physical or digital, that can communicate with humans or even other agents, often expressing ideas, thoughts, or emotions. In this study, participants were given the task of collecting a coffee cup standing on a table behind a free-standing conversational group of two embodied virtual agents in a Virtual Reality (VR) environment. Participants were presented with a social dilemma, since they had to choose whether to walk between the agents, violating their o-space, or through a narrower gap between the agents and the table. In each of the six conditions the agents expressed different interpersonal attitudes defined along Friendly/Hostile and Submissive/Dominant axes, shown through different verbal scripts and non-verbal behaviours. The 32 participants in this within-group user study walked around the agent group in 62.9% of the trials. In the four conditions where the participants were asked by one of the agents to wait, they were significantly more likely to walk around the agents rather than between them. Of those conditions, participants violated the agents’ o-space the most in the conditions in which agents’ attitudes were perceived to be the most Hostile and Dominant. Participants liked the Friendly agents the most and the Hostile agents the least. We discuss these findings in addition to their implications for the design of socially interactive agents.

This cross-cultural study (N=108) examines how cultural differences between Japan and Sweden influence participants social behaviors and perceptions when joining a free-standing group of two agents. Agents within the group, embodied as humans, robots, and virtual characters, respectively, use three distinct behaviors, varying with respect to politeness strategy, to request the participant to join on a specific side and position in the group. The experimental results showed that Japanese participants, from a culture characterized by higher power distance, masculinity, uncertainty avoidance, long-term orientation, restraint, and collectivism, were more likely to comply with the agent's request regarding the joining position, compared to Swedish participants. This trend was even more pronounced when comparing different types of embodiment: Japanese participants more strictly complied with human agents than with non-human agents. Additionally, Japanese females and Swedish males adhered more to social norms by avoiding walking between group members (i.e. through the group's o-space) when joining. Second, cultural differences also significantly impacted the perception of agents' politeness behaviors, while the effect of embodiment on feelings of friendliness and closeness varied depending on the culture. We reflect on our results as a basis for highlighting key challenges involved in the design of culturally adapted agents and their behaviors toward enhancing the localization of human-agent interaction.

Human footsteps play a significant role in everyday life, allowing individuals to discern the emotions, gender, and intentions of others solely from the sound of their footsteps. However, the influence of footstep sounds made when walking on different floor types in virtual reality (VR) environments when joining conversational groups remains unclear. In this paper, we present a controlled study (N=50) to assess the impact of five different floor types, associated with specific footstep sounds and visuals, on the persuasiveness of Embodied Conversational Agents (ECAs) when inviting participants to join a free-standing conversational group. We analyze routes taken by participants and the positions at which they join the group, which may be compliant or not with the agent’s request when approaching the group while walking on different virtual floor types. Our findings reveal that the type of floor being walked upon, defined by footstep sounds and visual appearance, significantly impacts the persuasiveness of ECAs and the trajectories taken by participants to join the group. Participants took longer paths and joined the group in the presence of more pleasant footstep sounds. Further, they tended to adhere to social norms by avoiding walking through the group’s center.

The creation of virtual humans increasingly leverages automated synthesis of speech and gestures, enabling expressive, adaptable agents that effectively engage users. However, the independent development of voice and gesture generation technologies, alongside the growing popularity of virtual reality (VR), presents significant questions about the integration of these signals and their ability to convey emotional detail in immersive environments. In this paper, we evaluate the influence of real and synthetic gestures and speech, alongside varying levels of immersion (VR vs. 2D displays) and emotional contexts (positive, neutral, negative) on user perceptions. We investigate how immersion affects the perceived match between gestures and speech and the impact on key aspects of user experience, including emotional and empathetic responses and the sense of co-presence. Our findings indicate that while VR enhances the perception of natural gesture–voice pairings, it does not similarly improve synthetic ones—amplifying the perceptual gap between them. These results highlight the need to reassess gesture appropriateness and refine AI-driven synthesis for immersive environments.

Existing methods for synthesizing 3D human gestures from speech have shown promising results but they do not explicitly model the impact of emotions on the generated gestures. Instead these methods directly output animations from speech without control over the expressed emotion. To address this limitation we present AMUSE an emotional speech-driven body animation model based on latent diffusion. Our observation is that content (i.e. gestures related to speech rhythm and word utterances) emotion and personal style are separable. To account for this AMUSE maps the driving audio to three disentangled latent vectors: one for content one for emotion and one for personal style. A latent diffusion model trained to generate gesture motion sequences is then conditioned on these latent vectors. Once trained AMUSE synthesizes 3D human gestures directly from speech with control over the expressed emotions and style by combining the content from the driving speech with the emotion and style of another speech sequence. Randomly sampling the noise of the diffusion model further generates variations of the gesture with the same emotional expressivity. Qualitative quantitative and perceptual evaluations demonstrate that AMUSE outputs realistic gesture sequences. Compared to the state of the art the generated gestures are better synchronized with the speech content and better represent the emotion expressed by the input speech. Our code is available at amuse.is.tue.mpg.de.

Politeness is a crucial aspect of human social interactions. While the infuence of politeness is well understood in human groups, it remains underexplored in group interactions with robots. Therefore, in this paper, we conduct an initial exploration into the infuence of the presence of humanoid robots on their persuasiveness and perceived politeness in small groups. We conducted a user study (N = 119) with co-present and remote robots that invited participants to join the group using six politeness behaviors derived from Brown and Levinson’s politeness theory. It requests participants to join them at the furthest side of the group, even though a closer side is also available to them, but would ignore the robot’s request. The results show that co-present robots are perceived to be less persuasive than remote ones. However, co-presence enhances the clarity of the robot’s requests and the perceived freedom of action while decreasing the perceived friendliness and ofensiveness.