kth.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Exploring the Effectiveness of Evaluation Practices for Computer-Generated Nonverbal Behaviour
Univ Ghent, IDLab Airo, imec, B-9052 Ghent, Belgium.;Radboud Univ Nijmegen, Donders Inst Brain Cognit & Behav, NL-6500 HB Nijmegen, Netherlands..ORCID iD: 0000-0002-7420-7181
KTH, School of Electrical Engineering and Computer Science (EECS), Intelligent systems, Speech, Music and Hearing, TMH.ORCID iD: 0000-0002-1643-1054
Univ Ghent, IDLab Airo, imec, B-9052 Ghent, Belgium..ORCID iD: 0000-0001-5207-7745
2024 (English)In: Applied Sciences, E-ISSN 2076-3417, Vol. 14, no 4, article id 1460Article in journal (Refereed) Published
Abstract [en]

This paper compares three methods for evaluating computer-generated motion behaviour for animated characters: two commonly used direct rating methods and a newly designed questionnaire. The questionnaire is specifically designed to measure the human-likeness, appropriateness, and intelligibility of the generated motion. Furthermore, this study investigates the suitability of these evaluation tools for assessing subtle forms of human behaviour, such as the subdued motion cues shown when listening to someone. This paper reports six user studies, namely studies that directly rate the appropriateness and human-likeness of a computer character's motion, along with studies that instead rely on a questionnaire to measure the quality of the motion. As test data, we used the motion generated by two generative models and recorded human gestures, which served as a gold standard. Our findings indicate that when evaluating gesturing motion, the direct rating of human-likeness and appropriateness is to be preferred over a questionnaire. However, when assessing the subtle motion of a computer character, even the direct rating method yields less conclusive results. Despite demonstrating high internal consistency, our questionnaire proves to be less sensitive than directly rating the quality of the motion. The results provide insights into the evaluation of human motion behaviour and highlight the complexities involved in capturing subtle nuances in nonverbal communication. These findings have implications for the development and improvement of motion generation models and can guide researchers in selecting appropriate evaluation methodologies for specific aspects of human behaviour.

Place, publisher, year, edition, pages
MDPI AG , 2024. Vol. 14, no 4, article id 1460
Keywords [en]
human-computer interaction, embodied conversational agents, subjective evaluations
National Category
Human Computer Interaction
Identifiers
URN: urn:nbn:se:kth:diva-344465DOI: 10.3390/app14041460ISI: 001170953500001Scopus ID: 2-s2.0-85192447790OAI: oai:DiVA.org:kth-344465DiVA, id: diva2:1845233
Note

QC 20240318

Available from: 2024-03-18 Created: 2024-03-18 Last updated: 2024-05-16Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopus

Authority records

Henter, Gustav Eje

Search in DiVA

By author/editor
Wolfert, PieterHenter, Gustav EjeBelpaeme, Tony
By organisation
Speech, Music and Hearing, TMH
In the same journal
Applied Sciences
Human Computer Interaction

Search outside of DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 72 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf