Multimodal User Feedback During Adaptive Robot-Human Presentations
2022 (English)In: Frontiers in Computer Science, E-ISSN 2624-9898, Vol. 3Article in journal (Refereed) Published
Abstract [en]
Feedback is an essential part of all communication, and agents communicating with humans must be able to both give and receive feedback in order to ensure mutual understanding. In this paper, we analyse multimodal feedback given by humans towards a robot that is presenting a piece of art in a shared environment, similar to a museum setting. The data analysed contains both video and audio recordings of 28 participants, and the data has been richly annotated both in terms of multimodal cues (speech, gaze, head gestures, facial expressions, and body pose), as well as the polarity of any feedback (negative, positive, or neutral). We train statistical and machine learning models on the dataset, and find that random forest models and multinomial regression models perform well on predicting the polarity of the participants' reactions. An analysis of the different modalities shows that most information is found in the participants' speech and head gestures, while much less information is found in their facial expressions, body pose and gaze. An analysis of the timing of the feedback shows that most feedback is given when the robot makes pauses (and thereby invites feedback), but that the more exact timing of the feedback does not affect its meaning.
Place, publisher, year, edition, pages
Frontiers Media SA , 2022. Vol. 3
Keywords [en]
feedback, presentation, agent, robot, grounding, polarity, backchannel, multimodal
Keywords [sv]
återmatning, presentation, agent, robot, återkoppling, polaritet, reaktion, multimodal
National Category
Human Computer Interaction
Research subject
Speech and Music Communication
Identifiers
URN: urn:nbn:se:kth:diva-307105DOI: 10.3389/fcomp.2021.741148ISI: 000745131900001Scopus ID: 2-s2.0-85123110812OAI: oai:DiVA.org:kth-307105DiVA, id: diva2:1626622
Projects
Co-adaptive Human-Robot Interactive Systems
Funder
Swedish Foundation for Strategic Research, COIN
Note
QC 20220112 QC 20220216
2022-01-112022-01-112022-06-25Bibliographically approved