kth.sePublications
Change search
Link to record
Permanent link

Direct link
Publications (10 of 135) Show all publications
Borg, A., Parodis, I. & Skantze, G. (2024). Creating Virtual Patients using Robots and Large Language Models: A Preliminary Study with Medical Students. In: HRI 2024 Companion - Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction: . Paper presented at 19th Annual ACM/IEEE International Conference on Human-Robot Interaction, HRI 2024, Boulder, United States of America, Mar 11 2024 - Mar 15 2024 (pp. 273-277). Association for Computing Machinery (ACM)
Open this publication in new window or tab >>Creating Virtual Patients using Robots and Large Language Models: A Preliminary Study with Medical Students
2024 (English)In: HRI 2024 Companion - Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction, Association for Computing Machinery (ACM) , 2024, p. 273-277Conference paper, Published paper (Refereed)
Abstract [en]

This paper presents a virtual patient (VP) platform for medical education, combining a social robot, Furhat, with large language models (LLMs). Aimed at enhancing clinical reasoning (CR) training, particularly in rheumatology, this approach introduces more interactive and realistic patient simulations. The use of LLMs both for driving the dialogue, but also for the expression of emotions in the robot's face, as well as automatic analysis and generation of feedback to the student, is discussed. The platform's effectiveness was tested in a pilot study with 15 medical students, comparing it against a traditional semi-linear VP platform. The evaluation indicates a preference for the robot platform in terms of authenticity and learning effect. We conclude that this novel integration of a social robot and LLMs in VP simulations shows potential in medical education, offering a more engaging learning experience.

Place, publisher, year, edition, pages
Association for Computing Machinery (ACM), 2024
Series
ACM/IEEE International Conference on Human-Robot Interaction, ISSN 2167-2148
Keywords
Human-Robot Interaction, Large Language Models, Robots in Education, Virtual Patients
National Category
Human Computer Interaction
Identifiers
urn:nbn:se:kth:diva-344812 (URN)10.1145/3610978.3640592 (DOI)2-s2.0-85188128048 (Scopus ID)
Conference
19th Annual ACM/IEEE International Conference on Human-Robot Interaction, HRI 2024, Boulder, United States of America, Mar 11 2024 - Mar 15 2024
Note

QC 20240402

Part of ISBN 9798400703232

Available from: 2024-03-28 Created: 2024-03-28 Last updated: 2024-04-02Bibliographically approved
Ashkenazi, S., Skantze, G., Stuart-Smith, J. & Foster, M. E. (2024). Goes to the Heart: Speaking the User's Native Language. In: HRI 2024 Companion - Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction: . Paper presented at 19th Annual ACM/IEEE International Conference on Human-Robot Interaction, HRI 2024, Boulder, United States of America, Mar 11 2024 - Mar 15 2024 (pp. 214-218). Association for Computing Machinery (ACM)
Open this publication in new window or tab >>Goes to the Heart: Speaking the User's Native Language
2024 (English)In: HRI 2024 Companion - Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction, Association for Computing Machinery (ACM) , 2024, p. 214-218Conference paper, Published paper (Refereed)
Abstract [en]

We are developing a social robot to work alongside human support workers who help new arrivals in a country to navigate the necessary bureaucratic processes in that country. The ultimate goal is to develop a robot that can support refugees and asylum seekers in the UK. As a first step, we are targeting a less vulnerable population with similar support needs: international students in the University of Glasgow. As the target users are in a new country and may be in a state of stress when they seek support, forcing them to communicate in a foreign language will only fuel their anxiety, so a crucial aspect of the robot design is that it should speak the users' native language if at all possible. We provide a technical description of the robot hardware and software, and describe the user study that will shortly be carried out. At the end, we explain how we are engaging with refugee support organisations to extend the robot into one that can also support refugees and asylum seekers.

Place, publisher, year, edition, pages
Association for Computing Machinery (ACM), 2024
Series
ACM/IEEE International Conference on Human-Robot Interaction, ISSN 2167-2148
Keywords
foreign language anxiety, human-robot interaction, multilingualism, robots in public spaces, social robotics
National Category
Human Computer Interaction
Identifiers
urn:nbn:se:kth:diva-344809 (URN)10.1145/3610978.3640633 (DOI)2-s2.0-85188094545 (Scopus ID)
Conference
19th Annual ACM/IEEE International Conference on Human-Robot Interaction, HRI 2024, Boulder, United States of America, Mar 11 2024 - Mar 15 2024
Note

QC 20240402

Part of ISBN 9798400703232

Available from: 2024-03-28 Created: 2024-03-28 Last updated: 2024-04-02Bibliographically approved
Axelsson, A., Vaddadi, B., Bogdan, C. M. & Skantze, G. (2024). Robots in autonomous buses: Who hosts when no human is there?. In: HRI 2024 Companion - Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction: . Paper presented at 19th Annual ACM/IEEE International Conference on Human-Robot Interaction, HRI 2024, Boulder, United States of America, Mar 11 2024 - Mar 15 2024 (pp. 1278-1280). Association for Computing Machinery (ACM)
Open this publication in new window or tab >>Robots in autonomous buses: Who hosts when no human is there?
2024 (English)In: HRI 2024 Companion - Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction, Association for Computing Machinery (ACM) , 2024, p. 1278-1280Conference paper, Published paper (Refereed)
Abstract [en]

In mid-2023, we performed an experiment in autonomous buses in Stockholm, Sweden, to evaluate the role that social robots might have in such settings, and their effects on passengers' feeling of safety and security, given the absence of human drivers or clerks. To address the situations that may occur in autonomous public transit (APT), we compared an embodied agent to a disembodied agent. In this video publication, we showcase some of the things that worked with the interactions we created, and some problematic issues that we had not anticipated.

Place, publisher, year, edition, pages
Association for Computing Machinery (ACM), 2024
Keywords
APT, assistant, autonomous, bus, clerk, guide, passenger, public transit, public transport, robot, self-driving, shuttle, wizard
National Category
Computer and Information Sciences
Identifiers
urn:nbn:se:kth:diva-344811 (URN)10.1145/3610978.3641115 (DOI)2-s2.0-85188117955 (Scopus ID)
Conference
19th Annual ACM/IEEE International Conference on Human-Robot Interaction, HRI 2024, Boulder, United States of America, Mar 11 2024 - Mar 15 2024
Note

QC 20240402

 Part of ISBN 9798400703232

Available from: 2024-03-28 Created: 2024-03-28 Last updated: 2024-04-02Bibliographically approved
Traum, D., Skantze, G., Nishizaki, H., Higashinaka, R., Minato, T. & Nagai, T. (2024). Special issue on multimodal processing and robotics for dialogue systems (Part II). Advanced Robotics, 38(4), 193-194
Open this publication in new window or tab >>Special issue on multimodal processing and robotics for dialogue systems (Part II)
Show others...
2024 (English)In: Advanced Robotics, ISSN 0169-1864, E-ISSN 1568-5535, Vol. 38, no 4, p. 193-194Article in journal, Editorial material (Other academic) Published
Place, publisher, year, edition, pages
Informa UK Limited, 2024
National Category
Computer and Information Sciences Electrical Engineering, Electronic Engineering, Information Engineering Robotics
Identifiers
urn:nbn:se:kth:diva-344825 (URN)10.1080/01691864.2024.2319945 (DOI)001187621300003 ()2-s2.0-85188103975 (Scopus ID)
Note

QC 20240405

Available from: 2024-03-28 Created: 2024-03-28 Last updated: 2024-04-05Bibliographically approved
Ekstedt, E., Wang, S., Székely, É., Gustafsson, J. & Skantze, G. (2023). Automatic Evaluation of Turn-taking Cues in Conversational Speech Synthesis. In: Interspeech 2023: . Paper presented at 24th International Speech Communication Association, Interspeech 2023, Dublin, Ireland, Aug 20 2023 - Aug 24 2023 (pp. 5481-5485). International Speech Communication Association
Open this publication in new window or tab >>Automatic Evaluation of Turn-taking Cues in Conversational Speech Synthesis
Show others...
2023 (English)In: Interspeech 2023, International Speech Communication Association , 2023, p. 5481-5485Conference paper, Published paper (Refereed)
Abstract [en]

Turn-taking is a fundamental aspect of human communication where speakers convey their intention to either hold, or yield, their turn through prosodic cues. Using the recently proposed Voice Activity Projection model, we propose an automatic evaluation approach to measure these aspects for conversational speech synthesis. We investigate the ability of three commercial, and two open-source, Text-To-Speech (TTS) systems ability to generate turn-taking cues over simulated turns. By varying the stimuli, or controlling the prosody, we analyze the models performances. We show that while commercial TTS largely provide appropriate cues, they often produce ambiguous signals, and that further improvements are possible. TTS, trained on read or spontaneous speech, produce strong turn-hold but weak turn-yield cues. We argue that this approach, that focus on functional aspects of interaction, provides a useful addition to other important speech metrics, such as intelligibility and naturalness.

Place, publisher, year, edition, pages
International Speech Communication Association, 2023
Keywords
human-computer interaction, text-to-speech, turn-taking
National Category
Language Technology (Computational Linguistics) Computer Sciences General Language Studies and Linguistics
Identifiers
urn:nbn:se:kth:diva-337873 (URN)10.21437/Interspeech.2023-2064 (DOI)2-s2.0-85171597862 (Scopus ID)
Conference
24th International Speech Communication Association, Interspeech 2023, Dublin, Ireland, Aug 20 2023 - Aug 24 2023
Note

QC 20231010

Available from: 2023-10-10 Created: 2023-10-10 Last updated: 2023-10-10Bibliographically approved
Figueroa, C., Ochs, M. & Skantze, G. (2023). Classification of Feedback Functions in Spoken Dialog Using Large Language Models and Prosodic Features. In: 27th Workshop on the Semantics and Pragmatics of Dialogue: . Paper presented at 27th Workshop on the Semantics and Pragmatics of Dialogue, 16th and 17th August 2023, University of Maribor, Faculty of Electrical Engineering and Computer Science, Maribor, Slovenia (pp. 15-24). Maribor: University of Maribor
Open this publication in new window or tab >>Classification of Feedback Functions in Spoken Dialog Using Large Language Models and Prosodic Features
2023 (English)In: 27th Workshop on the Semantics and Pragmatics of Dialogue, Maribor: University of Maribor , 2023, p. 15-24Conference paper, Published paper (Refereed)
Abstract [en]

Feedback utterances such as ‘yeah’, ‘mhm’,and ‘okay’, convey different communicative functions depending on their prosodic realizations, as well as the conversational context in which they are produced. In this paper, we investigate the performance of different models and features for classifying the communicative function of short feedback tokens in American English dialog. We experiment with a combination of lexical and prosodic features extracted from the feedback utterance, as well as context features from the preceding utterance of the interlocutor. Given the limited amount of training data, we explore the use of a pre-trained large language model (GPT-3) to encode contextual information, as well as SimCSE sentence embeddings. The results show that good performance can be achieved with only SimCSE and lexical features, while the best performance is achieved by solely fine-tuning GPT-3, even if it does not have access to any prosodic features.

Place, publisher, year, edition, pages
Maribor: University of Maribor, 2023
National Category
Language Technology (Computational Linguistics)
Identifiers
urn:nbn:se:kth:diva-341388 (URN)
Conference
27th Workshop on the Semantics and Pragmatics of Dialogue, 16th and 17th August 2023, University of Maribor, Faculty of Electrical Engineering and Computer Science, Maribor, Slovenia
Note

QC 20231220

Available from: 2023-12-19 Created: 2023-12-19 Last updated: 2023-12-20Bibliographically approved
Offrede, T., Mishra, C., Skantze, G., Fuchs, S. & Mooshammer, C. (2023). Do Humans Converge Phonetically When Talking to a Robot?. In: Proceedings of the 20th International Congress of Phonetic Sciences, Prague 2023: . Paper presented at International Congress of Phonetic Sciences (ICPhS), 7-11 August, Prague, Czech republic (pp. 3507-3511).
Open this publication in new window or tab >>Do Humans Converge Phonetically When Talking to a Robot?
Show others...
2023 (English)In: Proceedings of the 20th International Congress of Phonetic Sciences, Prague 2023, 2023, p. 3507-3511Conference paper, Published paper (Refereed)
National Category
Language Technology (Computational Linguistics)
Identifiers
urn:nbn:se:kth:diva-341384 (URN)
Conference
International Congress of Phonetic Sciences (ICPhS), 7-11 August, Prague, Czech republic
Note

QC 20231220

Part of ISBN 978-80-908 114-2-3

Available from: 2023-12-19 Created: 2023-12-19 Last updated: 2023-12-20Bibliographically approved
Axelsson, A. & Skantze, G. (2023). Do you follow?: A fully automated system for adaptive robot presenters. In: HRI 2023: Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction. Paper presented at 18th Annual ACM/IEEE International Conference on Human-Robot Interaction, HRI 2023, Stockholm, Sweden, Mar 13 2023 - Mar 16 2023 (pp. 102-111). Association for Computing Machinery (ACM)
Open this publication in new window or tab >>Do you follow?: A fully automated system for adaptive robot presenters
2023 (English)In: HRI 2023: Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction, Association for Computing Machinery (ACM) , 2023, p. 102-111Conference paper, Published paper (Refereed)
Abstract [en]

An interesting application for social robots is to act as a presenter, for example as a museum guide. In this paper, we present a fully automated system architecture for building adaptive presentations for embodied agents. The presentation is generated from a knowledge graph, which is also used to track the grounding state of information, based on multimodal feedback from the user. We introduce a novel way to use large-scale language models (GPT-3 in our case) to lexicalise arbitrary knowledge graph triples, greatly simplifying the design of this aspect of the system. We also present an evaluation where 43 participants interacted with the system. The results show that users prefer the adaptive system and consider it more human-like and flexible than a static version of the same system, but only partial results are seen in their learning of the facts presented by the robot.

Place, publisher, year, edition, pages
Association for Computing Machinery (ACM), 2023
Keywords
adaptation, behaviour tree, feedback, knowledge graph, learning, lexicalisation, multimodal
National Category
Computer Sciences
Identifiers
urn:nbn:se:kth:diva-333378 (URN)10.1145/3568162.3576958 (DOI)2-s2.0-85150369153 (Scopus ID)
Conference
18th Annual ACM/IEEE International Conference on Human-Robot Interaction, HRI 2023, Stockholm, Sweden, Mar 13 2023 - Mar 16 2023
Note

Part of ISBN 9781450399647

QC 20230801

Available from: 2023-08-01 Created: 2023-08-01 Last updated: 2023-08-01Bibliographically approved
Mishra, C., Offrede, T., Fuchs, S., Mooshammer, C. & Skantze, G. (2023). Does a robot's gaze aversion affect human gaze aversion?. Frontiers in Robotics and AI, 10, Article ID 1127626.
Open this publication in new window or tab >>Does a robot's gaze aversion affect human gaze aversion?
Show others...
2023 (English)In: Frontiers in Robotics and AI, E-ISSN 2296-9144, Vol. 10, article id 1127626Article in journal (Refereed) Published
Abstract [en]

Gaze cues serve an important role in facilitating human conversations and are generally considered to be one of the most important non-verbal cues. Gaze cues are used to manage turn-taking, coordinate joint attention, regulate intimacy, and signal cognitive effort. In particular, it is well established that gaze aversion is used in conversations to avoid prolonged periods of mutual gaze. Given the numerous functions of gaze cues, there has been extensive work on modelling these cues in social robots. Researchers have also tried to identify the impact of robot gaze on human participants. However, the influence of robot gaze behavior on human gaze behavior has been less explored. We conducted a within-subjects user study (N = 33) to verify if a robot's gaze aversion influenced human gaze aversion behavior. Our results show that participants tend to avert their gaze more when the robot keeps staring at them as compared to when the robot exhibits well-timed gaze aversions. We interpret our findings in terms of intimacy regulation: humans try to compensate for the robot's lack of gaze aversion.

Place, publisher, year, edition, pages
Frontiers Media SA, 2023
Keywords
gaze, gaze aversion, human-robot interaction, social robot, gaze control model, gaze behavior, intimacy, topic intimacy
National Category
Computer Vision and Robotics (Autonomous Systems)
Identifiers
urn:nbn:se:kth:diva-333231 (URN)10.3389/frobt.2023.1127626 (DOI)001025081900001 ()37427087 (PubMedID)2-s2.0-85164584325 (Scopus ID)
Note

QC 20230731

Available from: 2023-07-31 Created: 2023-07-31 Last updated: 2023-07-31Bibliographically approved
Kamelabad, A. M. & Skantze, G. (2023). I Learn Better Alone! Collaborative and Individual Word Learning With a Child and Adult Robot. In: Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction: . Paper presented at ACM/IEEE International Conference on Human-Robot Interaction, March 13-16, 2023 Stockholm, Sweden (pp. 368-377). New York, NY, United States: Association for Computing Machinery (ACM)
Open this publication in new window or tab >>I Learn Better Alone! Collaborative and Individual Word Learning With a Child and Adult Robot
2023 (English)In: Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction, New York, NY, United States: Association for Computing Machinery (ACM), 2023, p. 368-377Conference paper, Published paper (Refereed)
Abstract [en]

The use of social robots as a tool for language learning has been studied quite extensively recently. Although their effectiveness and comparison with other technologies are well studied, the effects of the robot’s appearance and the interaction setting have received less attention. As educational robots are envisioned to appear in household or school environments, it is important to investigate how their designed persona or interaction dynamics affect learning outcomes. In such environments, children may do the activities together or alone or perform them in the presence of an adult or another child. In this regard, we have identified two novel factors to investigate: the robot’s perceived age (adult or child) and the number of learners interacting with the robot simultaneously (one or two). We designed an incidental word learning card game with the Furhat robot and ran a between-subject experiment with 75 middle school participants. We investigated the interactions and effects of children’s word learning outcomes, speech activity, and perception of the robot’s role. The results show that children who played alone with the robot had better word retention and anthropomorphized the robot more, compared to those who played in pairs. Furthermore, unlike previous findings from human-human interactions, children did not show different behaviors in the presence of a robot designed as an adult or a child. We discuss these factors in detail and make a novel contribution to the direct comparison of collaborative versus individual learning and the new concept of the robot’s age.

Place, publisher, year, edition, pages
New York, NY, United States: Association for Computing Machinery (ACM), 2023
Series
HRI ’23
Keywords
robot social role, child language learning, word learning, robot-assisted language learning, collaborative learning, second language, social robotics, multi-party interaction, rall, depth of processing, l2 learning, task-induced involvement load
National Category
Human Computer Interaction
Identifiers
urn:nbn:se:kth:diva-324855 (URN)10.1145/3568162.3577004 (DOI)2-s2.0-85150388283 (Scopus ID)
Conference
ACM/IEEE International Conference on Human-Robot Interaction, March 13-16, 2023 Stockholm, Sweden
Projects
Early Language Development in the Digital Age (e-LADDA)
Funder
EU, Horizon 2020, 857897
Note

QC 20231122

Available from: 2023-03-19 Created: 2023-03-19 Last updated: 2023-11-22Bibliographically approved
Organisations
Identifiers
ORCID iD: ORCID iD iconorcid.org/0000-0002-8579-1790

Search in DiVA

Show all publications