kth.sePublications
Change search
Link to record
Permanent link

Direct link
Publications (9 of 9) Show all publications
Johansson, M., Hori, T., Skantze, G., Hothker, A. & Gustafson, J. (2016). Making Turn-Taking Decisions for an Active Listening Robot for Memory Training. In: SOCIAL ROBOTICS, (ICSR 2016): . Paper presented at 8th International Conference on Social Robotics (ICSR), NOV 01-03, 2016, Kansas City, MO (pp. 940-949). Springer
Open this publication in new window or tab >>Making Turn-Taking Decisions for an Active Listening Robot for Memory Training
Show others...
2016 (English)In: SOCIAL ROBOTICS, (ICSR 2016), Springer, 2016, p. 940-949Conference paper, Published paper (Refereed)
Abstract [en]

In this paper we present a dialogue system and response model that allows a robot to act as an active listener, encouraging users to tell the robot about their travel memories. The response model makes a combined decision about when to respond and what type of response to give, in order to elicit more elaborate descriptions from the user and avoid non-sequitur responses. The model was trained on human-robot dialogue data collected in a Wizard-of-Oz setting, and evaluated in a fully autonomous version of the same dialogue system. Compared to a baseline system, users perceived the dialogue system with the trained model to be a significantly better listener. The trained model also resulted in dialogues with significantly fewer mistakes, a larger proportion of user speech and fewer interruptions.

Place, publisher, year, edition, pages
Springer, 2016
Series
Lecture Notes in Artificial Intelligence, ISSN 0302-9743 ; 9979
Keywords
Turn-taking, Active listening, Social robotics, Memory training
National Category
Computer and Information Sciences
Identifiers
urn:nbn:se:kth:diva-200064 (URN)10.1007/978-3-319-47437-3_92 (DOI)000389816500092 ()2-s2.0-84992499074 (Scopus ID)978-3-319-47437-3 (ISBN)978-3-319-47436-6 (ISBN)
Conference
8th International Conference on Social Robotics (ICSR), NOV 01-03, 2016, Kansas City, MO
Note

QC 20170125

Available from: 2017-01-25 Created: 2017-01-20 Last updated: 2022-06-27Bibliographically approved
Skantze, G., Johansson, M. & Beskow, J. (2015). A Collaborative Human-Robot Game as a Test-bed for Modelling Multi-party, Situated Interaction. In: INTELLIGENT VIRTUAL AGENTS, IVA 2015: . Paper presented at 15th International Conference on Intelligent Virtual Agents (IVA), AUG 26-28, 2015, Delft, NETHERLANDS (pp. 348-351).
Open this publication in new window or tab >>A Collaborative Human-Robot Game as a Test-bed for Modelling Multi-party, Situated Interaction
2015 (English)In: INTELLIGENT VIRTUAL AGENTS, IVA 2015, 2015, p. 348-351Conference paper, Published paper (Refereed)
Abstract [en]

In this demonstration we present a test-bed for collecting data and testing out models for multi-party, situated interaction between humans and robots. Two users are playing a collaborative card sorting game together with the robot head Furhat. The cards are shown on a touch table between the players, thus constituting a target for joint attention. The system has been exhibited at the Swedish National Museum of Science and Technology during nine days, resulting in a rich multi-modal corpus with users of mixed ages.

Series
Lecture Notes in Artificial Intelligence, ISSN 0302-9743 ; 9238
National Category
Computer and Information Sciences
Identifiers
urn:nbn:se:kth:diva-177091 (URN)10.1007/978-3-319-21996-7_37 (DOI)000363485400037 ()2-s2.0-84943635137 (Scopus ID)978-3-319-21996-7 (ISBN)978-3-319-21995-0 (ISBN)
Conference
15th International Conference on Intelligent Virtual Agents (IVA), AUG 26-28, 2015, Delft, NETHERLANDS
Note

QC 20151118

Available from: 2015-11-18 Created: 2015-11-13 Last updated: 2022-06-23Bibliographically approved
Skantze, G., Johansson, M. & Beskow, J. (2015). Exploring Turn-taking Cues in Multi-party Human-robot Discussions about Objects. In: Proceedings of the 2015 ACM International Conference on Multimodal Interaction: . Paper presented at ACM International Conference on Multimodal Interaction, ICMI 2015; Seattle, United States; 9-13 November 2015. Association for Computing Machinery (ACM)
Open this publication in new window or tab >>Exploring Turn-taking Cues in Multi-party Human-robot Discussions about Objects
2015 (English)In: Proceedings of the 2015 ACM International Conference on Multimodal Interaction, Association for Computing Machinery (ACM), 2015Conference paper, Published paper (Refereed)
Abstract [en]

In this paper, we present a dialog system that was exhibited at the Swedish National Museum of Science and Technology. Two visitors at a time could play a collaborative card sorting game together with the robot head Furhat, where the three players discuss the solution together. The cards are shown on a touch table between the players, thus constituting a target for joint attention. We describe how the system was implemented in order to manage turn-taking and attention to users and objects in the shared physical space. We also discuss how multi-modal redundancy (from speech, card movements and head pose) is exploited to maintain meaningful discussions, given that the system has to process conversational speech from both children and adults in a noisy environment. Finally, we present an analysis of 373 interactions, where we investigate the robustness of the system, to what extent the system's attention can shape the users' turn-taking behaviour, and how the system can produce multi-modal turn-taking signals (filled pauses, facial gestures, breath and gaze) to deal with processing delays in the system.

Place, publisher, year, edition, pages
Association for Computing Machinery (ACM), 2015
National Category
Computer Sciences Natural Language Processing
Identifiers
urn:nbn:se:kth:diva-180422 (URN)10.1145/2818346.2820749 (DOI)000380609500012 ()2-s2.0-84959259564 (Scopus ID)
Conference
ACM International Conference on Multimodal Interaction, ICMI 2015; Seattle, United States; 9-13 November 2015
Note

QC 20160331

Available from: 2016-01-13 Created: 2016-01-13 Last updated: 2025-02-01Bibliographically approved
Skantze, G. & Johansson, M. (2015). Modelling situated human-robot interaction using IrisTK. In: Proceedings of the SIGDIAL 2015 Conference: . Paper presented at SIGDIAL 2015; Prague, Czech Republic; 2-4 September 2015 (pp. 165-167).
Open this publication in new window or tab >>Modelling situated human-robot interaction using IrisTK
2015 (English)In: Proceedings of the SIGDIAL 2015 Conference, 2015, p. 165-167Conference paper, Published paper (Refereed)
Abstract [en]

In this demonstration we show how situated multi-party human-robot interaction can be modelled using the open source framework IrisTK. We will demonstrate the capabilities of IrisTK by showing an application where two users are playing a collaborative card sorting game together with the robot head Furhat, where the cards are shown on a touch table between the players. The application is interesting from a research perspective, as it involves both multi-party interaction, as well as joint attention to the objects under discussion.

National Category
Computer Sciences Natural Language Processing
Identifiers
urn:nbn:se:kth:diva-180408 (URN)10.18653/v1/w15-4624 (DOI)2-s2.0-84988394205 (Scopus ID)
Conference
SIGDIAL 2015; Prague, Czech Republic; 2-4 September 2015
Note

QC 20160331

Available from: 2016-01-13 Created: 2016-01-13 Last updated: 2025-02-01Bibliographically approved
Johansson, M., Skantze, G. & Gustafson, J. (2014). Comparison of human-human and human-robot Turn-taking Behaviour in multi-party Situated interaction. In: UM3I '14: Proceedings of the 2014 workshop on Understanding and Modeling Multiparty, Multimodal Interactions. Paper presented at International Workshop on Understanding and Modeling Multiparty, Multimodal Interactions, at ICMI 2014, Bogazici University, Istanbul, Turkey. November 12-16th, 2014 (pp. 21-26). Istanbul, Turkey
Open this publication in new window or tab >>Comparison of human-human and human-robot Turn-taking Behaviour in multi-party Situated interaction
2014 (English)In: UM3I '14: Proceedings of the 2014 workshop on Understanding and Modeling Multiparty, Multimodal Interactions, Istanbul, Turkey, 2014, p. 21-26Conference paper, Published paper (Refereed)
Abstract [en]

In this paper, we present an experiment where two human subjects are given a team-building task to solve together with a robot. The setting requires that the speakers' attention is partly directed towards objects on the table between them, as well as to each other, in order to coordinate turn-taking. The symmetrical setup allows us to compare human-human and human-robot turn-taking behaviour in the same interactional setting. The analysis centres around the interlocutors' attention (as measured by head pose) and gap length between turns, depending on the pragmatic function of the utterances.

Place, publisher, year, edition, pages
Istanbul, Turkey: , 2014
National Category
Computer Sciences Natural Language Processing
Identifiers
urn:nbn:se:kth:diva-158170 (URN)10.1145/2666242.2666249 (DOI)2-s2.0-84919383902 (Scopus ID)978-1-4503-0652-2 (ISBN)
Conference
International Workshop on Understanding and Modeling Multiparty, Multimodal Interactions, at ICMI 2014, Bogazici University, Istanbul, Turkey. November 12-16th, 2014
Note

QC 20150417.

tmh_import_14_12_30, tmh_id_3949

Available from: 2014-12-30 Created: 2014-12-30 Last updated: 2025-02-01Bibliographically approved
Al Moubayed, S., Beskow, J., Bollepalli, B., Gustafson, J., Hussen-Abdelaziz, A., Johansson, M., . . . Varol, G. (2014). Human-robot Collaborative Tutoring Using Multiparty Multimodal Spoken Dialogue. In: : . Paper presented at 9th Annual ACM/IEEE International Conference on Human-Robot Interaction, Bielefeld, Germany. IEEE conference proceedings
Open this publication in new window or tab >>Human-robot Collaborative Tutoring Using Multiparty Multimodal Spoken Dialogue
Show others...
2014 (English)Conference paper, Published paper (Refereed)
Abstract [en]

In this paper, we describe a project that explores a novel experi-mental setup towards building a spoken, multi-modally rich, and human-like multiparty tutoring robot. A human-robotinteraction setup is designed, and a human-human dialogue corpus is collect-ed. The corpus targets the development of a dialogue system platform to study verbal and nonverbaltutoring strategies in mul-tiparty spoken interactions with robots which are capable of spo-ken dialogue. The dialogue task is centered on two participants involved in a dialogueaiming to solve a card-ordering game. Along with the participants sits a tutor (robot) that helps the par-ticipants perform the task, and organizes and balances their inter-action. Differentmultimodal signals captured and auto-synchronized by different audio-visual capture technologies, such as a microphone array, Kinects, and video cameras, were coupled with manual annotations. These are used build a situated model of the interaction based on the participants personalities, their state of attention, their conversational engagement and verbal domi-nance, and how that is correlated with the verbal and visual feed-back, turn-management, and conversation regulatory actions gen-erated by the tutor. Driven by the analysis of the corpus, we will show also the detailed design methodologies for an affective, and multimodally rich dialogue system that allows the robot to meas-ure incrementally the attention states, and the dominance for each participant, allowing the robot head Furhat to maintain a well-coordinated, balanced, and engaging conversation, that attempts to maximize the agreement and the contribution to solve the task. This project sets the first steps to explore the potential of us-ing multimodal dialogue systems to build interactive robots that can serve in educational, team building, and collaborative task solving applications.

Place, publisher, year, edition, pages
IEEE conference proceedings, 2014
Keywords
Furhat robot; Human-robot collaboration; Human-robot interaction; Multiparty interaction; Spoken dialog
National Category
Computer Sciences
Identifiers
urn:nbn:se:kth:diva-145511 (URN)10.1145/2559636.2563681 (DOI)000455229400029 ()2-s2.0-84896934381 (Scopus ID)
Conference
9th Annual ACM/IEEE International Conference on Human-Robot Interaction, Bielefeld, Germany
Note

QC 20161018

Available from: 2014-05-21 Created: 2014-05-21 Last updated: 2024-03-15Bibliographically approved
Koutsombogera, M., Al Moubayed, S., Bollepalli, B., Abdelaziz, A. H., Johansson, M., Aguas Lopes, J. D., . . . Varol, G. (2014). The Tutorbot Corpus - A Corpus for Studying Tutoring Behaviour in Multiparty Face-to-Face Spoken Dialogue. In: : . Paper presented at 9th International Conference on Language Resources and Evaluation, Reykjavik, Iceland. EUROPEAN LANGUAGE RESOURCES ASSOC-ELRA
Open this publication in new window or tab >>The Tutorbot Corpus - A Corpus for Studying Tutoring Behaviour in Multiparty Face-to-Face Spoken Dialogue
Show others...
2014 (English)Conference paper, Published paper (Refereed)
Abstract [en]

This paper describes a novel experimental setup exploiting state-of-the-art capture equipment to collect a multimodally rich game-solving collaborative multiparty dialogue corpus. The corpus is targeted and designed towards the development of a dialogue system platform to explore verbal and nonverbal tutoring strategies in multiparty spoken interactions. The dialogue task is centered on two participants involved in a dialogue aiming to solve a card-ordering game. The participants were paired into teams based on their degree of extraversion as resulted from a personality test. With the participants sits a tutor that helps them perform the task, organizes and balances their interaction and whose behavior was assessed by the participants after each interaction. Different multimodal signals captured and auto-synchronized by different audio-visual capture technologies, together with manual annotations of the tutor’s behavior constitute the Tutorbot corpus. This corpus is exploited to build a situated model of the interaction based on the participants’ temporally-changing state of attention, their conversational engagement and verbal dominance, and their correlation with the verbal and visual feedback and conversation regulatory actions generated by the tutor.

Place, publisher, year, edition, pages
EUROPEAN LANGUAGE RESOURCES ASSOC-ELRA, 2014
Keywords
Multimodal corpus; Multiparty Interaction; Tutor
National Category
Computer Sciences
Identifiers
urn:nbn:se:kth:diva-173469 (URN)000355611005138 ()2-s2.0-84990228583 (Scopus ID)
Conference
9th International Conference on Language Resources and Evaluation, Reykjavik, Iceland
Note

QC 20161017

Available from: 2015-09-15 Created: 2015-09-11 Last updated: 2024-03-15Bibliographically approved
Al Moubayed, S., Beskow, J., Bollepalli, B., Hussen-Abdelaziz, A., Johansson, M., Koutsombogera, M., . . . Varol, G. (2014). Tutoring Robots: Multiparty Multimodal Social Dialogue With an Embodied Tutor. In: : . Paper presented at 9th International Summer Workshop on Multimodal Interfaces, Lisbon, Portugal. Springer Berlin/Heidelberg
Open this publication in new window or tab >>Tutoring Robots: Multiparty Multimodal Social Dialogue With an Embodied Tutor
Show others...
2014 (English)Conference paper, Published paper (Refereed)
Abstract [en]

This project explores a novel experimental setup towards building spoken, multi-modally rich, and human-like multiparty tutoring agent. A setup is developed and a corpus is collected that targets the development of a dialogue system platform to explore verbal and nonverbal tutoring strategies in multiparty spoken interactions with embodied agents. The dialogue task is centered on two participants involved in a dialogue aiming to solve a card-ordering game. With the participants sits a tutor that helps the participants perform the task and organizes and balances their interaction. Different multimodal signals captured and auto-synchronized by different audio-visual capture technologies were coupled with manual annotations to build a situated model of the interaction based on the participants personalities, their temporally-changing state of attention, their conversational engagement and verbal dominance, and the way these are correlated with the verbal and visual feedback, turn-management, and conversation regulatory actions generated by the tutor. At the end of this chapter we discuss the potential areas of research and developments this work opens and some of the challenges that lie in the road ahead.

Place, publisher, year, edition, pages
Springer Berlin/Heidelberg, 2014
Keywords
Conversational Dominance; Embodied Agent; Multimodal; Multiparty; Non-verbal Signals; Social Robot; Spoken Dialogue; Turn-taking; Tutor; Visual Attention
National Category
Computer Sciences
Identifiers
urn:nbn:se:kth:diva-158149 (URN)000349440300004 ()2-s2.0-84927643008 (Scopus ID)
Conference
9th International Summer Workshop on Multimodal Interfaces, Lisbon, Portugal
Note

QC 20161018

Available from: 2014-12-30 Created: 2014-12-30 Last updated: 2024-03-15Bibliographically approved
Johansson, M., Skantze, G. & Gustafson, J. (2013). Head Pose Patterns in Multiparty Human-Robot Team-Building Interactions. In: Guido Herrmann, Martin J. Pearson, Alexander Lenz, Paul Bremner, Adam Spiers, Ute Leonards (Ed.), Social Robotics: 5th International Conference, ICSR 2013, Bristol, UK, October 27-29, 2013, Proceedings. Paper presented at 5th International Conference on Social Robotics, ICSR 2013; Bristol; United Kingdom; 27 October 2013 through 29 October 2013 (pp. 351-360). Springer
Open this publication in new window or tab >>Head Pose Patterns in Multiparty Human-Robot Team-Building Interactions
2013 (English)In: Social Robotics: 5th International Conference, ICSR 2013, Bristol, UK, October 27-29, 2013, Proceedings / [ed] Guido Herrmann, Martin J. Pearson, Alexander Lenz, Paul Bremner, Adam Spiers, Ute Leonards, Springer, 2013, p. 351-360Conference paper, Published paper (Refereed)
Abstract [en]

We present a data collection setup for exploring turn-taking in three-party human-robot interaction involving objects competing for attention. The collected corpus comprises 78 minutes in four interactions. Using automated techniques to record head pose and speech patterns, we analyze head pose patterns in turn-transitions. We find that introduction of objects makes addressee identification based on head pose more challenging. The symmetrical setup also allows us to compare human-human to human-robot behavior within the same interaction. We argue that this symmetry can be used to assess to what extent the system exhibits a human-like behavior.

Place, publisher, year, edition, pages
Springer, 2013
Series
Lecture Notes in Computer Science, ISSN 0302-9743 ; 8239
Keywords
Focus of attention, Human-robot interaction, Turn-taking
National Category
Computer Sciences Natural Language Processing
Identifiers
urn:nbn:se:kth:diva-134936 (URN)10.1007/978-3-319-02675-6_35 (DOI)000341015200035 ()2-s2.0-84892398814 (Scopus ID)978-3-319-02675-6 (ISBN)
Conference
5th International Conference on Social Robotics, ICSR 2013; Bristol; United Kingdom; 27 October 2013 through 29 October 2013
Note

QC 20140326

Available from: 2013-12-02 Created: 2013-12-02 Last updated: 2025-02-01Bibliographically approved
Organisations
Identifiers
ORCID iD: ORCID iD iconorcid.org/0000-0002-3242-3935

Search in DiVA

Show all publications