Change search
Link to record
Permanent link

Direct link
BETA
Gustafson, Joakim, professorORCID iD iconorcid.org/0000-0002-0397-6442
Alternative names
Publications (10 of 124) Show all publications
Kontogiorgos, D., Abelho Pereira, A. T., Sahindal, B., van Waveren, S. & Gustafson, J. (2020). Behavioural Responses to Robot Conversational Failures. In: : . Paper presented at International Conference on Human Robot Interaction (HRI), HRI ’20, March 23–26, 2020, Cambridge, United Kingdom. ACM Digital Library
Open this publication in new window or tab >>Behavioural Responses to Robot Conversational Failures
Show others...
2020 (English)Conference paper, Published paper (Refereed)
Abstract [en]

Humans and robots will increasingly collaborate in domestic environments which will cause users to encounter more failures in interactions. Robots should be able to infer conversational failures by detecting human users’ behavioural and social signals. In this paper, we study and analyse these behavioural cues in response to robot conversational failures. Using a guided task corpus, where robot embodiment and time pressure are manipulated, we ask human annotators to estimate whether user affective states differ during various types of robot failures. We also train a random forest classifier to detect whether a robot failure has occurred and compare results to human annotator benchmarks. Our findings show that human-like robots augment users’ reactions to failures, as shown in users’ visual attention, in comparison to non-humanlike smart-speaker embodiments. The results further suggest that speech behaviours are utilised more in responses to failures when non-human-like designs are present. This is particularly important to robot failure detection mechanisms that may need to consider the robot’s physical design in its failure detection model.

Place, publisher, year, edition, pages
ACM Digital Library, 2020
National Category
Interaction Technologies
Identifiers
urn:nbn:se:kth:diva-267231 (URN)10.1145/3319502.3374782 (DOI)2-s2.0-85082009759 (Scopus ID)978-1-4503-6746-2 (ISBN)
Conference
International Conference on Human Robot Interaction (HRI), HRI ’20, March 23–26, 2020, Cambridge, United Kingdom
Note

QC 20200214

Available from: 2020-02-04 Created: 2020-02-04 Last updated: 2020-05-11Bibliographically approved
Abelho Pereira, A. T., Oertel, C., Fermoselle, L., Mendelson, J. & Gustafson, J. (2020). Effects of Different Interaction Contexts when Evaluating Gaze Models in HRI. In: : . Paper presented at International Conference on Human Robot Interaction (HRI).
Open this publication in new window or tab >>Effects of Different Interaction Contexts when Evaluating Gaze Models in HRI
Show others...
2020 (English)Conference paper, Published paper (Refereed)
Abstract [en]

uses multimodal information from users engaged in a spatial reasoningtask with a robot and communicates joint attention viathe robot’s gaze behavior [25]. An initial evaluation of our systemwith adults showed it to improve users’ perceptions of therobot’s social presence. To investigate the repeatability of our priorfindings across settings and populations, here we conducted twofurther studies employing the same gaze system with the samerobot and task but in different contexts: evaluation of the systemwith external observers and evaluation with children. The externalobserver study suggests that third-person perspectives over videosof gaze manipulations can be used either as a manipulation checkbefore committing to costly real-time experiments or to furtherestablish previous findings. However, the replication of our originaladults study with children in school did not confirm the effectivenessof our gaze manipulation, suggesting that different interactioncontexts can affect the generalizability of results in human-robotinteraction gaze studies.

Keywords
Joint attention, mutual gaze, social robots, social presence
National Category
Interaction Technologies
Identifiers
urn:nbn:se:kth:diva-267230 (URN)
Conference
International Conference on Human Robot Interaction (HRI)
Note

QC 20200217

Available from: 2020-02-04 Created: 2020-02-04 Last updated: 2020-02-17Bibliographically approved
Kontogiorgos, D., van Waveren, S., Wallberg, O., Abelho Pereira, A. T., Leite, I. & Gustafson, J. (2020). Embodiment Effects in Interactions with Failing Robots. In: : . Paper presented at SIGCHI Conference on Human Factors in Computing Systems, CHI ’20, April 25–30, 2020, Honolulu, HI, USA. ACM Digital Library
Open this publication in new window or tab >>Embodiment Effects in Interactions with Failing Robots
Show others...
2020 (English)Conference paper, Published paper (Refereed)
Abstract [en]

The increasing use of robots in real-world applications will inevitably cause users to encounter more failures in interactions. While there is a longstanding effort in bringing human-likeness to robots, how robot embodiment affects users’ perception of failures remains largely unexplored. In this paper, we extend prior work on robot failures by assessing the impact that embodiment and failure severity have on people’s behaviours and their perception of robots. Our findings show that when using a smart-speaker embodiment, failures negatively affect users’ intention to frequently interact with the device, however not when using a human-like robot embodiment. Additionally, users significantly rate the human-like robot higher in terms of perceived intelligence and social presence. Our results further suggest that in higher severity situations, human-likeness is distracting and detrimental to the interaction. Drawing on quantitative findings, we discuss benefits and drawbacks of embodiment in robot failures that occur in guided tasks.

Place, publisher, year, edition, pages
ACM Digital Library, 2020
National Category
Interaction Technologies
Identifiers
urn:nbn:se:kth:diva-267232 (URN)10.1145/3313831.3376372 (DOI)978-1-4503-6708-0 (ISBN)
Conference
SIGCHI Conference on Human Factors in Computing Systems, CHI ’20, April 25–30, 2020, Honolulu, HI, USA
Note

QC 20200214

Available from: 2020-02-04 Created: 2020-02-04 Last updated: 2020-02-14Bibliographically approved
Székely, É., Henter, G. E. & Gustafson, J. (2019). CASTING TO CORPUS: SEGMENTING AND SELECTING SPONTANEOUS DIALOGUE FOR TTS WITH A CNN-LSTM SPEAKER-DEPENDENT BREATH DETECTOR. In: 2019 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP): . Paper presented at 44th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), MAY 12-17, 2019, Brighton, ENGLAND (pp. 6925-6929). IEEE
Open this publication in new window or tab >>CASTING TO CORPUS: SEGMENTING AND SELECTING SPONTANEOUS DIALOGUE FOR TTS WITH A CNN-LSTM SPEAKER-DEPENDENT BREATH DETECTOR
2019 (English)In: 2019 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), IEEE , 2019, p. 6925-6929Conference paper, Published paper (Refereed)
Abstract [en]

This paper considers utilising breaths to create improved spontaneous-speech corpora for conversational text-to-speech from found audio recordings such as dialogue podcasts. Breaths are of interest since they relate to prosody and speech planning and are independent of language and transcription. Specifically, we propose a semisupervised approach where a fraction of coarsely annotated data is used to train a convolutional and recurrent speaker-specific breath detector operating on spectrograms and zero-crossing rate. The classifier output is used to find target-speaker breath groups (audio segments delineated by breaths) and subsequently select those that constitute clean utterances appropriate for a synthesis corpus. An application to 11 hours of raw podcast audio extracts 1969 utterances (106 minutes), 87% of which are clean and correctly segmented. This outperforms a baseline that performs integrated VAD and speaker attribution without accounting for breaths.

Place, publisher, year, edition, pages
IEEE, 2019
Series
International Conference on Acoustics Speech and Signal Processing ICASSP, ISSN 1520-6149
Keywords
Spontaneous speech, found data, speech synthesis corpora, breath detection, computational paralinguistics
National Category
Language Technology (Computational Linguistics)
Identifiers
urn:nbn:se:kth:diva-261049 (URN)10.1109/ICASSP.2019.8683846 (DOI)000482554007032 ()2-s2.0-85069442973 (Scopus ID)978-1-4799-8131-1 (ISBN)
Conference
44th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), MAY 12-17, 2019, Brighton, ENGLAND
Note

QC 20191002

Available from: 2019-10-02 Created: 2019-10-02 Last updated: 2019-10-02Bibliographically approved
Kontogiorgos, D., Abelho Pereira, A. T. & Gustafson, J. (2019). Estimating Uncertainty in Task Oriented Dialogue. In: Wen Gao, Helen Mei Ling Meng, Matthew Turk, Susan R. Fussell (Ed.), ICMI 2019 - Proceedings of the 2019 International Conference on Multimodal Interaction: . Paper presented at 21st ACM International Conference on Multimodal Interaction, Suzhou, Jiangsu, China. October 14-18, 2019 (pp. 414-418). ACM Digital Library
Open this publication in new window or tab >>Estimating Uncertainty in Task Oriented Dialogue
2019 (English)In: ICMI 2019 - Proceedings of the 2019 International Conference on Multimodal Interaction / [ed] Wen Gao, Helen Mei Ling Meng, Matthew Turk, Susan R. Fussell, ACM Digital Library, 2019, p. 414-418Conference paper, Published paper (Refereed)
Abstract [en]

Situated multimodal systems that instruct humans need to handle user uncertainties, as expressed in behaviour, and plan their actions accordingly. Speakers’ decision to reformulate or repair previous utterances depends greatly on the listeners’ signals of uncertainty. In this paper, we estimate uncertainty in a situated guided task, as leveraged in non-verbal cues expressed by the listener, and predict that the speaker will reformulate their utterance. We use a corpus where people instruct how to assemble furniture, and extract their multimodal features. While uncertainty is in cases ver- bally expressed, most instances are expressed non-verbally, which indicates the importance of multimodal approaches. In this work, we present a model for uncertainty estimation. Our findings indicate that uncertainty estimation from non- verbal cues works well, and can exceed human annotator performance when verbal features cannot be perceived.

Place, publisher, year, edition, pages
ACM Digital Library, 2019
Keywords
situated interaction, dialogue and discourse, grounding
National Category
Human Computer Interaction
Identifiers
urn:nbn:se:kth:diva-261628 (URN)10.1145/3340555.3353722 (DOI)2-s2.0-85074940956 (Scopus ID)9781450368605 (ISBN)
Conference
21st ACM International Conference on Multimodal Interaction, Suzhou, Jiangsu, China. October 14-18, 2019
Note

QC 20191209. QC 20200214

Available from: 2019-10-08 Created: 2019-10-08 Last updated: 2020-03-05Bibliographically approved
Székely, É., Henter, G. E., Beskow, J. & Gustafson, J. (2019). How to train your fillers: uh and um in spontaneous speech synthesis. In: : . Paper presented at The 10th ISCA Speech Synthesis Workshop.
Open this publication in new window or tab >>How to train your fillers: uh and um in spontaneous speech synthesis
2019 (English)Conference paper, Published paper (Refereed)
National Category
Engineering and Technology
Identifiers
urn:nbn:se:kth:diva-261693 (URN)
Conference
The 10th ISCA Speech Synthesis Workshop
Note

QC 20191011

Available from: 2019-10-10 Created: 2019-10-10 Last updated: 2020-04-27Bibliographically approved
Skantze, G., Gustafson, J. & Beskow, J. (2019). Multimodal Conversational Interaction with Robots. In: Sharon Oviatt, Björn Schuller, Philip R. Cohen, Daniel Sonntag, Gerasimos Potamianos, Antonio Krüger (Ed.), The Handbook of Multimodal-Multisensor Interfaces, Volume 3: Language Processing, Software, Commercialization, and Emerging Directions. ACM Press
Open this publication in new window or tab >>Multimodal Conversational Interaction with Robots
2019 (English)In: The Handbook of Multimodal-Multisensor Interfaces, Volume 3: Language Processing, Software, Commercialization, and Emerging Directions / [ed] Sharon Oviatt, Björn Schuller, Philip R. Cohen, Daniel Sonntag, Gerasimos Potamianos, Antonio Krüger, ACM Press, 2019Chapter in book (Refereed)
Place, publisher, year, edition, pages
ACM Press, 2019
National Category
Human Computer Interaction
Identifiers
urn:nbn:se:kth:diva-254650 (URN)9781970001723 (ISBN)
Note

QC 20190821

Available from: 2019-07-02 Created: 2019-07-02 Last updated: 2019-08-21Bibliographically approved
Székely, É., Henter, G. E., Beskow, J. & Gustafson, J. (2019). Off the cuff: Exploring extemporaneous speech delivery with TTS. In: : . Paper presented at Interspeech.
Open this publication in new window or tab >>Off the cuff: Exploring extemporaneous speech delivery with TTS
2019 (English)Conference paper, Published paper (Refereed)
National Category
Engineering and Technology
Identifiers
urn:nbn:se:kth:diva-261691 (URN)
Conference
Interspeech
Note

QC 20191011

Available from: 2019-10-10 Created: 2019-10-10 Last updated: 2019-10-11Bibliographically approved
Székely, É., Henter, G. E., Beskow, J. & Gustafson, J. (2019). Off the cuff: Exploring extemporaneous speech delivery with TTS. In: : . Paper presented at The 20th Annual Conference of the International Speech Communication Association INTERSPEECH 2019 | Graz, Austria, Sep. 15-19, 2019. (pp. 3687-3688).
Open this publication in new window or tab >>Off the cuff: Exploring extemporaneous speech delivery with TTS
2019 (English)Conference paper, Published paper (Refereed)
National Category
Computer Sciences Language Technology (Computational Linguistics)
Identifiers
urn:nbn:se:kth:diva-260957 (URN)
Conference
The 20th Annual Conference of the International Speech Communication Association INTERSPEECH 2019 | Graz, Austria, Sep. 15-19, 2019.
Note

QC 20191113

Available from: 2019-09-30 Created: 2019-09-30 Last updated: 2019-11-13Bibliographically approved
Abelho Pereira, A. T., Oertel, C., Fermoselle, L., Mendelson, J. & Gustafson, J. (2019). Responsive Joint Attention in Human-Robot Interaction. In: : . Paper presented at 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 1080-1087).
Open this publication in new window or tab >>Responsive Joint Attention in Human-Robot Interaction
Show others...
2019 (English)Conference paper, Published paper (Refereed)
Abstract [en]

Joint attention has been shown to be not only crucial for human-human interaction but also human-robot interaction. Joint attention can help to make cooperation more efficient, support disambiguation in instances of uncertainty and make interactions appear more natural and familiar. In this paper, we present an autonomous gaze system that uses multimodal perception capabilities to model responsive joint attention mechanisms. We investigate the effects of our system on people’s perception of a robot within a problem-solving task. Results from a user study suggest that responsive joint attention mechanisms evoke higher perceived feelings of social presence on scales that regard the direction of the robot’s perception.

National Category
Interaction Technologies
Identifiers
urn:nbn:se:kth:diva-267228 (URN)10.1109/IROS40897.2019.8968130 (DOI)2-s2.0-85081154022 (Scopus ID)
Conference
2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
Note

QC 20200217

Available from: 2020-02-04 Created: 2020-02-04 Last updated: 2020-05-25Bibliographically approved
Organisations
Identifiers
ORCID iD: ORCID iD iconorcid.org/0000-0002-0397-6442

Search in DiVA

Show all publications