kth.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Socio-cultural perception of robot backchannels
KTH, School of Electrical Engineering and Computer Science (EECS), Intelligent systems, Speech, Music and Hearing, TMH, Speech Communication and Technology.ORCID iD: 0000-0003-4532-014X
KTH, School of Electrical Engineering and Computer Science (EECS), Intelligent systems, Speech, Music and Hearing, TMH.ORCID iD: 0000-0003-4472-4732
2023 (English)In: Frontiers in Robotics and AI, E-ISSN 2296-9144, Vol. 10Article in journal (Refereed) Published
Abstract [en]

Introduction: Backchannels, i.e., short interjections by an interlocutor to indicate attention, understanding or agreement regarding utterances by another conversation participant, are fundamental in human-human interaction. Lack of backchannels or if they have unexpected timing or formulation may influence the conversation negatively, as misinterpretations regarding attention, understanding or agreement may occur. However, several studies over the years have shown that there may be cultural differences in how backchannels are provided and perceived and that these differences may affect intercultural conversations. Culturally aware robots must hence be endowed with the capability to detect and adapt to the way these conversational markers are used across different cultures. Traditionally, culture has been defined in terms of nationality, but this is more and more considered to be a stereotypic simplification. We therefore investigate several socio-cultural factors, such as the participants’ gender, age, first language, extroversion and familiarity with robots, that may be relevant for the perception of backchannels.

Methods: We first cover existing research on cultural influence on backchannel formulation and perception in human-human interaction and on backchannel implementation in Human-Robot Interaction. We then present an experiment on second language spoken practice, in which we investigate how backchannels from the social robot Furhat influence interaction (investigated through speaking time ratios and ethnomethodology and multimodal conversation analysis) and impression of the robot (measured by post-session ratings). The experiment, made in a triad word game setting, is focused on if activity-adaptive robot backchannels may redistribute the participants’ speaking time ratio, and/or if the participants’ assessment of the robot is influenced by the backchannel strategy. The goal is to explore how robot backchannels should be adapted to different language learners to encourage their participation while being perceived as socio-culturally appropriate.

Results: We find that a strategy that displays more backchannels towards a less active speaker may substantially decrease the difference in speaking time between the two speakers, that different socio-cultural groups respond differently to the robot’s backchannel strategy and that they also perceive the robot differently after the session.

Discussion: We conclude that the robot may need different backchanneling strategies towards speakers from different socio-cultural groups in order to encourage them to speak and have a positive perception of the robot.

 

Place, publisher, year, edition, pages
Frontiers Media S.A., 2023. Vol. 10
Keywords [en]
backchannels, multiparty HRI, robot-assisted language learning, spoken practice, cultural effects
National Category
Human Computer Interaction
Research subject
Computer Science
Identifiers
URN: urn:nbn:se:kth:diva-323334DOI: 10.3389/frobt.2023.988042ISI: 000935012900001PubMedID: 36777379Scopus ID: 2-s2.0-85147686864OAI: oai:DiVA.org:kth-323334DiVA, id: diva2:1731278
Funder
Marcus and Amalia Wallenberg Foundation, 2020.0052
Note

QC 20230130

Available from: 2023-01-26 Created: 2023-01-26 Last updated: 2024-02-26Bibliographically approved
In thesis
1. Robots Beyond Borders: The Role of Social Robots in Spoken Second Language Practice
Open this publication in new window or tab >>Robots Beyond Borders: The Role of Social Robots in Spoken Second Language Practice
2024 (English)Doctoral thesis, comprehensive summary (Other academic)
Alternative title[sv]
Robotar bortom gränser : Sociala robotars roll i talat andraspråk
Abstract [en]

This thesis investigates how social robots can support adult second language (L2) learners in improving conversational skills. It recognizes the challenges inherent in adult L2 learning, including increased cognitive demands and the unique motivations driving adult education. While social robots hold potential for natural interactions and language education, research into conversational skill practice with adult learners remains underexplored. Thus, the thesis contributes to understanding these conversational dynamics, enhancing speaking practice, and examining cultural perspectives in this context.

To begin, this thesis investigates robot-led conversations with L2 learners, examining how learners respond to moments of uncertainty. The research reveals that when faced with uncertainty, learners frequently seek clarification, yet many remain unresponsive. As a result, effective strategies are required from robot conversational partners to address this challenge. These interactions are then used to evaluate the performance of off-the-shelf Automatic Speech Recognition (ASR) systems. The assessment highlights that speech recognition for L2 speakers is not as effective as for L1 speakers, with performance deteriorating for both groups during social conversations. Addressing these challenges is imperative for the successful integration of robots in conversational practice with L2 learners.

The thesis then explores the potential advantages of employing social robots in collaborative learning environments with multi-party interactions. It delves into strategies for improving speaking practice, including the use of non-verbal behaviors to encourage learners to speak. For instance, a robot's adaptive gazing behavior is used to effectively balance speaking contributions between L1 and L2 pairs of participants. Moreover, an adaptive use of encouraging backchannels significantly increases the speaking time of L2 learners.

Finally, the thesis highlights the importance of further research on cultural aspects in human-robot interactions. One study reveals distinct responses among various socio-cultural groups in interaction between L1 and L2 participants. For example, factors such as gender, age, extroversion, and familiarity with robots influence conversational engagement of L2 speakers. Additionally, another study investigates preconceptions related to the appearance and accents of nationality-encoded (virtual and physical) social robots. The results indicate that initial perceptions may lead to negative preconceptions, but that these perceptions diminish after actual interactions.

Despite technical limitations, social robots provide distinct benefits in supporting educational endeavors. This thesis emphasizes the potential of social robots as effective facilitators of spoken language practice for adult learners, advocating for continued exploration at the intersection of language education, human-robot interaction, and technology.

Abstract [sv]

Denna avhandling undersöker hur sociala robotar kan ge vuxna andraspråks\-inlärare stöd att förbättra sin konversationsförmåga på svenska. Andraspråks\-inlärning för vuxna, särskilt i migrationskontext, är mer komplext än för barn, bland annat på grund av att förutsättningarna för språkinlärning försämras med åren och att drivkrafterna ofta är andra. Sociala robotar har stor potential inom språkundervisning för att träna naturliga samtal, men fortfarande har lite forskning om hur robotar kan öva konversation med vuxna elever genomförts. Därför bidrar avhandlingen till att förstå samtal mellan andraspråksinlärare och robotar, förbättra dessa samtalsövningar och undersöka hur kulturella faktorer påverkar interaktionen.

Till att börja med undersöker avhandlingen hur andraspråkselever reagerar då de blir förbryllade eller osäkra i robotledda konversationsövningar. Resultaten visar att eleverna ofta försöker få roboten att ge förtydliganden när de är osäkra, men att de ibland helt enkelt inte svarar något alls, vilket innebär att roboten behöver kunna hantera sådana situationer. Konversationerna mellan andraspråksinlärare och en robot har även använts för att undersöka hur väl ledande system för taligenkänning kan tolka det adraspråkstalare säger. Det kan konstateras att systemen har väsentligt större svårigheter att känna igen andraspråkstalare än personer med svensk bakgrund, samt att de har utmananingar att tolka såväl svenska talare som andraspråkselever i friare sociala konversationer, vilket måste hanteras när robotar ska användas i samtalsövningar med andraspråkselever.

Avhandlingen undersöker sedan strategier för att uppmuntra andraspråks\-elever att prata mer och för att fördela ordet jämnare i trepartsövningar där två personer samtalar med roboten. Strategierna går ut på att modifiera hur roboten tittar på de två personerna eller ger icke-verbal återkoppling (hummanden) för att signalera förståelse och intresse för det eleverna säger.

Slutligen belyser avhandlingen vikten av ytterligare forskning om kulturella aspekter i interaktioner mellan människa och robot. En studie visar att faktorer som kön, ålder, tidigare erfarenhet av robotar och hur extrovert eleven är påverkar både hur mycket olika personer talar och hur de svarar på robotens försök att uppmuntra dem att tala mer genom icke-verbala signaler.

En andra studie undersöker om och hur förutfattade meningar relaterade till utseende och uttal påverkar hur människor uppfattar (virtuella och fysiska) sociala robotar som givits egenskaper (röst och ansikte) som kan kopplas till olika nationella bakgrunder. Resultaten visar att människors första intryck av en kulturellt färgad robot speglar förutfattade meningar, men att denna uppfattning inte alls får samma genomslag när personer faktiskt interagerat med roboten i ett realistiskt sammanhang.

En huvudsaklig slutsats i avhandlingen är att sociala robotar, trots att tekniska begränsningar finns kvar, har tydliga fördelar som kan utnyttjas inom utbildning. Specifikt betonar avhandlingen potentialen hos sociala robotar att leda samtalsövningar för vuxna andraspråkselever och förespråkar fortsatt forskning i skärningspunkten mellan språkundervisning, människa-robotinteraktion och teknik.

Place, publisher, year, edition, pages
Stockholm: KTH Royal Institute of Technology, 2024. p. 91
Series
TRITA-EECS-AVL ; 2024:23
Keywords
Conversations, gaze, backchannels, multi-party, accent, culture, Samtal, blick, återkoppling, gruppdynamik, brytning, kultur
National Category
Robotics and automation Natural Language Processing
Research subject
Speech and Music Communication
Identifiers
urn:nbn:se:kth:diva-343863 (URN)978-91-8040-858-5 (ISBN)
Public defence
2024-03-22, https://kth-se.zoom.us/j/65591848998, F3, Lindstedtsvägen 26, Stockholm, 10:00 (English)
Opponent
Supervisors
Note

QC 20240226

Available from: 2024-02-26 Created: 2024-02-26 Last updated: 2025-02-05Bibliographically approved

Open Access in DiVA

fulltext(8580 kB)475 downloads
File information
File name FULLTEXT01.pdfFile size 8580 kBChecksum SHA-512
5aebd32a1b2026f75257edf86eac4e32ac95fb76367737a63be0a3b0bc2ff191ae7cfee1d589abc65ff6392561f886c7ee30ea71158db747d853e1bd0c3abb72
Type fulltextMimetype application/pdf

Other links

Publisher's full textPubMedScopus

Authority records

Engwall, OlovCumbal, Ronald

Search in DiVA

By author/editor
Engwall, OlovCumbal, RonaldMajlesi, Ali Reza
By organisation
Speech Communication and TechnologySpeech, Music and Hearing, TMH
In the same journal
Frontiers in Robotics and AI
Human Computer Interaction

Search outside of DiVA

GoogleGoogle Scholar
Total: 476 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

doi
pubmed
urn-nbn

Altmetric score

doi
pubmed
urn-nbn
Total: 479 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf