kth.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
PepperOSC: enabling interactive sonification of a robot's expressive movement
KTH, School of Electrical Engineering and Computer Science (EECS), Human Centered Technology, Media Technology and Interaction Design, MID. (Sound and Music Computing)ORCID iD: 0000-0003-3572-6429
KTH, School of Electrical Engineering and Computer Science (EECS), Human Centered Technology, Media Technology and Interaction Design, MID. (Sound and Music Computing)ORCID iD: 0000-0002-3086-0322
2023 (English)In: Journal on Multimodal User Interfaces, ISSN 1783-7677, E-ISSN 1783-8738, Vol. 17, no 4, p. 231-239Article in journal (Refereed) Published
Abstract [en]

This paper presents the design and development of PepperOSC, an interface that connects Pepper and NAO robots with soundproduction tools to enable the development of interactive sonification in human-robot interaction (HRI). The interface usesOpen Sound Control (OSC) messages to stream kinematic data from robots to various sound design and music productiontools. The goals of PepperOSC are twofold: (i) to provide a tool for HRI researchers in developing multimodal user interfacesthrough sonification, and (ii) to lower the barrier for sound designers to contribute to HRI. To demonstrate the potential useof PepperOSC, this paper also presents two applications we have conducted: (i) a course project by two master’s studentswho created a robot sound model in Pure Data, and (ii) a museum installation of Pepper robot, employing sound modelsdeveloped by a sound designer and a composer/researcher in music technology usingMaxMSP and SuperCollider respectively.Furthermore, we discuss the potential use cases of PepperOSC in social robotics and artistic contexts. These applicationsdemonstrate the versatility of PepperOSC and its ability to explore diverse aesthetic strategies for robot movement sonification,offering a promising approach to enhance the effectiveness and appeal of human-robot interactions.

Place, publisher, year, edition, pages
Springer Nature , 2023. Vol. 17, no 4, p. 231-239
Keywords [en]
Interactive sonification, human-robot interaction, robot sound design
National Category
Robotics Media and Communication Technology Human Computer Interaction
Research subject
Art, Technology and Design
Identifiers
URN: urn:nbn:se:kth:diva-342756DOI: 10.1007/s12193-023-00414-9ISI: 001061715800001Scopus ID: 2-s2.0-85176385871OAI: oai:DiVA.org:kth-342756DiVA, id: diva2:1832789
Projects
SONAO
Funder
Swedish Research Council, 2017-03979NordForsk, 86892
Note

QC 20240130

Available from: 2024-01-30 Created: 2024-01-30 Last updated: 2024-01-30Bibliographically approved
In thesis
1. From Motion Pictures to Robotic Features: Adopting film sound design practices to foster sonic expression in social robotics through interactive sonification
Open this publication in new window or tab >>From Motion Pictures to Robotic Features: Adopting film sound design practices to foster sonic expression in social robotics through interactive sonification
2024 (English)Doctoral thesis, comprehensive summary (Other academic) [Artistic work]
Alternative title[sv]
Från filmer till robotfunktioner : Användning av praxis inom filmljuddesign för att främja ljuduttryck i social robotik genom interaktiv sonifiering
Abstract [en]

This dissertation investigates the role of sound design in social robotics, drawing inspiration from robot depictions in science-fiction films. It addresses the limitations of robots’ movements and expressive behavior by integrating principles from film sound design, seeking to improve human-robot interaction through expressive gestures and non-verbal sounds.

The compiled works are structured into two parts. The first part focuses on perceptual studies, exploring how people perceive non-verbal sounds displayed by a Pepper robot related to its movement. These studies highlighted preferences for more refined sound models, subtle sounds that blend with ambient sounds, and sound characteristics matching the robot’s visual attributes. This part also resulted in a programming interface connecting the Pepper robot with sound production tools.

The second part focuses on a structured analysis of robot sounds in films, revealing three narrative themes related to robot sounds in films with implications for social robotics. The first theme involves sounds associated with the physical attributes of robots, encompassing sub-themes of sound linked to robot size, exposed mechanisms, build quality, and anthropomorphic traits. The second theme delves into sounds accentuating robots’ internal workings, with sub-themes related to learning and decision-making processes. Lastly, the third theme revolves around sounds utilized in robots’ interactions with other characters within the film scenes.

Based on these works, the dissertation discusses sound design recommendations for social robotics inspired by practices in film sound design. These recommendations encompass selecting the appropriate sound materials and sonic characteristics such as pitch and timbre, employing movement sound for effective communication and emotional expression, and integrating narrative and context into the interaction.

Abstract [sv]

Denna avhandling undersöker ljuddesignens roll i social robotik, med inspiration från robotskildringar i science fiction filmer. Avhandlingen diskuterar begränsningar i robotars uttrycksfulla beteenden genom att integrera principer från filmljuddesign. Arbetet syftar till att främja interaktionen mellan människa och robot genom att förse robotar med uttrycksfulla gester och icke-verbala ljud.

Denna sammanläggningsavhandling inkluderar ett antal artiklar som är strukturerade i två separata delar. Den första delen fokuserar på perceptuella studier och undersöker hur människor uppfattar de icke-verbala ljud som roboten Pepper producerar i samband med sina rörelser. Dessa studier belyste preferenser för mer förfinade ljudmodeller, subtila ljud som blandas med omgivande ljud, och ljudegenskaper som matchar robotens visuella attribut. Denna del resulterade också i ett programmeringsgränssnitt som sammankopplar Pepper-roboten och ljudproduktionsverktyg.

Den andra delen fokuserar på en strukturerad analys av robotljud i filmer och avslöjar tre narrativa teman relaterade till robotljud i filmer med implikationer för social robotik. Det första temat handlar om ljud som förknippas med robotarnas fysiska attribut och omfattar underteman av ljud som är kopplade till robotstorlek, exponerade mekanismer, byggkvalitet, och antropomorfa drag. Det andra temat fördjupar sig i ljud som betonar robotarnas interna arbete, med underteman relaterade till inlärnings- och beslutsprocesser. Slutligen kretsar det tredje temat kring ljud som används i robotarnas interaktion med andra karaktärer i filmscenerna.

Baserat på ovan beskrivna arbeten diskuterar denna avhandling rekommendationer för ljuddesign inom social robotik inspirerade av praxis inom filmljuddesign. Dessa rekommendationer omfattar att välja lämpliga ljudmaterial och ljudegenskaper såsom tonhöjd och klangfärg, att använda rörelseljud för effektiv kommunikation och känslomässiga uttryck, samt att integrera narrativ och sammanhang i interaktionen.

Place, publisher, year, edition, pages
Stockholm, Sweden: KTH Royal Institute of Technology, 2024. p. xiii, 54
Series
TRITA-EECS-AVL ; 2024:13
Keywords
human-robot interaction, social robotics, film sound design, robot sound, interactive sonification, människa-robotinteraktion, social robotik, filmljuddesign, robotljud, interaktiv sonifiering
National Category
Robotics Human Computer Interaction Studies on Film
Research subject
Media Technology
Identifiers
urn:nbn:se:kth:diva-342759 (URN)978-91-8040-831-8 (ISBN)
Public defence
2024-02-22, https://kth-se.zoom.us/j/61765490226, Kollegiesalen, Brinellvägen 8, Stockholm, 10:00 (English)
Opponent
Supervisors
Note

QC 20240131

Available from: 2024-01-31 Created: 2024-01-30 Last updated: 2024-02-05Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopusPublisher

Search in DiVA

By author/editor
Latupeirissa, Adrian B.Bresin, Roberto
By organisation
Media Technology and Interaction Design, MID
In the same journal
Journal on Multimodal User Interfaces
RoboticsMedia and Communication TechnologyHuman Computer Interaction

Search outside of DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 71 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf