Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Interactive sonification of motion: Design, implementation and control of expressive auditory feedback with mobile devices
KTH, School of Computer Science and Communication (CSC), Speech, Music and Hearing, TMH, Music Acoustics.ORCID iD: 0000-0002-8830-963X
2013 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Sound and motion are intrinsically related, by their physical nature and through the link between auditory perception and motor control. If sound provides information about the characteristics of a movement, a movement can also be influenced or triggered by a sound pattern. This thesis investigates how this link can be reinforced by means of interactive sonification. Sonification, the use of sound to communicate, perceptualize and interpret data, can be used in many different contexts. It is particularly well suited for time-related tasks such as monitoring and synchronization, and is therefore an ideal candidate to support the design of applications related to physical training. Our objectives are to develop and investigate computational models for the sonification of motion data with a particular focus on expressive movement and gesture, and for the sonification of elite athletes movements.  We chose to develop our applications on a mobile platform in order to make use of advanced interaction modes using an easily accessible technology. In addition, networking capabilities of modern smartphones potentially allow for adding a social dimension to our sonification applications by extending them to several collaborating users. The sport of rowing was chosen to illustrate the assistance that an interactive sonification system can provide to elite athletes. Bringing into play complex interactions between various kinematic and kinetic quantities, studies on rowing kinematics provide guidelines to optimize rowing efficiency, e.g. by minimizing velocity fluctuations around average velocity. However, rowers can only rely on sparse cues to get information relative to boat velocity, such as the sound made by the water splashing on the hull. We believe that an interactive augmented feedback communicating the dynamic evolution of some kinematic quantities could represent a promising way of enhancing the training of elite rowers. Since only limited space is available on a rowing boat, the use of mobile phones appears appropriate for handling streams of incoming data from various sensors and generating an auditory feedback simultaneously. The development of sonification models for rowing and their design evaluation in offline conditions are presented in Paper I. In Paper II, three different models for sonifying the synchronization of the movements of two users holding a mobile phone are explored. Sonification of expressive gestures by means of expressive music performance is tackled in Paper III. In Paper IV, we introduce a database of mobile applications related to sound and music computing. An overview of the field of sonification is presented in Paper V, along with a systematic review of mapping strategies for sonifying physical quantities. Physical and auditory dimensions were both classified into generic conceptual dimensions, and proportion of use was analyzed in order to identify the most popular mappings. Finally, Paper VI summarizes experiments conducted with the Swedish national rowing team in order to assess sonification models in an interactive context.

Place, publisher, year, edition, pages
Stockholm: KTH Royal Institute of Technology, 2013. , xiii, 33 p.
Series
Trita-CSC-A, ISSN 1653-5723 ; 2013:09
National Category
Computer Science Human Computer Interaction
Identifiers
URN: urn:nbn:se:kth:diva-127944ISBN: 978-91-7501-858-4 (print)OAI: oai:DiVA.org:kth-127944DiVA: diva2:646701
Public defence
2013-09-27, F3, Lindstedtsvägen 26, KTH, Stockholm, 13:00 (English)
Opponent
Supervisors
Note

QC 20130910

Available from: 2013-09-10 Created: 2013-09-09 Last updated: 2013-09-10Bibliographically approved
List of papers
1. Evaluation of four models for the sonification of elite rowing
Open this publication in new window or tab >>Evaluation of four models for the sonification of elite rowing
2012 (English)In: Journal on Multimodal User Interfaces, ISSN 1783-7677, E-ISSN 1783-8738, Vol. 5, no 3-4, 143-156 p.Article in journal (Refereed) Published
Abstract [en]

Many aspects of sonification represent potential benefits for the practice of sports. Taking advantage of the characteristics of auditory perception, interactive sonification offers promising opportunities for enhancing the training of athletes. The efficient learning and memorizing abilities pertaining to the sense of hearing, together with the strong coupling between auditory and sensorimotor systems, make the use of sound a natural field of investigation in quest of efficiency optimization in individual sports at a high level. This study presents an application of sonification to elite rowing, introducing and evaluating four sonification models.The rapid development of mobile technology capable of efficiently handling numerical information offers new possibilities for interactive auditory display. Thus, these models have been developed under the specific constraints of a mobile platform, from data acquisition to the generation of a meaningful sound feedback. In order to evaluate the models, two listening experiments have then been carried out with elite rowers. Results show a good ability of the participants to efficiently extract basic characteristics of the sonified data, even in a non-interactive context. Qualitative assessment of the models highlights the need for a balance between function and aesthetics in interactive sonification design. Consequently, particular attention on usability is required for future displays to become widespread.

Place, publisher, year, edition, pages
Springer, 2012
Keyword
Sonification, Rowing, Sculler, Sports, Accelerometer
National Category
Computer Science Language Technology (Computational Linguistics)
Identifiers
urn:nbn:se:kth:diva-52253 (URN)10.1007/s12193-011-0085-1 (DOI)000309998300007 ()2-s2.0-84860990213 (Scopus ID)
Funder
Swedish Research CouncilEU, FP7, Seventh Framework Programme, FP7-ICT- STREP-215749
Note

QC 20120628

Available from: 2012-04-23 Created: 2011-12-14 Last updated: 2017-12-08Bibliographically approved
2. Interactive sonification of synchronisation of motoric behaviour in social active listening to music with mobile devices
Open this publication in new window or tab >>Interactive sonification of synchronisation of motoric behaviour in social active listening to music with mobile devices
Show others...
2012 (English)In: Journal on Multimodal User Interfaces, ISSN 1783-7677, E-ISSN 1783-8738, Vol. 5, no 3-4, 157-173 p.Article in journal (Refereed) Published
Abstract [en]

This paper evaluates three different interactive sonifications of dyadic coordinated human rhythmic activity. An index of phase synchronisation of gestures was chosen as coordination metric. The sonifications are implemented as three prototype applications exploiting mobile devices: Sync’n’Moog, Sync’n’Move, and Sync’n’Mood. Sync’n’Moog sonifies the phase synchronisation index by acting directly on the audio signal and applying a nonlinear time-varying filtering technique. Sync’n’Move intervenes on the multi-track music content by making the single instruments emerge and hide. Sync’n’Mood manipulates the affective features of the music performance. The three sonifications were also tested against a condition without sonification.

Place, publisher, year, edition, pages
Springer Berlin/Heidelberg, 2012
Keyword
Interactive sonification, Interactive systems, Audio systems, Sound and music computing, Active music listening, Synchronisation
National Category
Computer Science Human Computer Interaction Psychology Media and Communication Technology
Identifiers
urn:nbn:se:kth:diva-52200 (URN)10.1007/s12193-011-0079-z (DOI)000309998300008 ()2-s2.0-84861014654 (Scopus ID)
Projects
SAME
Funder
EU, FP7, Seventh Framework Programme, 215749 SAMESwedish Research Council, 2010-4654
Note

QC 20150623

Available from: 2011-12-14 Created: 2011-12-14 Last updated: 2017-12-08Bibliographically approved
3. Interactive sonification of expressive hand gestures on a handheld device
Open this publication in new window or tab >>Interactive sonification of expressive hand gestures on a handheld device
2012 (English)In: Journal on Multimodal User Interfaces, ISSN 1783-7677, E-ISSN 1783-8738, Vol. 6, no 1-2, 49-57 p.Article in journal (Refereed) Published
Abstract [en]

We present here a mobile phone application called MoodifierLive which aims at using expressive music performances for the sonification of expressive gestures through the mapping of the phone’s accelerometer data to the performance parameters (i.e. tempo, sound level, and articulation). The application, and in particular the sonification principle, is described in detail. An experiment was carried out to evaluate the perceived matching between the gesture and the music performance that it produced, using two distinct mappings between gestures and performance. The results show that the application produces consistent performances, and that the mapping based on data collected from real gestures works better than one defined a priori by the authors.

Keyword
Automatic music performance, Emotional hand gestures, Mobile phone, Sonification
National Category
Computer and Information Science Human Computer Interaction Psychology
Identifiers
urn:nbn:se:kth:diva-34084 (URN)10.1007/s12193-011-0076-2 (DOI)000309998800006 ()2-s2.0-84863200318 (Scopus ID)
Projects
SAME
Funder
Swedish Research Council, 2010-4654EU, FP7, Seventh Framework Programme, FP7-ICT-STREP-215749
Note

QC 20120809. Updated from submitted to published.

Available from: 2011-05-25 Created: 2011-05-25 Last updated: 2017-12-11Bibliographically approved
4. An overview of sound and music applications for Android available on the market
Open this publication in new window or tab >>An overview of sound and music applications for Android available on the market
2012 (English)In: Proceedings of the 9th Sound and Music Computing Conference, SMC 2012 / [ed] Serafin, Stefania, Sound and music Computing network , 2012, 541-546 p.Conference paper, Published paper (Refereed)
Abstract [en]

This paper introduces a database of sound-based applications running on the Android mobile platform. The longterm objective is to provide a state-of-the-art of mobile applications dealing with sound and music interaction. After exposing the method used to build up and maintain the database using a non-hierarchical structure based on tags, we present a classification according to various categories of applications, and we conduct a preliminary analysis of the repartition of these categories reflecting the current state of the database.

Place, publisher, year, edition, pages
Sound and music Computing network, 2012
Keyword
Android (operating system), Mobile applications, Mobile platform, Music applications, Music interaction, Preliminary analysis, Structure-based
National Category
Computer Science Language Technology (Computational Linguistics)
Identifiers
urn:nbn:se:kth:diva-109381 (URN)2-s2.0-84905197265 (Scopus ID)978-383253180-5 (ISBN)
Conference
9th Sound and Music Computing Conference, SMC 2012, Copenhagen, Denmark, 11 July 2012 through 14 July 2012
Note

QC 20150507. QC 20160115

Available from: 2013-01-02 Created: 2013-01-02 Last updated: 2016-01-15Bibliographically approved
5. A Systematic Review of Mapping Strategies for the Sonification of Physical Quantities
Open this publication in new window or tab >>A Systematic Review of Mapping Strategies for the Sonification of Physical Quantities
2013 (English)In: PLoS ONE, ISSN 1932-6203, E-ISSN 1932-6203, Vol. 8, no 12, e82491- p.Article in journal (Refereed) Published
Abstract [en]

The field of sonification has progressed greatly over the past twenty years and currently constitutes an established area of research. This article aims at exploiting and organizing the knowledge accumulated in previous experimental studies to build a foundation for future sonification works. A systematic review of these studies may reveal trends in sonification design, and therefore support the development of design guidelines. To this end, we have reviewed and analyzed 179 scientific publications related to sonification of physical quantities. Using a bottom-up approach, we set up a list of conceptual dimensions belonging to both physical and auditory domains. Mappings used in the reviewed works were identified, forming a database of 495 entries. Frequency of use was analyzed among these conceptual dimensions as well as higher-level categories. Results confirm two hypotheses formulated in a preliminary study: pitch is by far the most used auditory dimension in sonification applications, and spatial auditory dimensions are almost exclusively used to sonify kinematic quantities. To detect successful as well as unsuccessful sonification strategies, assessment of mapping efficiency conducted in the reviewed works was considered. Results show that a proper evaluation of sonification mappings is performed only in a marginal proportion of publications. Additional aspects of the publication database were investigated: historical distribution of sonification works is presented, projects are classified according to their primary function, and the sonic material used in the auditory display is discussed. Finally, a mapping-based approach for characterizing sonification is proposed.

Keyword
sonification, theory, mapping, meta-analysis, review, physical, auditory, dimension
National Category
Computer Science Human Computer Interaction
Identifiers
urn:nbn:se:kth:diva-127939 (URN)10.1371/journal.pone.0082491 (DOI)000328737700019 ()2-s2.0-84893070216 (Scopus ID)
Projects
SOM - The sound of motion: Providing sound feedback to human movements
Funder
Swedish Research Council, 2010-4654
Note

QC 20140123. Updated from manuscript to article in journal.

Available from: 2013-09-09 Created: 2013-09-09 Last updated: 2017-12-06Bibliographically approved
6. Evaluation of a system for the sonification of elite rowing in an interactive context
Open this publication in new window or tab >>Evaluation of a system for the sonification of elite rowing in an interactive context
(English)Manuscript (preprint) (Other academic)
Keyword
sonification, rowing, evaluation, auditory display, sport
National Category
Computer Science Sport and Fitness Sciences Human Computer Interaction
Identifiers
urn:nbn:se:kth:diva-127943 (URN)
Projects
SOM - The sound of motion: Providing sound feedback to human movementsSONEA - Sonification of elite athletes
Funder
Swedish Research Council, 2010-4654
Note

QS 2013

Available from: 2013-09-09 Created: 2013-09-09 Last updated: 2013-09-10Bibliographically approved

Open Access in DiVA

fulltext(4682 kB)718 downloads
File information
File name FULLTEXT01.pdfFile size 4682 kBChecksum SHA-512
0eb7de216c8cf7494b4764088450cf37b8e1473f431bd2c85788d5c24fd201676fdd7758f51c822f79b6119a2a62ab173c7b2c87168dacd566084791c2c46bc5
Type fulltextMimetype application/pdf

Authority records BETA

Dubus, Gaël

Search in DiVA

By author/editor
Dubus, Gaël
By organisation
Music Acoustics
Computer ScienceHuman Computer Interaction

Search outside of DiVA

GoogleGoogle Scholar
Total: 718 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 792 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf