Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Interactive sonification of expressive hand gestures on a handheld device
KTH, School of Computer Science and Communication (CSC), Speech, Music and Hearing, TMH, Music Acoustics.
KTH, School of Computer Science and Communication (CSC), Speech, Music and Hearing, TMH, Music Acoustics.ORCID iD: 0000-0002-3086-0322
KTH, School of Computer Science and Communication (CSC), Speech, Music and Hearing, TMH, Music Acoustics.ORCID iD: 0000-0002-8830-963X
2012 (English)In: Journal on Multimodal User Interfaces, ISSN 1783-7677, E-ISSN 1783-8738, Vol. 6, no 1-2, 49-57 p.Article in journal (Refereed) Published
Abstract [en]

We present here a mobile phone application called MoodifierLive which aims at using expressive music performances for the sonification of expressive gestures through the mapping of the phone’s accelerometer data to the performance parameters (i.e. tempo, sound level, and articulation). The application, and in particular the sonification principle, is described in detail. An experiment was carried out to evaluate the perceived matching between the gesture and the music performance that it produced, using two distinct mappings between gestures and performance. The results show that the application produces consistent performances, and that the mapping based on data collected from real gestures works better than one defined a priori by the authors.

Place, publisher, year, edition, pages
2012. Vol. 6, no 1-2, 49-57 p.
Keyword [en]
Automatic music performance, Emotional hand gestures, Mobile phone, Sonification
National Category
Computer and Information Science Human Computer Interaction Psychology
Identifiers
URN: urn:nbn:se:kth:diva-34084DOI: 10.1007/s12193-011-0076-2ISI: 000309998800006Scopus ID: 2-s2.0-84863200318OAI: oai:DiVA.org:kth-34084DiVA: diva2:419080
Projects
SAME
Funder
Swedish Research Council, 2010-4654EU, FP7, Seventh Framework Programme, FP7-ICT-STREP-215749
Note

QC 20120809. Updated from submitted to published.

Available from: 2011-05-25 Created: 2011-05-25 Last updated: 2017-12-11Bibliographically approved
In thesis
1. Interactive computer-aided expressive music performance: Analysis, control, modification and synthesis
Open this publication in new window or tab >>Interactive computer-aided expressive music performance: Analysis, control, modification and synthesis
2011 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

This thesis describes the design and implementation process of two applications (PerMORFer and MoodifierLive) for the interactive manipulation of music performance. Such applications aim at closing the gap between the musicians, who play the music, and the listeners, who passively listen to it. The goal was to create computer programs that allow the user to actively control how the music is performed. This is achieved by modifying such parameters as tempo, dynamics, and articulation, much like a musician does when playing an instrument. An overview of similar systems and the problems related to their development is given in the first of the included papers.

Four requirements were defined for the applications: (1) to produce a natural, high quality sound; (2) to allow for realistic modifications of the performance parameters; (3) to be easy to control, even for non-musicians; (4) to be portable. Although there are many similarities between PerMORFer and MoodifierLive, the two applications fulfill different requirements. The first two were addressed in PerMORFer, with which the user can manipulate pre-recorded audio performance. The last two were addressed in MoodifierLive, a mobile phone application for gesture-based control of a MIDI score file. The tone-by tone modifications in both applications are based on the KTH rule system for music performance. The included papers describe studies, methods, and algorithms used in the development of the two applications.

Audio recordings of real performance have been used in PerMORFer toachieve a natural sound. The tone-by-tone manipulations defined by the KTH rules first require an analysis of the original performance to separate the tones and estimate their parameters (IOI, duration, dynamics). Available methods were combined with novel solutions, such as an approach to the separation of two overlapping sinusoidal components. On the topic of performance analysis, ad-hoc algorithms were also developed to analyze DJ scratching recordings.

A particularly complex problem is the estimation of a tone’s dynamic level. A study was conducted to identify the perceptual cues that listeners use to determinethe dynamics of a tone. The results showed that timbre is as important as loudness. These findings were applied in a partly unsuccessful attempt to estimate dynamics from spectral features.

The manipulation of tempo is a relatively simple problem, as is that of articulation (i.e. legato-staccato) as long as the tone can be separated. The modification of dynamics on the other hand is more difficult, as was its estimation. Following the findings of the previously mentioned perceptual study, a method to modify both loudness and timbre using a database of spectral models was implemented.

MoodifierLive was used to experiment with performance control interfaces. In particular, the mobile phone’s built-in accelerometer was used to track, analyze, and interpret the movements of the user. Expressive gestures were then mapped to corresponding expressive music performances. Evaluation showed that modes based on natural gestures were easier to use than those created witha top-down approach.

Place, publisher, year, edition, pages
Stockholm: KTH Royal Institute of Technology, 2011. 69 p.
Series
Trita-CSC-A, ISSN 1653-5723 ; 2011:12
National Category
Fluid Mechanics and Acoustics
Identifiers
urn:nbn:se:kth:diva-34099 (URN)978-91-7501-031-1 (ISBN)
Public defence
2011-06-15, F3, Lindstedtsvägen 26, KTH, Stockholm, 10:00 (English)
Opponent
Supervisors
Note
QC 20110607Available from: 2011-06-07 Created: 2011-05-25 Last updated: 2012-03-22Bibliographically approved
2. Interactive sonification of motion: Design, implementation and control of expressive auditory feedback with mobile devices
Open this publication in new window or tab >>Interactive sonification of motion: Design, implementation and control of expressive auditory feedback with mobile devices
2013 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Sound and motion are intrinsically related, by their physical nature and through the link between auditory perception and motor control. If sound provides information about the characteristics of a movement, a movement can also be influenced or triggered by a sound pattern. This thesis investigates how this link can be reinforced by means of interactive sonification. Sonification, the use of sound to communicate, perceptualize and interpret data, can be used in many different contexts. It is particularly well suited for time-related tasks such as monitoring and synchronization, and is therefore an ideal candidate to support the design of applications related to physical training. Our objectives are to develop and investigate computational models for the sonification of motion data with a particular focus on expressive movement and gesture, and for the sonification of elite athletes movements.  We chose to develop our applications on a mobile platform in order to make use of advanced interaction modes using an easily accessible technology. In addition, networking capabilities of modern smartphones potentially allow for adding a social dimension to our sonification applications by extending them to several collaborating users. The sport of rowing was chosen to illustrate the assistance that an interactive sonification system can provide to elite athletes. Bringing into play complex interactions between various kinematic and kinetic quantities, studies on rowing kinematics provide guidelines to optimize rowing efficiency, e.g. by minimizing velocity fluctuations around average velocity. However, rowers can only rely on sparse cues to get information relative to boat velocity, such as the sound made by the water splashing on the hull. We believe that an interactive augmented feedback communicating the dynamic evolution of some kinematic quantities could represent a promising way of enhancing the training of elite rowers. Since only limited space is available on a rowing boat, the use of mobile phones appears appropriate for handling streams of incoming data from various sensors and generating an auditory feedback simultaneously. The development of sonification models for rowing and their design evaluation in offline conditions are presented in Paper I. In Paper II, three different models for sonifying the synchronization of the movements of two users holding a mobile phone are explored. Sonification of expressive gestures by means of expressive music performance is tackled in Paper III. In Paper IV, we introduce a database of mobile applications related to sound and music computing. An overview of the field of sonification is presented in Paper V, along with a systematic review of mapping strategies for sonifying physical quantities. Physical and auditory dimensions were both classified into generic conceptual dimensions, and proportion of use was analyzed in order to identify the most popular mappings. Finally, Paper VI summarizes experiments conducted with the Swedish national rowing team in order to assess sonification models in an interactive context.

Place, publisher, year, edition, pages
Stockholm: KTH Royal Institute of Technology, 2013. xiii, 33 p.
Series
Trita-CSC-A, ISSN 1653-5723 ; 2013:09
National Category
Computer Science Human Computer Interaction
Identifiers
urn:nbn:se:kth:diva-127944 (URN)978-91-7501-858-4 (ISBN)
Public defence
2013-09-27, F3, Lindstedtsvägen 26, KTH, Stockholm, 13:00 (English)
Opponent
Supervisors
Note

QC 20130910

Available from: 2013-09-10 Created: 2013-09-09 Last updated: 2013-09-10Bibliographically approved

Open Access in DiVA

No full text

Other links

Publisher's full textScopus

Authority records BETA

Bresin, RobertoDubus, Gaël

Search in DiVA

By author/editor
Fabiani, MarcoBresin, RobertoDubus, Gaël
By organisation
Music Acoustics
In the same journal
Journal on Multimodal User Interfaces
Computer and Information ScienceHuman Computer InteractionPsychology

Search outside of DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 231 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf