Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart...

Motion–Sound Mapping by Demonstration

Jules Françoise

{Sound Music Movement} Interaction

UMR STMS

—

IRCAM, CNRS, UPMC

Jury:

Catherine Achard

Olivier Chapuis

Thierry Dutoit

Rebecca Fiebrink

Sergi Jordà

Marcelo Wanderley

UPMC

INRIA, Orsay

U. of Mons

Goldsmiths U. of London

UPF, Barcelona

U. McGill, Montreal

(examinatrice)

(examinateur)

(Rapporteur)

(examinatrice)

(examinateur)

(Rapporteur)

Advisors:

Frédéric Bevilacqua (Ircam)

Thierry Artières (UPMC, AMU)

phd defense — Ircam — 18/03/2015

2

Augmented Violin

Pedagogy

New Interfaces

Gaming

Music Games

!

May 13, 2011

For$a$Violinist,$Success$Means$A$New$Low$Point$By MATTHEW GUREWITSCH$

!

Librado Romero/The New York Times More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of �subharmonics,� the violinist Mari Kimura has been exploring the new horizons opened up by an electronic motion sensor, a half-glove equipped with electrodes that monitor the angle and speed of her bowing arm.

!

SINCE!Pythagoras,!musicians!and!scientists!have!known!(or!thought!they!knew)!that!the!lowest!pitch!a!string!stretched!taut!can!produce!�!the!fundamental!�!is!the!pitch!it!�� d!the!vibrating!section!sounds!at!a!higher!pitch.!Theoretically,!there!is!no!ceiling.!As!with!�� !

For!the!last!five!centuries,!give!or!take,!the!range!of!a!violin!bottomed!out!on!the!G!below!middle!C,!the!pitch!of!the!open!G!string.!But!fiddlers!are!not!like!dancers!after!all.!For!nearly!two!decades,!the!Japanese!violinist!Mari!Kimura,!48,!has!been!exploring!unsuspected!subterranean!sounds!as!much!as!an!octave!deeper.!!

Complementing!the!familiar!concept!of!harmonics!(pitches!drawn!from!the!overtones!of!a!given!fundamental),!Ms.!Kimura!has!named!her!freshly!discovered!sonorities!�� astered!the!subharmonic!octave,!third,!second!and!

Movement Learning

Music Rehabilitation

Background | {Sound Music Movement} Interaction team

2

3

Motion & Sound

Relationships

mapping

44

outline

Related Work

Mapping-by-Demonstration

Probabilistic Movement Models

Probabilistic Models for Sound Parameter Generation

Playing Sound Textures

Motion–Sound Interaction through Vocalization

Background & Concepts

Models

Applications

55

outline

Related Work







Models

Applications

Sound Synthesis

Motion Capture

Motion Param.

SoundParam.

6

Related Work | motion-sound mapping: from wires to models

feedback

?

Hunt, A., Wanderley, M. M., & Kirk, R. (2000). Towards a Model for Instrumental Mapping in Expert Musical Interaction. In Proceedings of the 2000 International Computer Music Conference.

Explicit Mapping

Sound Synthesis

Motion Capture

Motion Param.

SoundParam.

6


feedback

Motion parameters directly wired to sound control parameters

Hunt, A., Wanderley, M. M., & Kirk, R. (2000). Towards a Model for Instrumental Mapping in Expert Musical Interaction. In Proceedings of the 2000 International Computer Music Conference.

Implicit Mapping

Use an intermediate model of interaction

> Physical models

> Geometrical approaches

> Machine Learning

Sound Synthesis

Motion Capture

Motion Param.

SoundParam.

Mapping Model

7


feedback

[Momeni et al., 2006]

[Van Nort et al., 2014]

[Caramiaux, 2013]

Machine Learning

8

Discrete gesture recognition

Continuous gesture recognition & temporal mapping

Regression


Sound Synthesis

Motion Capture

Motion Param.

SoundParam.

Mapping Model

User Examples

feedback

[Gillian, 2011]

[Bevilacqua al., 2011]

[Caramiaux, 2013]

Related Work | interactive machine learning

9

Machine Learning

Fiebrink, R. A. (2011). Real-time Human Interaction with Supervised Learning Algorithms for Music Composition and Performance. PhD Dissertation.


9

Machine Learning

Training DataMachine Learning

Model

training



9

Machine Learning


Model

training

new input

predictions



9

Machine Learning


Model

training

new input

predictions

evaluate results



9

Interactive Machine Learning

create training

examples


Model

training

new input

predictions

adjust parameters

& train

evaluate results



9


create training

examples

adjust parameters

& train

evaluate results


1010

outline

Related Work







Models

Applications

11

Related Work | mapping through listening

Caramiaux, B., Françoise, J., Schnell, N., & Bevilacqua, F. (2014). Mapping Through Listening. Computer Music Journal, 38(3), 34–48.

Sound Synthesis

Motion Capture

Motion Param.

SoundParam.

feedback

?

11

Related Work | mapping through listening

listening as a starting point for mapping design

Caramiaux, B., Françoise, J., Schnell, N., & Bevilacqua, F. (2014). Mapping Through Listening. Computer Music Journal, 38(3), 34–48.

Sound Synthesis

Motion Capture

Motion Param.

SoundParam.

?

listening


=

mapping through listening

+

interactive machine learning

Robot Programming by Demonstration

Billard, A., Calinon, S., Dillmann, R., & Schaal, S. (2008). Robot programming by demonstration. Handbook of Robotics.

14

Mapping by Demonstration | Overview

demonstration performance

14


Listening


14


Listening

Acting


14


Listening

Acting


Mappingtraining

14


Listening

Acting


Mappingtraining

Performance

14


Listening

Acting


Mappingtraining

Performance

Feedback

14


Feedback

Listening

Acting

Performing

training

example

Françoise, J., Schnell, N., & Bevilacqua, F. (2013). A Multimodal Probabilistic Model for Gesture-based Control of Sound Synthesis. In Proceedings of the 21st ACM international conference on Multimedia (MM’13) (pp. 705–708). Barcelona, Spain.

16

Mapping by Demonstration | requirements

quick training

from 1 to several examples

continuous & causal inference

(real-time)

Recognition Paradigm

Generation Paradigm

vs

17

Mapping by Demonstration | 2 criteria


Generation Paradigm

vs

17


Probabilistic Models

Synthesisers

modelX

modelY

modelZ

SynthX

SynthY

SynthZ


Generation Paradigm

vs

17


like

liho

od

s

movement parameters


Synthesisers

modelX

modelY

modelZ

SynthX

SynthY

SynthZ

motion


Generation Paradigm

vs

17


generatedsound parameters

like

liho

od

s

movement parameters


Synthesisers

modelX

modelY

modelZ

SynthX

SynthY

SynthZ

motion


Generation Paradigm

Instantaneous

Temporal

vs

vs

17


inputmotion

instantaneous


Generation Paradigm

Instantaneous

Temporal

vs

vs

17


temporal

inputmotion

Recognition Paradigm Generation Paradigm

Inst

an

tan

eo

us

Tem

po

ral

17


18

Mapping by Demonstration | 4 models

Inst

an

tan

eo

us

Tem

po

ral

Gaussian Mixture Models gmm

Gaussian Mixture Regression gmr

Hierarchical Hidden Markov Models

hhmm

Hierarchical Hidden Markov Regression

hhmr

Recognition Paradigm Generation Paradigm

1919

outline

Related Work







GMM HHMM

GMR HHMR

Models

Applications

GMR HHMR

2020

outline

Related Work







GMM HHMM

GMR HHMR

Models

Applications

GMR HHMR

2020

outline

Related Work







GMM HHMM

GMR

HHMR

Models

Applications

2121

outline

Related Work







GMM HHMM

GMR HHMR

Models

Applications

GMR

HHMR

22

Probabilistic Movement Models | Gaussian Mixture Models

time

mo

vem

en

t p

ara

me

ter

mo

tio

n p

ara

me

ter

23

Probabilistic Movement Models | Gaussian Mixture Models

likelihoods

motionparameter

inputmotion

mo

vem

en

t p

ara

me

ter

24

Probabilistic Movement Models | Hidden Markov Models

mo

tio

n p

ara

me

ter

time


25

motionparameter


25

motionparameter

likelihoods

inputmotion


25

motionparameter

likelihoods

inputmotion

time

predictedsound

parameters

time

26

Probabilistic Movement Models | GMMs & HMMs for interaction


Training

> 1 to several examples

User-defined complexity

> number of states/gaussians

26



Training




User-defined Regularization

> numerical errors

> generalization on few examples

26



Training




User-defined Regularization

> numerical errors

> generalization on few examples

Continuous Recognition

> Forward inference in HMMs

27

Probabilistic Movement Models | Hierarchical Hidden Markov Models

Françoise, J., Caramiaux, B., & Bevilacqua, F. (2012). A Hierarchical Approach for the Design of Gesture-to-Sound Mappings. SMC’2012.

27

1 2 3



27

1 2 3

ROOT

transitionprior

exit

hidden state

exit state

SignalLevel

SegmentLevel



1 2 3

ROOT

transitionprior

exit

hidden state

exit state

SignalLevel

SegmentLevel

28


Dynamic Bayesian Networks

Linear time inference

Forward algorithm

> causal estimation of signal & segment state probabilities

29

P A S R

Example gesture

PASR decomposition

Preparation

Attack

Sustain

Release

(P)

(A)

(S)

(R)

> anticipation gesture

> retraction gesture

4 Phases:

Probabilistic Movement Models | HHMM — PASR

Probabilistic Movement Models | HHMM — PASR

30

P A S R

Root

p = 0.5 p = 0.5

p = 1.0 p = 1.0 p = 1.0

p = 0.5

p = 0.5

Exit

p = 1.0

Possibility of authoring the transition parameters

> prior : P et A states

> exit : S et R states

2 playing modes :

> entire gesture with preparation and release

> quick transitions from sustain to attack

31

example: demonstration

32

example: performance

3333

outline

Related Work







GMM HHMM

GMR HHMR

Models

Applications

GMR

HHMR

34

Probabilistic Motion-Sound Models | background: speech & motion

Similar applications from Speech processing

Acoustic-articulatory Inversion

Silent Speech Interfaces

Speech-driven animation

Robotics & Animation

Programming-by-Demonstration (Motor behavior)

Animation

Probabilistic Approaches

GMM-based / HMM-based

[Zhang & Renals, 2008]

[Hueber and Badin, 2011]

[Ding, 2014]

[Calinon, 2010]

[Tilmanne, 2013]

Probabilistic Motion-Sound Models | General Approach

35training performancedemonstration


Motion Features

Sound Features

Joint Features



Motion Features

Sound Features

Joint Features

Training Joint density distribution



Motion Features

Sound Features

Joint Features

Training Joint density distribution

35

Conditional density

distribution

training performancedemonstration

Probabilistic Motion-Sound Models | Gaussian Mixture Regression

36


36

observations

x = [x (m) , x (s)]

x = [x (m) , x (s)]

x = [x (m) , x (s)]


36

observations

x = [x (m) , x (s)]

joint pdf

µk =�µ(m)

k ;µ(s)k

�(1)

�k =��(mm)

k �(ms)k

�(sm)k �(ss)

k

�

(2)

p(x i | θ) =K∑

k=1wkN

(x i ; µk ,Σk

)

x = [x (m) , x (s)]

x = [x (m) , x (s)]

p(x i | θ) =K∑

k=1wkN

(x i ; µk ,Σk

)prior

mean

covariance


36

observations

x = [x (m) , x (s)]

Conditional pdf

p�x (s) | x (m),�

�=

K�

k=1�k (x (m)) N

�x (s) ; µ̂(s)

k (x (m)),�̂(ss)k

�

joint pdf

µk =�µ(m)

k ;µ(s)k

�(1)

�k =��(mm)

k �(ms)k

�(sm)k �(ss)

k

�

(2)

p(x i | θ) =K∑

k=1wkN

(x i ; µk ,Σk

)

x = [x (m) , x (s)]

x = [x (m) , x (s)]

p(x i | θ) =K∑

k=1wkN

(x i ; µk ,Σk

)prior

mean

covariance

responsibility

mean

covariance

�k (x (m))

µ̂(s)k (x (m)) =µ(s)

k +�(sm)k

��(mm)

k

��1 �x (m) �µ(m)

k

�(1)

�̂(ss)k =�(ss)

k ��(sm)k

��(mm)

k

��1�(ms)

k (2)


36

observations

x = [x (m) , x (s)]

x̂ (s) =K�

k=1�k (x (m))µ̂(s)

k (x (m))

Least Squares EstimateConditional pdf

p�x (s) | x (m),�

�=

K�

k=1�k (x (m)) N

�x (s) ; µ̂(s)

k (x (m)),�̂(ss)k

�

joint pdf

µk =�µ(m)

k ;µ(s)k

�(1)

�k =��(mm)

k �(ms)k

�(sm)k �(ss)

k

�

(2)

p(x i | θ) =K∑

k=1wkN

(x i ; µk ,Σk

)

x = [x (m) , x (s)]

x = [x (m) , x (s)]

p(x i | θ) =K∑

k=1wkN

(x i ; µk ,Σk

)prior

mean

covariance

responsibility

mean

covariance

�k (x (m))

µ̂(s)k (x (m)) =µ(s)

k +�(sm)k

��(mm)

k

��1 �x (m) �µ(m)

k

�(1)

�̂(ss)k =�(ss)

k ��(sm)k

��(mm)

k

��1�(ms)

k (2)

37

Probabilistic Motion-Sound Models | GMR — number of gaussians

38

Probabilistic Motion-Sound Models | GMR — regularization

39

Probabilistic Motion-Sound Models | Hidden Markov Regression

movement parameter

sou

nd

p

ara

me

ter

39


movement parameter

sou

nd

p

ara

me

ter

time

39


likelihoods

movement parameter

movement parameters

sou

nd

p

ara

me

ter

time

39


predictedsound

parameters

likelihoods

movement parameter

movement parameters

sou

nd

p

ara

me

ter

time

39



Joint Features

Training

HMM with Joint

Observation probability distribution

40

joint observation pdf:

µk =�µ(m)

k ;µ(s)k

�(1)

�k =��(mm)

k �(ms)k

�(sm)k �(ss)

k

�

(2)

mean

covariance

p (x t | zt = k) =N�x t ;µk ,�k

�


Joint Features

Training

HMM with Joint


40

HMM with Conditional Observation probability distribution

mean

covariance

p�

x (s)t | x (m)

t , zt = k�=N

�x t ; µ̂k (x (m)),�̂

(ss)k

�conditional observation pdf:

µ̂(s)k (x (m)

t ) =µ(s)k +�(sm)

k

��(mm)

k

��1 �x (m)

t �µ(m)k

�(1)

�̂(ss)k =�(ss)

k ��(sm)k

��(mm)

k

��1�(ms)

k (2)


Joint Features

Training

HMM with Joint


40


mean

covariance

µ̂(s)k (x (m)

t ) =µ(s)k +�(sm)

k

��(mm)

k

��1 �x (m)

t �µ(m)k

�(1)

�̂(ss)k =�(ss)

k ��(sm)k

��(mm)

k

��1�(ms)

k (2)

online estimation (forward)

p(x (s)t |x (m)

1:t ,�) =


Joint Features

Training

HMM with Joint


40


mean

covariance

µ̂(s)k (x (m)

t ) =µ(s)k +�(sm)

k

��(mm)

k

��1 �x (m)

t �µ(m)k

�(1)

�̂(ss)k =�(ss)

k ��(sm)k

��(mm)

k

��1�(ms)

k (2)


p(x (s)t |x (m)

1:t ,�) = N�

x (s) ; µ̂(s)i (x (m)

t ),�̂(ss)i

�


Joint Features

Training

HMM with Joint


40


mean

covariance

µ̂(s)k (x (m)

t ) =µ(s)k +�(sm)

k

��(mm)

k

��1 �x (m)

t �µ(m)k

�(1)

�̂(ss)k =�(ss)

k ��(sm)k

��(mm)

k

��1�(ms)

k (2)


p(x (s)t |x (m)

1:t ,�) = N�

x (s) ; µ̂(s)i (x (m)

t ),�̂(ss)i

��(m)

t (i )


Joint Features

Training

HMM with Joint


40


mean

covariance

µ̂(s)k (x (m)

t ) =µ(s)k +�(sm)

k

��(mm)

k

��1 �x (m)

t �µ(m)k

�(1)

�̂(ss)k =�(ss)

k ��(sm)k

��(mm)

k

��1�(ms)

k (2)


p(x (s)t |x (m)

1:t ,�) = N�

x (s) ; µ̂(s)i (x (m)

t ),�̂(ss)i

��(m)

t (i )N�

i=1


Joint Features

Training

HMM with Joint


40


mean

covariance

µ̂(s)k (x (m)

t ) =µ(s)k +�(sm)

k

��(mm)

k

��1 �x (m)

t �µ(m)k

�(1)

�̂(ss)k =�(ss)

k ��(sm)k

��(mm)

k

��1�(ms)

k (2)

x̂ (s)t =

N�

i=1�(m)

t (i )µ̂(s)k (x (m)

t )

Least Squares Estimate

41


Interactive Machine Learning implementation

Number of Hidden States

> complexity / temporal grain

Regularization

> smoothness / generalization

Hierarchical Extension

> high-level structure

Evaluation on Tai Chi Movement Sequences

4242

outline

Related Work







GMM HHMM

GMR HHMR

Models

Applications

GMR

HHMR

Playing Sound Textures | siggraph’14 Studio installation

Françoise, J., Schnell, N., & Bevilacqua, F. (2014). MaD: Mapping by Demonstration for Continuous Sonification. In ACM SIGGRAPH 2014 Emerging Technologies.

Corpus-based Descriptor-driven Sound Synthesis

CataRT (Schwarz)

Sound Corpora from DIRTI

User Studio, Diemo Schwarz, Roldan Cahen

Advantages

Complex gestures

> implicit sonification strategies

Adaptive

> personalize gestures

> personalize sonification

44

Playing Sound Textures | texture sonification for gesture learning

Protocol

> reproduce a gesture as precisely as possible

> training from audio-visual demonstration

> reproduction: 3 recording blocks (10 trials)

silent vs sonified

> 11 participants

> 4 gestures (2 sonified / 2 silent)

Analysis

> correlation distance to reference

45



*

46

4747

outline

Related Work







GMM HHMM

GMR HHMR

Models

Applications

GMR

HHMR

48

Sonification through Vocalization | imitation game


48



MicrophoneWireless Motion Sensor

50


Imitation Game: adaptation

51



52

Evaluation

Sequence game

> 59 participants recorded at Siggraph’14

Analysis

> investigate log-likelihoods during gesture reproduction

Results

> sensitive to user expertise

> fast adaptation with sound feedback

==> learning

53

Sonification of movement

qualities

HMR System trained with Expert

performances

Exploratory workshop with dancers

> integration in teaching session

Promising for pedagogy

> Further evaluation planned

Vocalization | Application in dance

Françoise, J., Fdili Alaoui, S., Schiphorst, T., & Bevilacqua, F. (2014). Vocalizing Dance Movement for Interactive Sonification of Laban Effort Factors. Designing Interactive Systems (DIS’14), Vancouver, Canada.

54

Vocalization| movement sequence analysis (tai chi)

MO Sensors(Accel. + Gyro.)

(a)

(b) (c)

Tai Chi

> long sequences

> expertise (consistency over trials)

Motion Capture

> 3 inertial sensors (6DOF: acc + gyro)

Task

> 10 performances of 45s sequence

> compare expert & student

Françoise, J., Roby-Brami, A., Riboud, N. & Bevilacqua, F. (2013). [SUBMITTED]

Movement Sequence Analysis | vocalization in performance

55

conclusions

57

Conclusions

Mapping by Demonstration

> mapping through listening + interactive machine learning

> powerful tool for designing sonic interactions

> leverages expertise

57

Conclusions





GMM HHMM

GMR HHMM

recognition paradigm

generative paradigm

instantaneous temporal

General framework for designing

motion & sound relationships

with probabilistic models

> learning from few examples

> continuous & causal inference

> generative approach

57

Conclusions





GMM HHMM

GMR HHMM

recognition paradigm

generative paradigm

instantaneous temporal

XMM library

open-source C++ library

Max implementation

General framework for designing

motion & sound relationships

with probabilistic models

> learning from few examples

> continuous & causal inference

> generative approach

Modeling and sonifying complex movements

> music

> dance

> rehabilitation

MOCO: International Workshop on Movement & Computing

Vocalization

Domain-specific Expertise

> Sonic Interaction Design

> Dance & Movement Practice

58

Applications & Perspectives

(Greg Beller’s Synekine project)

(Moving Stories Project, Thecla Schiphorst)

(LEGOS project & Labex Smart)

(SkAT-VG EU project)

thank you

… and many thanks to collaborators (incomplete pseudo-alphabetical list):

Frédéric Bevilacqua, Omid Alemi, Pablo Arias, Thierry Artières, Caroline Barbot, Greg

Beller, Riccardo Borghesi, Éric Boyer, Amélie Briaucourt, Baptiste Caramiaux, Fabien

Cesari, Olivier Chapuis, Sarah Fdili Alaoui, Masha Fedorova, Emmanuel Fléty, Sylvain

Hanneton, Olivier Houix, Stacy Hsueh, Ianis Lallemand, Jean-Philippe Lambert,

Benjamin Matuszewski, Nicolas Rasamimanana, Natacha Riboud, Sébastien

Robaszkiewicz, Agnès Roby-Brami, Victor Saiz, Kevin Sanlaville, Thecla Schiphorst,

Norbert Schnell, Diemo Schwarz, Alejandro Van Zandt-Escobar, …

Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart...

Documents

Transcript of Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart...