1

Interaction et

Communication Verbale

Mohamed CHETOUANImohamed.chetouani@upmc.frInstitut des Systèmes Intelligents et deRobotique (ISIR)UMR 7222

2

Communication - interaction

! La communication est un processus de transfert d’information entre

deux entités.

3

Communication - interaction

! Interaction: le récepteur décode le message et renvoie un accusé (feedback)

! Même protocole et même moyens: auditif (parler), physique (langage corporelbody language)…

4

Communication - interaction

! Interaction face-à-face requiert plusieurs compétences (entendre, parler, observer,analyser…)

! La communication est processus cognitif

2

5

Types de communication

! Un résultat d’une analyse d’Albert Mehrabian’s montre que:

" 55%: body language (Visage, Posture)

" 38%: ton de la voix

" 7%: “mots” (contenu)

6

Types de communication

! Communication verbale et non-verbale

7

Types de communication

! Communication verbal: utilisation d’un code linguistique connu des participants enutilisant une ou plusieurs méthodes (oral, gestuel, tactile ou graphique)

" Les éléments qui codifient le sens sont appelés des “signes”

! Communication non-verbale: communication sans les “mots”

" Expressions faciales, regard, toucher, ton de la voix…

8

Pourquoi utiliser de lacommunication non-verbale?

! Les mots sont limités: décrire une forme, donner une direction…

! Les signaux non-verbaux sont « puissants »: expression d’état interne

! Le message non-verbal est souvent authentique: les comportements non-

verbaux sont « moins contrôlables » que le contenu.

! La communication non-verbal permet de transmettre des messagescomplexes: Un locuteur peut modifier et ajouter des informations au

message verbal en utilisant des signaux non-verbaux.

3

9

Comment utiliser la communicationnon-verbale?

! Peut-on apprendre ??

! http://uk.video.yahoo.com/watch/118095/955469

10

Interaction naturelle

! L’interaction naturelle signifie que l’on autorise les

utilisateurs à communiquer dans la manière qu’ils le font

avec d’autres personnes.

! Utilisation de la communication verbale et non-verbale.

11

Interaction naturelle

! Communication verbale: sémantique du message

! Communication non-verbale: geste, regard, tonalité…

12

Communication verbale

! Les bébés utilisent les sons pour communiquer

! En grandissant, ils apprennent à former des mots à partir

de ces sons.

4

13

Communication verbale! Aspects dévelopementaux:

14

Communication verbale

Saint-Georges et al, submitted

Research work with

Prof. David COHEN

Hôpital de la Pitié-

Salpetrière:

Parent-infant

interaction and

Austim

Mahdhaoui et al, International Journal of Methods in Psychiatric Research (in press)

15

Communication verbale! Plus de 3000 langues et dialectes majeurs sont parlés dans

le monde…

16

Communication verbale

! Principe de production de la parole

Cordes vocales

5

17

Communication verbale

! Plusieurs niveaux de description:

" Acoustique

" Phonétique:

" Phonologique

" Morphologique

" Syntaxique

" Sémantique

" Pragmatique

18

Utilisation sociale du langage

! Langage différent en fonction de l’objectif:

" Saluer, informer, demander, promettre…

! Changer de langage en fonction de l’interlocuteur

" Parler à un enfant ! parler à un adulte

! Suivre des règles durant une conversation:

" Prendre les tours de parole, rester sur le même sujet…

19

Communication non-verbale

! La communication non-verbale est apprise peu après la naissance etpratiquée et raffinée tout au long de la vie d’une personne

! Le comportement non-verbal est une source continue de signaux quitransmet des informations sur les sentiments, l’état mental, lapersonnalité et d’autres traits.

20

Communication non-verbale

! Caractéristiques non-verbales pertinentes:

" Visage: expressions faciales, échange de regard

" Comportement vocal: disfluences, tour de parole, silences,

pauses…

" Gestes et postures: mouvement de la tête, orientation du corps

" Apparence physique

" Exploitation de l’espace et de l’environnement: distances inter-

personnelles

Vinciarelli et al., Social signal procesing: a survey of an emerging domain,

Image and Vison Computing (in press 2009)

6

21

Le visage

! Le visage est impliqué dans plusieurs activités:

" Identité

" Production de la parole

" Communication des états affectifs, intentions => expressions faciales

" Personnalité, âge, genre…

22

Le visage! Expressions faciales

! Ekman a proposé 6 emotions

élémentaires

" Dégoût

" Joie

" Tristesse

" Colère

" Peur

" Surprise

Esposito, Cognitive Computation (2009)

23

Le visage

! Expressions faciales

" Intentionnelles ou pas

" Volontaires: masquer son émotion

! Communication par expressions faciales

24

Comportement vocal

! Le comportement vocal non-verbal inclue tous les mots, sons

prononcés, modification du message verbal:

" Qualité vocale

" Filled pauses

" Vocalisations non-linguistiques

" Silences

" Tour de parole

7

25

Comportement vocal

! Qualité vocale:

" Caractéristiques prosodiques, acoustiques…

" How something is said:

Adult-directed speech Infant-directed speech

26

Comportement vocal

! Filled pauses:

" “Non-words”: “ehm”, “ah-ah”, “hum”

" Utilisées de manière spontanée

" Eventuellement en remplacement de mots: répondre à unequestion…

" Disfluences: embarrassement

" Back-channeling: accompagner son interlocuteur

! Vocalisations non-linguistiques

" Non-verbal sounds

" Rire, tousser, pleurer…

" Eventuellement accompagnée de mots.

27

Comportement vocal

! Silences:

" Non-speech event

" Silence d’hésitation silence, silence interactiv

" Signe de respect

" Ignorer une personne, attirer l’attention

! Turn-taking (tour de parole)

" Régulation de la conversation (coordination)

" Utilisation du regard, de la qualité vocale, silences, disfluences…

" Synchronisation, chevauchement de parole(disputes)

28

Gestures and postures

! Posture and body/limb change with emotion expressed

! Head inclination, posture shifting often accompany social

affective states like shame and embarrassment.

! 90% of body gestures are associates with speech

McNeil, Hand and Mind: what gestures reveal about thought.

University of Chicago (1996)

8

29

Gestures and postures

! To communicate specific meaning (thumbs up)

! To punctuate discourse

! To greet

! Unconscious gestures:

" Self-manipulations (ear touching…)

" Manipulation of small objects (pens…)

" Self-protection gestures (moving legs…)

M.L. Knapp and J.A. Hall, Nonverbal Communication in Human Interaction,

Harcourt Brace College Publishers, New York (1972).30

Gestures and postures

! Posture:

" Used to indicate attitudes, status, affective moods

" Self-confidence, energy, fatigue, boredom

" Conveys gross or overall affect while specific emotions are

communicated by other signals (facial expressions, speech…).

! Face-to-face vs. parallel body orientation

! Congruence vs. Incongruence: symmetric postures

31

Physical appearance

! Height, body shape, hair color…

! Clothes, ornaments, make up…

! Attractiveness

! “Halo effect”: first traits we recognize in other people

! Do not necessarily correspond to the reality" See robot capacities vs. embodiment

32

Space and environment

! Proxemics:

"People often refer to their need for personal space

9

33

Behaviors and technology

34

Interpretation of signals

! Improvement of verbal interpretation

! Emotion

! Intention

! …

! Understand interactions:

" Role recognition

" Compare populations

! Autism, typical developing children

35

Successfull interaction

! Interpretation of signals

36

Successfull interaction

! Multimodality:

" Human communication is multi-modal

" Multiple sources of information provide redundant information,

which helps recover from noise in each source.

! Multi-sensorial approaches:! Physiological sensors, touching, haptic…

! Integration and fusion with cognitive architectures

10

37

Design of Social HRI System

! Robot design (1st lecture)

! Data acquisition

! Annotation

! Detection of signals

! Interpretation of signals

! Evaluation (Lecture on metrics for HRI)

38

Data acquitision

! Refers to using sensors

! Real-world scenarios

! Common sensors: microphones, cameras

" Single camera/microphone to networks/arrays

! Physiological sensors: ECG, skin conductivity, EEG…

! Sensors initially used for other objectives: laser, infra-red sensors, sonar…

39

Data acquitision

! Main challenges:" Privacy: ethical issues to be addressed when people are recorded

during spontaneous interactions

" Passiveness: unintrusive, changing the behavior of the

interaction

40

Annotation

! The meaning of social signals is not universal

! Real-world applications often require to re-precise thesignals

! Annotation approaches is one the methods used to studyinteractions to obtain a consensus for given application:

" Manual annotations

" Automatic detection of simple elements

" Combining manual and automatic annotations

11

41

Annotation! Agreement between annotators

Kappa coefficient:

! Statistical measure of inter-rate agreement for qualitativeitems: head nod, smiles, pointing….

Pr(a) is the relative observed agreement among raters

Pr(e) is the hypothetical probability of chance agreement

42

Annotation! Interpretation

Kappa Interpretation

<0 No Agreement

0-0.2 Poor Agreement

0.21-0.4 Fair Agreement

0.41-0.6 Moderate Agreement

0.61-0.8 Substantial Agreement

0.81-1.0 Almost Perfect Agreement

43

Annotation! Example

Observed level of agreement:

Total independence between the raters:

44

Automatic detection! Face detection

! Person detection

! Movement detection

! Environment: objects, text…

200 400 600 800 1000 1200 1400 1600 1800 2000

200

400

600

800

1000

1200

1400

200 400 600 800 1000 1200 1400 1600 1800 2000

200

400

600

800

1000

1200

1400

12

45

Automatic detection! In robotic:

" Several faces, voices…

" The robot is moving: the partner is not in front the robot

" Noise : environment + robot

! In addition, HRI conferences require more than just a new detection orrecognition algorithm:

! From HRI 2010 website (http://hri2010.org/authors/):" “For example, a paper that describes a new face tracking algorithm or involves an

elderly population needs to establish how the results directly contribute to HRI.”

" “For example, a paper contributing a face recognition technology would likely usestandard recognition metrics (e.g., ROC curve). In addition, the paper would likelyinclude an evaluation of interactive performance in a given scenario.”

46

Recognition and interpretation! Using state-of-the-art pattern recognition techniques:

" Machine learning: SVM, HMM…

" Data fusion: bayesian, fuzzy…

! Example: Speech processing

" From the acoustic wave to the semantic information

" But we can also extract:

! Speaker identity

! Language

! Emotion

! Intention

! …

47

Recognition and interpretation

! Speech recognition

48

Recognition and interpretation

! Intention recognition

Similar prosodic contours

13

49

Example: Feedback generationContext :

! Exploitation of multi-modal signals for the development of an active robot/agent

listener

! Storytelling experience :

" Speakers told a story of an animated cartoon they had just seen

eNTERFACE’08 (Summer School): Multimodal Communication with robots and Virtual agents

Al Moubayed et al., Generating Robot/Agent Backchannels During a Storytelling Experiment, ICRA’09.

1- See the cartoon

2- Tell the story to a robot or an agent

50

Example: Feedback generationActive listening :

" During natural interaction, speakers see if the statements have been correctly

understood (or at least heard).

" Robots/agents should also have active listening skills…

Face-to-Face corpus

Human

Computer/Robot

Interaction

51

Example: Feedback generationCorpus collection:

Audio visual recordings of a storytelling between a speaker and a listener.

! 22 storytelling sessions telling the “Tweety and Sylvester - Canary row” cartoon

story.

! Several conditions (speaker and listener): same language, different.

! Languages: Arabic, French, Turkish and Slovak

! Annotation oriented to interaction analysis:

" Smile, Head nod, shake, Eye brow, Acoustic prominence

52

Example: Feedback generation

A multimodal approach:

Overlaps of mono-modal

signals

Annotation of

Feedbacks Signals

Extraction ofRULES

Probability modelling:

Co-occurrence matrix

14

53

Example: Feedback generation

Corrected Kappa=0.61

Agreement=80%

Coder Agreement

We have used kappa coefficient to

measure the agreement betweendifferent annotators in order to take

into account agreement occuring by

chance

54

Example: Feedback generationTrack Name Cohen’s Kappa Corrected Kappa Agreement(%)

Speaker.Face 0.473 0.786 89.306

Speaker.Acoustic 0.099 0.786 84.500

Listener.Face 0.436 0.559 77.960

Listener.Head Nod 0.464 0.694 84.622

Listener.Acoustic 0.408 0.929 95.972

Differences:

-Speaker vs Listener

- Communication signals

55

Example: Feedback generationMicrophone Camera

Speech features

ExtractorHead & Face

features Extractor

(feature,time)(feature,time)

Multimodal Fusion

AIBO BML tag GRETA BML tag

GRETA Talking AgentThe AIBO Robot

Realtime BackChanneling

Agent State

ASR – Keyword Spotter

56

Example: Feedback generationSpeech Feature Extraction

Speech Stream

VADetection Pitch Tracking

Feature Extraction

Lowering_pitchRaising_pitch Stable_pitchUtterance beginning Utterance End Prominence

15

57

Example: Feedback generationHead & Face features Extractor

Aim : Detect the face and key features

for feedback analysis

Overview :

58

Example: Feedback generationMultimodal Fusion and real-time back channeling

Aim : - Receives messages from both component : Speech feature extractor and

Face feature extractor

- Parses the rules file and send the appropriate message to the agents

Speech feature

extractor

Face feature

extractor

59

Example: Feedback generationRobot AIBO feedbacks

• Lexical non-verbal behaviors : smile, acceptance, non-acceptance, laughter

…

• Behaviors represented by a sequence of actions :

•Synchronization constraint (mechanic movement):

• Ex: GRETA and AIBO smiles.

60

Example: Feedback generationEvaluation protocol

# Evaluation research recent

# virtual characters(Dehn & van Mulken 00, Ruttkay & Pelachaud 04, Buisine

& Martin 07)

# robot(Olsen & Goodrich 2003, Elara et al. 2007)

# Goal of our « evaluation »

#Compare feedback provided

by a virtual character and a

robot

16

61

Example: Feedback generationPreliminary results (12 users)

62

Example: Feedback generation

63

Summary! Social signals play an important role in Human-Human Interaction

! Usefulness in robotics require:

" Multi-modality

" Understand the impact and the meaning of these signals

" Robustness and advanced recognition techniques

! Multi-disciplinarity:

" Speech, Signal, image processing, pattern recognition

" Robotic

" Computer science, cognitive computation

" Psychology, cognition

" …