8/3/2019 Humaine Kaiser
1/33
First HUMAINE Workshop, Geneva June 17-19,
2004
Possible solutions from models of emotional
expression ???
Susanne KaiserGeneva Emotion Research Group, University of Geneva
*This research was supported by grants from the Swiss National Science Foundation (FNRS
11-39551.93/049629.96 "Dynamic man-computer interactions as a paradigm in emotion
research") granted to S. Kaiser and T. Wehrle.
http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
2/33
Outline of the presentation
1. The complexity of facial expression
2. Ekman & Friesens Facial Action Coding
System
3. Discrete emotion theory and facial expression
4. Appraisal theory and facial expression
5. Examples of empirical studies in our research
group
6. Conclusions for animating facial expressions
http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
3/33
The multi-functionality of facial behavior
A smile or a frown, for instance, can have different meanings.
It can be :
a speech-regulation signal
a speech-related signal
a means for signaling relationship
an indicator for cognitive processes
an indicator for an emotion.
A given facial expression can have several meanings at the
same time.
http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
4/33
Expression controlDisplay rules:de-intensifying
affectless or neutral (poker face).
masking
over-intensifying.
Facial indicators of emotional processes are often verysubtle
Approaches to measure facial expressions objectively and
on a micro-analytic level are indispensable.
http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
5/33
Facial Action Coding System (Ekman, P. & Friesen,W., 1978; Ekman, P., Friesen, W. V., & Hager, J. C., 2002)
http://face-and-emotion.com/dataface/facs/description.jsp
FACS is anatomically based and allows the reliable coding ofany facial action in terms of the smallest visible unit of muscular
activity. These smallest visible units are called Action Units
(AU), each referred to by a numerical code.
With FACS, data collection is independent of data interpretation.
There is no relation between the code and the meaning of thefacial action.
As a consequence, coding is independent of prior assumptions
about prototypical emotion expressions.
http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
6/33
Source: Ekman, P., & Friesen, W. V. (1978). The Facial Action Coding System: A technique
for the measurement of facial movement . Palo Alto: Consulting Psychologist Press.
http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
7/33
http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
8/33
Discrete emotion theory and facial expression
Ekman & Friesen have presented emotion
prediction tables that list the Action Unit
combinations that build up the postulated
prototypical facial expressions of basic emotions
(EMFACS).
http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
9/33
Examples for happiness, surprise, anger, and disgust
Source: Picard, R. W. (1997).Affective computing (p. 177). Cambridge: The MIT Press.
http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
10/33
Major problems with the prototypical expression patterns
are:
These prototypes or major variants do only very rarely
occur during spontaneous interactions.
The meaning of a facial expression depends to a high
degree on the situational context (verbal, other
nonverbal channels, behavior of the interaction
partner, timing).
There exists a high inter-individual variability. Manyof the facial expressions shown by a person are
person-specific.
http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
11/33
Componential Appraisal Theory
Appraisal theorists following a componential approach share
the assumption that:
a) emotions are elicited by a cognitive evaluation of
antecedent situations and events and that
b) the patterning of the reactions in the different response
domains (physiology, expression, action tendencies,
feeling) is determined by the outcome of this evaluation
process.
http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
12/33
Componential Appraisal Theory
They argue that there are a large number of highly differentiated
emotional states, of which the current emotion labels capture
only clusters or central tendencies of regularly recurring ones,
referred to as modal emotions. In line with this reasoning, facial expressions are analyzed as
indicators of appraisal processes in addition to or as an
alternative to verbal report measures. Facial expressions are not
seen as the readout of motor programs but as indicators of
mental states and evaluation processes.
http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
13/33
Appraisal
dimensions
Hot Anger AUs Elation AUs
Suddenness high AU1b + AU2b + AU25 high/medium AU1b + AU2b + AU25
Familiarity low AU4a + AU7
Predictability low AU4b low
Intrinsic
Pleasantness
high AU6 + 12b
Concern
relevance
high high
Outcome
probability
very high very high
Expectation low low
Conduciveness obstruct AU4b + AU7b + AU17b +
AU23b
very high AU6 + AU12d
Urgency high intensification, high tension low
Cause agent otherCause motive intent chance/intent
Control high
Power high AU17c + AU24
Adjustment high medium
External norms low AU10c high
Internal norms low high
http://www.clicktoconvert.com/http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
14/33
Wehrle, T., Kaiser, S., Schmidt, S. & Scherer, K. R (2000). Studying the dynamics of emotional expression via
synthesized facial muscle movements.Journal of Personality and Social Psychology , 78 (1), 105-119.
Hot Anger
http://www.clicktoconvert.com/http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
15/33
The Geneva Appraisal Manipulation
Environment (GAME)
The Geneva Appraisal Manipulation Environment (GAME; Wehrle 1996)
is a tool for generating experimental computer games that translate
psychological theories into specific micro-world scenarios. The creation of micro world scenarios is based on theoretical predictions
concerning emotion antecedent appraisal and emotion specific action
tendencies as postulated by different componential appraisal theorists
(Scherer, 1988; Frijda, 1986).
Furthermore, game scenarios have been related to motivational
dispositions and corresponding coping strategies as postulated by Bischof
(1985).
http://www.clicktoconvert.com/http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
16/33
Example of a maze like game level created with GAME
http://www.clicktoconvert.com/http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
17/33
Can situational appraisal profiles differentiate
between different types of identically labeled
emotions?
We can describe at least 3 types of anger:
being angry as an reaction to an unfair event butwithout blaming anybody
being angry and blaming somebody else for
having caused it on purpose
being angry and blaming the other as well asoneself
http://www.clicktoconvert.com/http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
18/33
An example of anger type I
We see the reaction of a subject to amigos
intervention reducing speed at the fast level.
She reports anger.
http://www.clicktoconvert.com/http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
19/33
An example of anger type I
As can be seen in the following appraisal profile, the participant
evaluates the situation as very sudden, very new, very unpleasant,
and as not at all expected.
Additionally, she thinks that although the situation was difficult to
control - she had enough power to handle it, that she can easily
adjust to its consequences, and that her behaviour was adequate
(referred to as Selfin the figure).
She does not blame anybody or circumstances, however she
evaluates the situation as being very unfair (referred to as Norm
(-) in the figure).
http://www.clicktoconvert.com/http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
20/33
being angry as an reaction to an unfair event but
without blaming anybody
Cognitive Appraisal of Subject 2 in Situation: Speed Reduction
Reported Emotion: Anger
0
1
2
3
4
5
6
Sudd
ennes
s
Nove
lty
Plea
santnes
s
Releva
nce
Outcom
e
Expect
ation
Obstru
ct
Urgen
cy
AgentSelf
IntentSelf
AgentO
ther
IntentO
ther
Cha
nce
Cont
rol
Po
wer
Adjustm
ent
Norm
(-)
Self
Appraisal Components
Value
http://www.clicktoconvert.com/http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
21/33
An example of anger type I
As can bee seen in the next figure, she reacts by raising her
eyebrows (AU1 + AU2), only. Within an emotion-based approach,
this action unit combination could not be linked to anger.
The duration of AU1 and AU2, however, is rather long, i.e. 2.04
seconds. The mean duration of AU1 and AU2 in these studies is
1.04 seconds. Whereas the results show that short innervation of
AU1 and AU2 are linked to the appraisal dimension of
unexpectedness, in the case of the example shown in the figure, the
holding of the expression can be interpreted as an indicator ofappraising the situation as unfair.
http://www.clicktoconvert.com/http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
22/33
http://www.clicktoconvert.com/http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
23/33
An example of anger type II
We see the reaction of another subject to
amigos intervention reducing speed at the
fast level. She reports anger.
http://www.clicktoconvert.com/http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
24/33
being angry and blaming somebody else for having
caused it on purpose
Cognitive Appraisal of Subject 6 in Situation 9: Speed Reduction
Reported Emotion: Anger
0
1
2
3
4
5
6
Sud
Nov
Plea
s
Relev
Outc
ome
Expe
ct
Obstruct
Urge
ncy
Agtslf
Intslf
Agtoth
Intoth
Chan
ce
Control
Powe
rAd
j
Norm
oth
Self
Appraisal Components
value
This watermark does not appear in the registered version - http://www.clicktoconvert.com
http://www.clicktoconvert.com/http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
25/33
being angry and blaming somebody else for having
caused it on purpose
pp g p
http://www.clicktoconvert.com/http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
26/33
An example of anger type III
In the following example, the subject is reacting to
JANUS, an animated figure in the game that up to
this moment had always helped the player byendowing her with the force to defeat the enemies.
Unexpectedly, and for the first time, JANUS
misbehaves and takes power away, thus making
the player's agent a sitting duck for the enemies.She reports anger.
http://www.clicktoconvert.com/http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
27/33
being angry and blaming the other as well as
oneself
Cognitive Appraisal of Subject 21 in Situation 10: Janus Misbehaving
Reported Emotion: Anger
0
1
2
3
4
5
6
Sudd
enne
ss
Nove
lty
Plea
sant
ness
Relev
ance
Outco
me
Expe
ctatio
n
Obstr
uct
Urge
ncy
Agen
tSelf
Inte
ntSelf
Agen
tOth
er
Inte
ntOth
er
Chan
ce
Cont
rol
Powe
r
Adjus
tmen
t
Norm
(-)
Self
Appraisal Components
Value Subjects 21
http://www.clicktoconvert.com/http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
28/33
An example of anger type III The rapid sequence of eyebrow raising, followed by an immediate
frown, can be explained as the output of the sequential processing of
the event as being novel/unexpected and as creating an obstruction.
The change from raised eyebrows to the furrowed brow occurs
within two frames (0.08 seconds) and the whole brow movement
lasts less than a second.
After one second the subject shows AU14 (dimpler), an Action Unit
that tightens the mouth corners and produces a dimplelike wrinkle
beyond the lip corners. This could be interpreted as an indicator of
frustration and as a reaction to the evaluation of JANUS behavior as
being unfair.
This watermark does not appear in the registered version - http://www.clicktoconvert.com
http://www.clicktoconvert.com/http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
29/33
being angry and blaming the other as well as oneself
http://www.clicktoconvert.com/http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
30/33
Conclusions for animating facial expressions
I tried to show how an appraisal-based approach might
help us to better understand how emotions are expressed
and perceived.
With respect to human-computer interactions, these twoprocesses refer to the synthesis of emotional agents and
the analysis of a users facial expression.
http://www.clicktoconvert.com/http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
31/33
Conclusions for animating facial expressions
From the beginning of the early eighties appraisal theories
have given important inputs to emotion synthesis and
affective user modeling (e.g., OCC).
Appraisal theory is commonly used to implement thecognitive component of an emotional interface.
The outcome of the appraisal process is then mapped into
an emotion category, which determines the synthesis of the
respective facial expression pattern.
http://www.clicktoconvert.com/http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
32/33
Conclusions for animating facial expressions
This might be an unnecessary and complicated procedure
that also reduces the available information.
Furthermore, linking facial expression and other
components of emotion directly to appraisal dimensionscan take advantage of the computational representations
that are commonly already defined in the respective
applications.
http://www.clicktoconvert.com/8/3/2019 Humaine Kaiser
33/33
References
Kaiser, S. (2002). Facial expressions as indicators of "functional" and "dysfunctional"
emotional processes. In M. Katsikitis (Ed.), The Human Face: Measurement and Meaning (pp.235-254). Dordrecht: Kluwer.
Kaiser, S. & Wehrle, T. (2001a). The Role of Facial Expression in Intra-individual and Inter-individual Emotion Regulation. In D. Caamero (Ed.)Emotional and Intelligent II: TheTangled Knot of Cognition . Papers from the 2001 AAAI Fall Symposium. Technical Report FS-01-02 (pp. 61-66). Menlo Park, CA: AAAI Press.
Kaiser, S. & Wehrle, T. (2001b). Facial expressions as indicator of appraisal processes. In K. R.
Scherer, A. Schorr, & T. Johnstone (Eds.),Appraisal theories of emotions: Theories, methods,research (pp. 285-300). New York: Oxford University Press.
Wehrle, T. (1995/1999). The Facial Action Composing Environment (FACE) [Unpublishedcomputer software]. University of Geneva, Switzerland.
Wehrle, T. (1996). The Geneva Appraisal Manipulation Environment (GAME) [Unpublishedcomputer software]. University of Geneva, Switzerland.
Wehrle, T., Kaiser, S., Schmidt, S. & Scherer, K. R (2000). Studying the dynamics of emotionalexpression via synthesized facial muscle movements.Journal of Personality and SocialPsychology , 78 (1), 105-119.
For online publications and more details see:
http://www.unige.ch/fapse/emotion/members/kaiser/kaiser.html