Perception of Emotional Body Expressions in Narrative Scenarios and across Cultures

Abstract

Emotions are ubiquitous in human lives and their importance is now well-recognised in science. Non-verbal emotional expression in human communication is realised via various means: facial expressions, emotional prosody and body motions. In real life these means complement and amplify each other, making emotion expression maximally specific, unambiguous and fitting the context. However, such multimodal expression is challenging to study due to natural innumerability of possible combinations and variations. Hence, experiments on emotion perception are often performed with well-controlled uni-modal stimuli, with the exception of studies on multi-sensory integration. In this thesis we examine perception of emotional body expressions, since especially this channel of information has been largely under-investigated despite it being important for emotion expression.

Some aspects of core affective phenomena, such as fearful reaction to threat, aggression, bonding, and drive for procreation are universal for people and several other animal species. Nevertheless, the way people express and perceive emotions via body motion greatly depends on multiple factors, such as immediate context, social and cultural background. In this thesis we present several studies that aim to gain deeper understanding of perception of emotional body expressions. We focus on body motion generated by humans during meaningful speech in naturalistic narrative scenarios and investigate its perception by using categorisation and rating tasks. Across our experiments we use a rich set of emotion categories that aims to cover many aspects of human emotional experience. Our list includes five positive categories (amusement, joy, pride, relief, surprise), five negative categories(anger, disgust, fear, sadness, shame) and the neutral category.

We report on the acquisition process and final form for a large database of motion capture data we collected for our research. The actors who took part in the motion capture experiment were narrating stories, expressing emotions in a natural way using all available media in a normal way without unnecessary exaggeration of motion patterns. The database is in open access and contains not only the collected motion sequences but also various meta-information, such as intended emotion of the actor, physical properties of the motion sequences, and, importantly, categories assigned to the motion sequences by naive observers, the latter being collected during the second study.

In the study that followed the motion capture experiment we investigated the perception of collected motion samples using stick-figure displays of the upper body alone. Our results show that despite the fact that motion samples came from naturalistic scenarios and are thus expressed emotions in a subtle, non-stereotypical manner, most of the emotions are recognised at above chance level. The agreement among observers is high as most of the motion sequences have unique emotion labels that were assigned to them by most participants. We also found a significant bias towards anger, as observers were accurate at recognising anger when it was intended as such by the actors, additionally observers tended to assign the category of anger to many other motion sequences that were intended as other emotions, hence, the false alarm rate for anger was high as well. Our analysis shows a connection between properties of motion (such as average speech, span and number of peaks) with what emotion categories the observers assign to motion sequences.

In the third study we used a subset of the previously collected motion sequences and ran an exhaustive cross-cultural study among participants from Germany, England and South Korea. In this experiment we used motion sequences representing ten emotion categories (all categories used previously except for the neutral category). We asked our participants to rate motion sequences along three affect scales (valence, arousal and dominance) and three motion scales (speed, span and brokenness). Additionally, we recorded participants' display rules for each of the emotion categories. Our results show multiple cross-cultural differences in the perception of motion sequences, these differences tracing back to the cross-cultural differences in display rules. One of the fundamental and previously unreported differences we found is that perceived increase in brokenness of motion patterns was associated with an increase in arousal levels in German and English cultures, in contrast to Korean culture, where an increase in brokenness was associated with a decrease in valence.

Research presented in this thesis expands current knowledge on the perception of emotional body expressions and fits within the existing context of related research. Our collected data and the results of the experiments can be useful to researchers from various scientific fields: neuroscientists, cognitive psychologists, computer scientists and linguists. On the application side, our findings can inform automatic emotion recognition and motion synthesis for animated virtual characters.