Also published on UX Collective : https://uxdesign.cc/emotions-and-micro-expressions-in-ux-research-user-testing-a58b66612de

In a professional experience I was unexpectedly sent to Spain, where I don’t speak the language, for a user testing session of a system I did not design with the promise of a translator. Once there, no translator. Nice. But the possibility of capturing the video of the faces of the users and their interactions with the tested system.

So here I was, with no translator and no help and still got to do my job. How am I going to get insights out of this user test session ?

When viewing it, I realise that Spanish people tend to be very expressive and communicate a lot through body language and facial expressions. Unable to understand what they say with words, I used the FACS method initiated by Paul Ekman (Facial Action Coding System) to analyse user’s non-verbal communication and, more specifically, micro-expressions. Thanks to the simultaneous capture of the user’s faces and their interactions with the tested system, I was able to relate these micro-expressions to the interactions.

I realised that through the non-verbal expression of their emotions, users communicate not only their states but also their potential behaviour towards the system being tested following these states.

Micro-expressions in user testing : pros and cons

Pros

✔ The decoding of non-verbal communication cannot be done instinctively and must be objectivated (eg laughter or tears). If you think you can decode the emotional states of your users on instinct, remember that even the experts of Paul Ekman’s FACS method were unable to detect and decode all of the micro-expressions expressed by a test subject. Cal Lightman of Lie to Me does not exist in real life.

✔ A user who knows that he is being watched will tend to keep silent about certain comments he would like to make and not freely express his emotions. A field survey is always a social situation in itself.

✔ The peculiarity of micro-expressions is that they are involuntary and uncontrollable. They betray the true emotions of the users.

✔ Many facial expressions reflect universal emotions, it is a message that is not transformed by culture or language.

✔ Micro-expressions are temporalized. Very brief (by definition), it is often easy to relate them to a specific interaction if the study support allows it.

✔ By communicating their emotional states, users also communicate their probable behaviour in response to these states (abandonment, rejection, etc.).

✔ A micro-expression can be linked to an emotion but also (admittedly less objectively) be interpreted according to an intensity scale.

✔ Observation alone allows information to be gathered. The method is therefore non-intrusive in itself.

Cons

✘ Almost undetectable to the naked eye, they require the use of a video recording of the users’ faces and their interactions with the system under test, and careful viewing.

✘ The analysis of micro-expressions is like food supplements: their analysis does not exempt from applying the complete study methodology.

✘ Like any empirical observation in UX Research, they do not represent a conclusion in themselves but must be explained.

✘ The human face often presents mixtures of facial expressions (example of surprise), which complicates the task of decoding.

✘ Micro-expression analysis systems only consider expressions related to emotions. This presupposes an analysis focused on the emotional experience more than functional, even if bridges can be made.

Universal emotions and micro-expression decoding in a nutshell

Ekman’s universal emotions

What part of determinism in the emotional definition of human beings? To what extent are these emotions formatted by society, culture? This is the question Paul Ekman (like other scholars before him) attempted to answer. Through his work, which took place all over the world, Paul Ekman has shown that certain emotions are common to all of humanity.

“Emotions change the way we see the world and how we interpret the actions of others.” — Paul Ekman

Accordingly to his work, there are 7 of these universal emotions: Anger, Contempt, Disgust, Enjoyment, Fear, Sadness and Surprise. You can learn more about these emotions on the dedicated page of the Ekman’s offical website but I strongly recommend you to buy his latest book if you are really interested. To each of these emotions corresponds a physical expression, also universal, which is recognisable thanks to the Facial Action Coding System method of the same author.

Ekman’s Facial Action Coding System

In Paul Ekman’s Facial Action Coding System (FACS) method, each facial movement (called AU: Action Unit) corresponds to a code. For exemple, Brow lowerer=4 ; Nose wrinkler=9 ; Lip corner depressor=15 ; Chin raiser=17…

These AUs can be combined with others. Thus, to each micro-expression corresponds a code expressed by the combination of several AUs. For example, Happiness is codified by 6+12 as 6=Cheek raiser and 12=Lip corner puller.

The peculiarity of this approach is that the subject on which we study the micro-expressions will sometimes express a mixture of several expressions. If for example the person shows Cheek raiser (6), Lip corner puller (12) and Brow lowerer (4), it is undoubtedly that his expression of Happiness is blended with a second emotion. The code for this micro-expression will be 6+12+4 and should be interpreted differently.

To go further, it is also possible in Paul Ekman’s Facial Action Coding System method to code the intensity of an observed Action Unit (AU). For this, we add to the number of the AU a letter going from A to E (A=Trace, E=Maximum).

There are many other subtleties in this method of coding micro-expressions that I will not detail in this article because at my level it is not necessary to go so far. If this area is unknown to you and you wish to explore it I encourage you to consult the page dedicated to this subject on the official website of Paul Ekman. You can also consult the Wikipedia page of the Facial Action Coding System (FACS) to find out how the coding of micro-expressions works.

Case study : insights and limits, how it really goes

Context & Methodology

The user test was carried out with 7 users (men and women, from Spain, 20 to 40 yo), spread over 2 use scenarios. The tested system is under client confidentiality but it was a huge online form on desktop web. The applied methodology is the one you may already know :

• Protocol & Realisation
• Data compilation
• Analysis & recommandations
• Communication to sponsors & stakeholders

Realisation

Users were dispatched in 3 rooms with 3 non-UX spanish facilitators that was given to me from my client, to whom I had to quickly present the protocol and methodology. Crossed questions asked during the test allowed us to collect important information. Due to lack of time (this user test session was unexpected for me and I had 2 days to prepare it) some users had to self-administrate the test. A consent form were presented to each user before starting recording them. At this time I had not planned to analyse users micro-expressions.

I just felt like I should tell the client that the test would fail due to lack of time to organise it and lack of help from a local researcher. That was before I spotted micro-expressions on the videos.

Analysis

Always capture video whenever possible ! As I replayed the test videos, I noticed that users had very brief physical reactions when interacting with the system under test. I paused the video and scrolled through the frames one by one. I realised that while the users had a soft general emotional expression during the entire test, they had for the time of a frame or two a very exaggerated facial expression. This exacerbation was often concomitant with a specific interaction: the discouragement of a new page, the reaction or non-reaction to the click of a button. They were micro-expressions.

Note : I used Open Broadcaster Software to simultaneously record screen and users faces. In the software you can place zones as you want and resize them to have 2/3 on the video dedicated to screen capture (to capture interactions) and 1/3 dedicated to user face. Remember you can crop zones on your OBS canvas by resizing them while pressing “option” on Mac.

Friendly reminder: don’t forget to ask users for permission before filming them and recording their actions, using a consent form 😉

A micro-expression refers to the expression of emotion through the muscles of the face. Accordingly to Paul Ekman’s work, because micro-expressions betrays the person’s real emotional state, without a social filter, it is always very brief and lasts less than half a second. After this half-second, the person regains control of their facial expressions to filter what they express; this is because social pressure often prevents us from freely expressing negative emotions such as anger or fear.

It is therefore very interesting to watch users betray their true emotional states for less than half a second. When this happens, it is often indicative of a lot.

Note that a user can also express emotion without it being a micro-expression. If this expression lasts longer than half a second, it is no longer a micro-expression. It is nonetheless valid in itself but we must keep in mind that it could be falsified by the person, consciously or not, for reasons of modesty or social pressure. The peculiarity of micro-expressions is that they are never falsified.

Once a micro-expression has been captured, it is possible (if the study medium allows it) to link it to an interaction. By analyzing micro-expression, therefore by revealing the emotion it communicates and its intensity, we can understand what the user’s reaction is to this specific interaction. If the user is expressing anger, it may be the result of frustration due to the inability to complete a task. Even if the user does not talk about their anger state, this is an issue to consider.

Depending on the emotion expressed by the user, the danger to the system is not the same. An angry user could lose control and leave their route. In extreme cases, he might even have an aggressive gesture towards his device. A user expressing disgust is very likely to want to get away from the system. His disgust will most often be generated by unpleasant sensory stimuli (visual, auditory, haptic). If this problem appears, there are user testing methods geared towards analyzing these sensory stimuli that can correct it.

A surprised user might also want to get away from the system. Even if the surprise is followed by a positive feeling, the experience of the surprise itself may later generate the anticipation of a feeling of fear, in which case the user would keep a negative experience in memory. The expression of happiness in your users is the only positive expression that can be analysed according to the work of Paul Ekman. In this case, the elements identified as being at the origin of this expression of happiness must be preserved.

Moment of truth : how will the stakeholders and sponsors react ?

For restitution, I presented on each slide a micro-expression, the FACS code associated, an explanation of the context and an UX recommandation.

At the end I could find around 20 frames describing a micro or macro-expression. The restitution was very well received by the business and sponsors who saw an innovative approach in this process. And because my UX team had made them understand that the conditions were really tough to generate insights from this user testing session.

Unfortunately, some users couldn’t complete their scenario due to technical issues, and some users who had to self-administrate the test skipped a few questions. But thanks to micro-expression analysis I still managed to generate a total of 18 recommendations sorted by criticality and to refine the Personae with to the crossed questions. Developers have implemented the recommendations within the limits of the project budget.

In conclusion

Deprived of my usual tools and faced with the language barrier, the analysis of micro-expressions was a tool that allowed me to bounce back, the key that allowed me to discover a new approach to user testing. This tool still too little popular, too little present among the tools of the “classic” methodology, has many advantages to offer. You still need to have the right equipment to capture the faces and interactions of users, and their permissions for ethical reasons. But I bet this approach will soon be a standard; for me it already is.