Acquiring Multimodal Data in the EmotionAI Box

When acquiring data for affective computing, we aim for a multimodal mix of data as it allows for a more comprehensive and accurate understanding of human emotions and behaviors. By collecting multimodal data, such as facial expressions, physiological signals, or behavioral changes, we can capture a more nuanced and holistic picture of an individual’s emotional responses. This, of course, leads to more reliable and robust results and high-quality data, which in turn is the prerequisite for the sound classification of emotions with AI-based algorithms.

Main challenges

What sounds good in theory, though, poses two major challenges to any (applied) research endeavor aiming to obtain state-of-the-art data on affective states:

Challenge no. 1: There is no one-size-fits-all solution when it comes to measurement modalities. Different study designs and use cases may have individual requirements to measurement modalities and protocols.

Challenge no. 2: While researchers – at least in their heart – wish for the optimum when it comes to study design and data, we are all constrained by feasibility and budgetary considerations.

And that’s where the EmotionAI Box comes in: We developed a closed, interference-free measurement cabin that facilitates the combination of various modalities – customized to individual needs. It provides researchers with

• a controlled environment for data acquisition and precise emotion recognition, e.g., through predefined lighting conditions and noise environment
• simple and affordable data acquisition, e.g., through wearables and extensive sensor and camera equipment
• flexible and customized selection of measurement modalities, tailored to the specific content, purpose, and needs of each study
• fast and trouble-free study setup
• easy data synchronization and accurate mapping of stimulus and response
• high level of data security during the acquisition and analysis phase, taking into account all necessary ethical guidelines and data protection regulations.

Inside the box: Multimodal data acquisition

The main benefit of an exposure cabin for emotion analysis research is the combination of a controlled environment to present stimuli and the sensor and monitoring equipment. Inside the box, the study participants are wired up to measure physiological signals ranging from respiration, pulse, and blood oxygen saturation to intestinal activity. Additionally, they can be equipped with an EEG cap. Outside the box, a study monitor carefully observes and monitors the study participants and the measurement protocols.

The configuration of the EmotionAI Box for specific study setups basically follows the modular principle: Which of the measurement modalities are chosen depends on the study’s content, purpose, and on budgetary constraints (see challenge no. 2). Including an EEG into the analysis, for example, generates great data on brain activity and attentiveness, but it also generates higher costs on account of the labor-intensive data evaluation. The automotive industry, on the other hand, is always looking to make as many of these analyses contactless (for instance, when assessing drivers’ cognitive states) – not for cost considerations, but in order to closely resemble real-world applications and measurement modalities in vehicles.

The cabin’s integrated camera equipment, therefore, serves multiple purposes. It cannot only derive emotions from facial expressions (read more on facial analysis in our Emotion 101), it can also accurately measure heart rate. This is possible because the facial color subtly changes with each heartbeat. Moreover, slight variations in pulse, known as heart rate variability, can provide insights into stress levels or feelings of being overwhelmed.

Why choose one over the other?

Once the study setup, data synchronization protocols, and the sample population are defined, the data is recorded in the EmotionAI Box. The study participants are exposed to various stimuli: visual images, videos, or audio clips, tasks they are asked to complete, or products and commercials they are asked to evaluate.

But why combine subjective assessment and emotion analysis in the first place? The benefit of 360° emotion recognition lies in the objectivity and robustness of the data. Especially when it comes to analyzing emotional reactions to products or online services, this can be off an advantage:

1. Physiological and behavioral data capture unfiltered reactions. Study participants may not want to admit in an interview or questionnaire that they were struggling with a technical device or when navigating an online shop’s website. (Yep, even in scientific contexts it can all be about the art of saving face.)
2. Affective reactions often occur unconsciously and are thus hard to access through retrospective questioning. Real-time emotion analysis can capture more of both the conscious and unconscious facet of a subject’s affective reactions – at least more than any post-hoc user feedback analysis ever could.
3. Real-time emotion analysis is more precise. In retrospect, a test subjective may not only have trouble motivating why they lost interest in a product, but also when exactly this occurred. By accurately mapping stimulus and response, multimodal data acquisition techniques can precisely determine when a customer lost interest or made a purchasing decision, for example.

Are you interested in knowing whether 360° emotion analysis can benefit you project? Just reach out to us!

Image copyright: Fraunhofer IIS / Paul Pulkert