Fast and Furious? — Assessing Cognitive Load in Autonomous Driving

Not only does the future of mobility look autonomous: the present already is. Features like adaptive cruise control and lane centering support drivers and reduce required interactions with the vehicle, thereby introducing levels of autonomous driving. Assessing the driver’s emotional and cognitive load in autonomous driving environments, is a central safety feature in driver monitoring systems. Because even with hands off the steering wheel and feet off the pedals, the driver is in fact still driving, meaning he/she has to continuously observe traffic and to take over control in critical situations. Inattention or distraction due to additional tasks while driving lower to mid-level autonomous vehicles can lead to accidents.

Driver support features are designed to enhance the safety, convenience, and comfort of drivers. But even with these features, driving is a multitasking job. Engaging in secondary tasks, such as talking to a passenger or browsing through a playlist, can quickly distract us from our primary task: driving. After all, attention is a finite resource. Inattention while driving can manifest on different levels:

• visual inattention (taking your eyes off the road or not monitoring the vehicle)
• manual inattention (hands off the steering wheel)
• cognitive components, referring to the total amount of mental effort or resources necessary to perform a task (including attention, memory, and problem-solving abilities).

Detecting phases of high cognitive states while driving can significantly improve traffic safety. Driver monitoring systems capture the driver’s attention and alertness while the vehicle is in autonomous mode, ensuring that he/she is ready to take over control if necessary. They typically use a combination of cameras and sensors to track the driver’s head position, eye movements, and facial expressions. When assessing the driver’s cognitive states, cognitive load serves as a central component.

Driver-state estimation and cognitive load

Cognitive load refers to the measurement of the mental burden placed on an individual’s cognitive resources during task performance. It represents the capacity or limit of information that a person can effectively hold and process in their working memory while engaged in a specific task, for example driving. When the cognitive load exceeds the driver’s capacity, it can lead to challenges in processing information, reduced performance, and an increased likelihood of errors, potentially resulting in accidents. Just imagine having to focus on an intense conversation with a passenger while driving on a section of the German Autobahn with no general speed limit. Maybe not such a good idea.

Cognitive load is a complex, multidimensional construct and highly context-dependent. (Just imagine having to have that intense conversation while driving after not catching enough sleep.) The concept of cognitive load combines different aspects of various cognitive processes (e.g., perception, attention, memory, and problem-solving). Unlike physical parameters that can be directly measured (e.g., heart rate or blood pressure), cognitive load does not have a direct physical manifestation that can be easily quantified. It relies on indirect measures such as task performance, physiological responses, self-report, or behavioral indicators.

So far, studies have used these metrics individually or in subsets, but not combined during the same data acquisition protocol. This ist where ADABase (Autonomous Driving Cognitive Load Assessment Database) comes into play. ADABase was developed using a multimodal approach, by continuously measuring multimodal signals:

• Physiological measurements to infer cognitive load: electrocardiogram (ECG), electrodermal activity (EDA), electromyography (EMG), pulse rate, respiration rate, skin temperature, and eye-tracker data (e.g., fixation, blinks, or pupil diameter). Increased cognitive load may lead to physiological changes associated with arousal and stress.
• Behavioral measurements (action units extracted from facial videos to detect facial cues) can provide insights into cognitive load levels (read more about the methodology in our Emotion Analysis 101).
• Performance metrics (reaction time) are used as indicators of cognitive load. Higher cognitive load often leads to decreased performance or increased errors.
• Subjective feedback using questionnaires: Participants rate their subjective experience and perceived mental effort during a task.

ADABase fills an additional research gap by providing data of the same subject engaged in different tasks with different intensities of cognitive load in one recording session. The recordings of 30 subjects are made publicly accessible to the research community to enable the development of novel multimodal machine learning algorithms.

Get fast and easy access to ADABase

ADABase is an open hierarchical data format v5, readable with all common programming languages, including python using pandas package.

Study setup

The ADABase study setup simulates a close-to-real-world driving experience in an autonomously driving car, where only little interaction of the driver with the vehicle is required. The simulator controls the steering, gas, and brake control as well as gear shifting. Left with only few driver-car interactions, the 51 test subjects were asked to participate in a test containing different levels of cognitive load (n-back test). In addition, they were asked to complete a secondary task of interacting with a car infotainment system while observing the vehicle:

• n-back test: In this test, a series of stimuli (such as shapes or letters) are presented to the test subject one at a time. The participant’s task is to indicate whether the current stimulus matches the one presented n steps earlier in the sequence. For example, in a 2-back test, the participant would have to match the current stimulus with the one presented two steps earlier. A 3-back test is about matching the stimuli presented three steps earlier, and so on. The n-back test is used to continuously assess the test subject’s working memory and attention capacity.
• k-drive test: The subject is asked to perform an additional secondary task while observing the autonomously driving vehicle (with k indicating the number of actions the user needs to react to). The study’s application task is taken straight from real life: searching and adding songs to a playlist using a music streaming app on a tablet computer.

For both tests, positive hits (reaction was correct) and negative hits (wrong reaction or no reaction occurred) as well as reaction time for true positives were recorded as performance metrics. After every phase of the n-back and k-drive tests, the subjects were asked to self-evaluate and report the effort it took them to complete the given task. While the subjects were performing the tests, a camera system and an eye-tracking device recorded the abovementioned physiological parameters.

Behind the wheel: How to estimate cognitive overload

ADABase was used to train and assess several machine learning models, aiming to pinpoint the predictive power of each of the measurement modalities when it comes to estimating cognitive overload in autonomous driving. Using single and multimodal inputs, each machine learning model was evaluated, and feature importance examined (see Oppelt et al. 2023).

The authors’ main takeaway is that combining modalities enhances classification accuracy, particularly when incorporating data from both tasks. This underscores the significance of adopting a multimodal approach when estimating cognitive load.

Read the full article

Oppelt, M.P.; Foltyn, A.;
Deuschel, J.; Lang, N.R.; Holzer, N.;
Eskofier, B.M.; Yang, S.H.; ADABase:
A Multimodal Dataset for Cognitive
Load Estimation. Sensors 2023, 23, 340.
https://doi.org/10.3390/s23010340

Image copyright: Fraunhofer IIS