Multimodal Machine Learning for Automated Assessment of Attention-Related Processes during Learning

Abstract

Attention is a key factor for successful learning, with research indicating strong associations between (in)attention and learning outcomes. This dissertation advanced the field by focusing on the automated detection of attention-related processes using eye tracking, computer vision, and machine learning, offering a more objective, continuous, and scalable assessment than traditional methods such as self-reports or observations. It introduced novel computational approaches for assessing various dimensions of (in)attention in online and classroom learning settings and addressing the challenges of precise fine-granular assessment, generalizability, and in-the-wild data quality.

First, this dissertation explored the automated detection of mind-wandering, a shift in attention away from the learning task. Temporal patterns in aware and unaware mind wandering and their associations with learning outcomes were investigated. The two types of mind wandering, previously conflated in detection research, were distinguished using predictive modeling based on gaze data. Based on this, the precision and robustness of aware and unaware mind-wandering detection were enhanced by employing a novel multimodal approach that integrated eye tracking, video, and physiological data and outperformed unimodal approaches. Further, the generalizability of scalable webcam-based mind-wandering detection across diverse tasks, settings, and target groups was examined using a fine-tuned transfer learning approach to address low-quality data in real-world settings. Second, this thesis investigated attention indicators during online learning, inferring information from the group level. Eye-tracking analyses revealed significantly greater gaze synchronization among attentive learners. Third, it addressed attention-related processes in classroom learning by detecting hand-raising as an indicator of behavioral engagement using a novel view-invariant and occlusion-robust skeleton-based approach. It further explored the correlation between automatically annotated hand-raisings and self-reported learner engagement, interest, and involvement, demonstrating the potential of automated assessments for large-scale video analysis.

This thesis advanced the automated assessment of attention-related processes within educational settings by developing and refining methods for detecting mind wandering, on-task behavior, and behavioral engagement. It bridges educational theory with advanced methods from computer science, enhancing our understanding of attention-related processes that significantly impact learning outcomes and educational practices.