Assessment of Human Behavior in Virtual Reality by Eye Tracking

Abstract

Virtual reality (VR) is not a new technology but has been in development for decades, driven by advances in computer technology such as computer graphics, simulation, visualization, hardware and software, and human-computer interaction. Currently, VR technology is increasingly being used in applications to enable immersive, yet controlled research settings. Education and entertainment are two important application areas, where VR has been considered a key enabler of immersive experiences and their further advancement. At the same time, the study of human behavior in such innovative environments is expected to contribute to a better design of VR applications. Therefore, modern VR devices are consistently equipped with eye-tracking technology, enabling thus further studies of human behavior through the collection of process data. In particular, eye-tracking technology in combination with machine learning techniques and explainable models can provide new insights for a deeper understanding of human behavior during immersion in virtual environments.

In this work, a systematic computational framework based on eye-tracking and behavioral user data and state-of-the-art machine learning approaches is proposed to understand human behavior and individual differences in VR contexts. This computational framework is then employed in three user studies across two different domains, namely education, and entertainment. In the educational domain, the exploration of human behavior during educational activities is a timely and challenging question that can only be addressed in an interdisciplinary setting, to which educational VR platforms such as immersive VR classrooms can contribute. In this way, two different immersive VR classrooms were created where students can learn computational thinking skills and teachers can train in classroom management. Students' and teachers' visual perception and cognitive processing behaviors are investigated using eye-tracking data and machine learning techniques in combination with explainable models. Results show that eye movements reveal different human behaviors as well as individual differences during immersion in VR, providing important insights for immersive and effective VR classroom design. In terms of VR entertainment, eye movements open a new avenue to evaluate VR locomotion techniques from the perspective of user cognitive load and user experience using machine learning methods. Research in two domains demonstrates the effectiveness of eye movements as a proxy for evaluating human behavior in educational and entertainment VR contexts. In summary, this work paves the way for assessing human behavior in VR scenarios and provides profound insights into the way of designing, evaluating, and improving interactive VR systems. In particular, more effective and customizable virtual environments can be created to provide users with tailored experiences.