Abstract:
The quality of classroom teaching plays a central role in students’ academic achievement and motivation (Burroughs et al., 2019). A model frequently used in German-speaking countries defines three basic dimensions of teaching quality: cognitive activation, student support, and classroom management (Klieme et al., 2009). To promote teaching quality based on these dimensions, reliable and valid methods for assessing teaching quality are required. One central method is classroom observation by external observers (O’Leary, 2020). Classroom observation, often conducted using classroom videos, allows for systematic and detailed analysis of teaching in both research and practice contexts. Although the perspective of external observers offers numerous advantages, there are clear limitations in capturing teaching quality through observer ratings, such as the difficulty of observing certain teaching-quality aspects from an outside perspective or the influence of idiosyncratic rater characteristics on the ratings (Fauth et al., 2020; Göllner et al., 2016). Previous research has shown that these challenges are often reflected in the limited psychometric quality of observer ratings of teaching quality (e.g., Kelly et al., 2020). Given the potential of (video-based) classroom observations for teacher education and educational research, it is important to investigate under which conditions external observers can provide the most accurate assessments of teaching quality possible.
The aim of the present dissertation was to examine conditions for accurate observation-based assessments of teaching quality. To this end, a theoretical model of observers’ assessment accuracy in classroom observations was developed, based on Funder’s (1995) model of judgment accuracy in personality assessment. The proposed model outlines a four-step process necessary for accurate assessments of teaching quality: relevance, availability, detection, and utilization of behavioral cues in the teaching. Additionally, the model identifies two key factors that can influence this assessment process: the observers’ disposition and the observation environment in which the classroom observation takes place. This dissertation includes three empirical studies, each representing a different approach to examining observer ratings of teaching quality and systematically investigating aspects of the proposed theoretical model of observers’ assessment accuracy in classroom observations. The first study (Can teachers be trained to provide valid teaching-quality ratings?) focused on the role of the observers’ disposition in providing observer ratings of teaching quality, specifically the level of training in the use of standardized classroom observation systems. In this study, a rater training for in-service teachers in the use of a standardized classroom observation system was followed across five time points. The study examined how the reliability of teaching-quality ratings provided by teachers, based on classroom videos, developed throughout the training. Following the training, a validation study was conducted to assess the validity of the ratings provided by the trained teachers. The results showed that the teachers were able to rate the observed teaching-quality aspects with increasing agreement over the course of the training, with varying developments of interrater agreement depending on the aspect being assessed. Furthermore, three validity arguments for using the trained teachers’ ratings were derived: increased agreement with expert ratings, a fitting factor structure, and high convergent correlations with ratings from a comparable classroom observation system. With these findings, the first study emphasizes the importance of rater training in using standardized classroom observation systems for accurate assessments of teaching quality from the perspective of external observers. The second study (Immersive insights: Unveiling the impact of 360-degree videos on pre-service teachers’ classroom observation experiences and teaching-quality ratings) investigated the impact of the observation environment, comparing two different video environments: a traditional video environment on a computer and an immersive 360-degree video environment using virtual reality (VR) headsets. This study examined the classroom observation experience and resulting observer ratings of teaching quality of pre-service teachers in both video environments. The results indicated that in the immersive 360-degree environment, the pre-service teachers felt more cognitively, affectively, and physiologically involved in the classroom teaching and reported higher motivation, without reporting a higher mental workload. Teaching-quality ratings in both video environments were largely similar, except for one aspect of cognitive activation (focus on key concepts), which was rated more accurately in the immersive 360-degree video environment. This study highlights the potential of immersive 360-degree classroom videos for teacher education and the assessment of teaching quality. The third study (Connecting gaze behavior and ratings of teaching quality) was based on the assumption of the proposed theoretical model that the observer must successfully detect the quality-relevant classroom events for an accurate assessment of teaching quality. Using eye-tracking data from classroom observations, this study examined the relationship between the gaze behavior of pre-service teachers and their observer ratings of teaching quality. It also compared these relationships between classroom observations in traditional and immersive 360-degree video environments. The results indicated that visual attention to specific areas in the classroom, as well as pupil diameter during quality-relevant classroom events, were related to the accuracy of observer ratings of teaching quality, with these relationships being particularly pronounced in immersive 360-degree video environments. Thus, the third study highlights the potential of using eye-tracking data as process data from classroom observations to deepen our understanding of the perceptual processes underlying observer ratings of teaching quality in classroom observations. The present dissertation provides novel insights into the conditions for successfully assessing teaching quality through classroom observations, thereby enhancing our understanding of observer ratings of teaching quality. By proposing a theoretical model of observers’ assessment accuracy in classroom observations and presenting innovative research approaches—such as the systematic investigation of rater training, the use of immersive 360-degree classroom videos, and the integration of eye-tracking data as process data from classroom observations—this dissertation makes significant theoretical, methodological, and empirical contributions for applying classroom observations in teacher education, professional development, and classroom research.
Classroom Observation