Educational Effectiveness at the End of Upper Secondary School: Further Insights Into the Effects of Statewide Policy Reforms

Abstract

For several decades, educational policy reforms have been understood as major instruments of educational governance that can impact existing educational practices, for instance, in terms of changes in teaching strategies, learning materials, and students’ achievements (Fullan, 1983). However, in contrast to their huge sociopolitical relevance, scientific evaluations of such reforms are scarce (e.g., OECD, 2015). Rigorous evaluations and deeper investigations of reforms are of special societal importance for several reasons: (a) to legitimize sometimes very controversial legislative policy decisions, which are to be implemented by the educational administration, (b) to test and minimize aspects of educational policies, which are predominantly based on normative arguments and which are now implemented as trial and error policies, and (c) to increase knowledge about when educational policy reforms and curricular programs lead to intended or unintended effects for students (e.g., Black & Wiliam, 2009; McConnell, 2010; Schaffer, Nesselrodt, & Stringfield, 1997). Moreover, formative and summative evaluations of educational policy reforms against objective standards (e.g., Campbell, 1969; Konstantopoulos & Hedges, 2008) are important for decreasing the likelihood of unintended side effects right from the start of their implementation. A systematic, empirically grounded evaluation of educational policy reforms is also of special importance in the face of the high complexity of the multilayered education system, where reforms are usually focused on impacting surface structures (e.g., Elmore, 1995) but shall actually trigger students’ individual educational processes, for instance, competence growth. For these reasons, the effects of policy reforms are generally very complicated to anticipate during the construction and implementation of the reforms (Fusarelli, 2002; Young & Lewis, 2015). In the present dissertation, I investigate a variety of central psychological factors at the student level before and after the implementation of two central educational policy reforms at the end of upper secondary school. In this project, I do not merely analyze the reforms in a loose, isolated framework, but I integrate and critically reflect on them more closely in a disciplinary context. In fact, in this dissertation, I make an attempt to integrate the four studies into a larger, more general context of educational reform, which can be addressed only in an interdisciplinary way. Therefore, this dissertation also focuses on developments in educational policy and educational science in general, which define the central foundations for introducing policy reforms in the education system. Moreover, I also focus on developments related to educational governance and discussions about the increase in demands for evidence-based policy (see Chapters 2 and 3) before outlining the need to include psychological factors and related theoretical models in reform evaluations (see Chapter 3). The two reforms that are analyzed here are the reform of upper secondary school and the G8 reform, both of which were implemented at the beginning of the new millennium in most German states. The two reforms are still critically discussed in the society and by educational policy. In doing this, I use prominent theoretical models, for instance, a model of achievement motivation (e.g., Eccles & Wigfield, 2002) in order to generate appropriate hypotheses and integrate the results of the effects of the reforms into recent research. The reform of upper secondary school mainly implemented mandatory course choice on an advanced course level in upper secondary school and therefore depicts a curricular intensification (CI) reform. The G8 reform reduced overall school time in high track schools (Gymnasium) from a total of 9 to 8 years by means of a compression of school time in terms of an increase in allocated time per week in lower secondary schools. I analyzed the reform of upper secondary school using a large representative sample from Thuringia (Additional Study Thuringia of the National Educational Panel Study) and Baden-Württemberg (TOSCA study; Blossfeld, Rossbach, & Maurice, 2011; Köller, Watermann, Trautwein, & Lüdtke, 2004; Trautwein, Neumann, Nagy, Lüdtke, & Maaz, 2010). Furthermore, I conducted the analysis of the G8 reform by considering a large representative data set from Baden-Württemberg (Additional Study Baden-Württemberg of the National Educational Panel Study; Blossfeld et al., 2011). In Study 1 (Maximizing Gender Equality by Minimizing Course Choice Options? Effects of Obligatory Coursework in Math on Gender Differences in STEM; Journal of Educational Psychology), differential effects of the upper secondary school reform on advanced math, math self-concept, and vocational interests were investigated. Furthermore, potential differences before and after the upper secondary school reform regarding the field of study at university in STEM (science, technique, engineering, and mathematics) subjects were focused on. Results showed that whereas gender differences in math achievement were lower after the reform, differences were larger on all other outcome variables. In spite of these results, no differences before or after the reform were found for the choice of the field of study at university. Study 2 (Putting All Students in One Basket Does not Produce Equality: Gender-Specific Effects of Curricular Intensification in Upper Secondary School; Manuscript submitted for publication) expanded the results of Study 1 by considering data from another German state, namely, Thuringia. In Study 2, it was possible to analyze a broader variety of student outcome measures in English reading, mathematics, biology, and physics, as well as students’ subject- specific self-concepts and interests in these subjects. The results of this study indicated no statistically significant average differences on achievement measures. However, differential effects on English reading and a higher English self-concept in favor of young men were found after the reform, whereas the reform had a negative effect on young women’s math self-concept. In Study 3 (Comparing Apples and Oranges: Reforms can Change the Meaning of Students’ Grades!; Manuscript submitted for publication), analyses of reform effects were extended to school grades. Students’ grades at the end of upper secondary school are of special importance for college and university access and later job employment. However, research has shown striking differences between teacher-assigned grades and standardized student achievement. Furthermore, grades are oftentimes assigned on a norm-referenced basis and are therefore strongly oriented toward a class’ achievement composition, which changed when detracking was introduced by the CI reform. Therefore, Study 3 was focused on the research question of whether students’ standardized achievement differed between before and after the CI reform, given similar grades. Results suggested considerable differences in students’ standardized test achievement before and after the reform, given similar grades. Compared with basic courses, standardized achievement given a similar grade in core courses was higher. However, the opposite pattern was found when comparing achievement between advanced and core courses, given a similar grade. Furthermore, for math these effects were found to vary among high and low grade levels. Finally, Study 4 (The G8 reform in Baden-Württemberg: Competencies, Well-Being, and Leisure Time Before and After the Reform; Zeitschrift für Erziehungswissenschaft) is one of the first studies to investigate effects of the G8 reform at the end of upper secondary school. In contrast to the reform of upper secondary school, the G8 reform increased the time allocated in lower secondary school in order to reduce the total time spent in school by 1 year. Study 4 therefore focused on potential changes in student achievement in mathematics, English reading, biology, and physics from before to after the reform. In addition, potential effects on variables related to students’ well-being (stress and health) and leisure time use were analyzed. Differences between G8 and G9 students were found in English reading, biology, and in well-being measures in favor of the G9 students. All studies in this dissertation investigated the research questions using advanced statistical methods such as multidimensional multiple-group IRT models or structural equation models with continuous indicators and considered survey weights, missing data, and the clustered structure of the data. The reforms that the dissertation focused on were chosen specifically in order to investigate central individual aspects but also have an exemplary, more general function in the context of investigating changes in specific surface structures of the education system on specific psychological factors related to achievement. Similarly, all reforms were implemented in the highest track school, the Gymnasium, which is currently the most frequently attended school type in lower and upper secondary school. The Gymnasium is important because the results of the upper secondary school examination strongly determine whether a student is eligible to enroll in university. In the beginning of this dissertation, I will first provide a general introduction regarding the meaning and expectations of educational policy reforms. I will subsequently integrate this material into the central findings and developments of educational effectiveness research and educational governance in Chapters 2 and 3. After presenting Studies 1 to 4 in Chapter 4, I will outline the strengths and limitations and implications of the dissertation in Chapter 5.

Bildungsreformen werden seit einigen Jahrzehnten als zentraler Bestandteil der politischen Steuerung des Bildungswesens verstanden, die Einfluss auf die schulische Bildungspraxis nehmen können und beispielsweise Veränderungen der bestehenden Unterrichtsstrategien, Lernmaterialien und Schülerleistungen intendieren (Fullan, 1983). Trotz ihrer hohen gesellschaftlichen Relevanz sind Bildungsreformen nur selten Gegenstand systematischer Untersuchungen (OECD, 2015). Profunde Evaluationen und vertiefende Analysen von Reformprogrammen sind von besonderer gesellschaftlicher Bedeutung, u.a. zur Legitimierung bildungspolitischer Entscheidungen sowie zur vertieften Untersuchung und Prüfung normativer Programmanteile (z.B. McConnell, 2010). Darüber hinaus ist eine systematische, empirisch fundierte Begleitforschung von Bildungsreformen in besonderer Weise relevant, da in Anbetracht der Komplexität und mehrdimensionalen Struktur des Bildungswesens Effekte von Bildungsreformen einerseits Oberflächenstrukturen betreffen (z.B. Elmore, 1995), jedoch andererseits auch individuelle Bildungsprozesse bei den Schülerinnen und Schülern auslösen und z.B. Kompetenzzuwächse bewirken sollen. Derartige Veränderungen sind ohne vertiefende wissenschaftliche Analysen nur schwer im Entwicklungsprozess der Reform zu antizipieren und im Umsetzungsprozess zu erkennen (Conley, 1994; Fusarelli, 2002; Young & Lewis, 2015). Die vorliegende Dissertation untersucht verschiedene Schülervariablen vor und nach zwei zentralen bildungspolitischen Reformprogrammen am Ende der Sekundarstufe II. Die Reformen werden zum einen in den vier Beiträgen hinsichtlich ihrer jeweiligen Spezifität sowie ihrer inhaltlichen und methodischen Qualität dargestellt. Darüber hinaus werden zum anderen die zugrundeliegenden Beiträge auch in einen größeren und im Grundsatz nur interdisziplinär zu bearbeitenden Kontext eingeordnet. Daher fokussiert die Dissertation ebenfalls zentrale bildungspolitische und wissenschaftliche Entwicklungstendenzen, die den Rahmen von reformpolitischem Handeln im Bildungssektor definieren. In Studie 1 wurden Unterschiede vor und nach der Oberstufenreform in Baden-Württemberg in Bezug auf die Leistung, das Selbstkonzept in Mathematik sowie die beruflichen Interessen der Schülerinnen und Schüler analysiert. Studie 2 erweitert diese Einblicke um Ergebnisse zur Oberstufenreform in Thüringen. In Studie 3 wird auf Grundlage der repräsentativen Datensätze der ersten beiden Studien untersucht, ob sich nach der Oberstufenreform Unterschiede in der kursspezifischen Benotung von Lehrerinnen und Lehrern, bezogen auf die standardisierte Schülerleistung, zeigen. Abschließend werden Unterschiede von Schülerinnen und Schülern vor und nach der G8-Reform in Baden-Württemberg genauer betrachtet. Die Ergebnisse der Studien unterstreichen die besondere Bedeutung der leistungsbezogenen Komposition von Schülerinnen und Schülern und deuten darauf hin, dass vergleichbare Reformprogramme in unterschiedlichen Bundesländern auch zu unterschiedlichen Ergebnismustern führen können. Zusammenfassend verweisen die Studien auf die Notwendigkeit groß angelegter, längsschnittlicher Reformstudien, die eine systematische Untersuchung intendierter und nicht intendierter Effekte ermöglichen.