uncovering the hidden meaning of cross-curriculum comparison results on the force concept inventory

In a recent study, Caballero and colleagues conducted a large-scale evaluation using the Force Concept Inventory (FCI) to compare student learning outcomes between two introductory physics curricula: the Matter and Interactions (M&I) mechanics course and a pedagogically-reformed-traditional-content (PRTC) mechanics course. Using a conventional approach to detect between-curriculum differences, the researchers found that students in the PRTC course outperformed their M&I peers on nearly all the FCI items. However, this conventional approach does not control for the levels of student mastery of the tested concepts, and hence falls short of revealing deep information about the key differences between the two curricula. To this end, we use a stratum-specific method, common for detecting differential item functioning, to analyze a subset of the empirical data in the study of Caballero et al. By comparing students of equal mastery levels measured by the FCI (stratum-specific analysis), we find that the M&I students in fact demonstrate an advantage on the questions regarding Newton’s first and second laws. In addition, the previously reported underperformance of the M&I students on kinematics diminishes in our stratum-specific analysis. Results from this study show better alignment with students’ exposure to relevant topics in their respective courses. Both our findings and methodological approach provide useful implications for research on effective curriculum evaluation.