Technology-based versus paper-pencil: Sources of mode effects in large-scale assessment

Kirsten, K.; Greefrath, G.; Emmrich, R.

Abstract

As many large-scale tests now use technology-based assessments (TBA), their comparability with traditional paper-and-pencil assessments (PPA) has become increasingly important. This study investigates the presence and sources of mode effects by comparing the TBA and PPA versions of the German comparative test ‘VERgleichsArbeiten’ (VERA) and its results for Berlin students. Based on cognitive load theory, we hypothesise that the empirical item difficulty increases when cognitive load increases. Thus, mode effects may be caused by personal characteristics but also by specific item properties, which are influenced when switching from PPA to TBA. In this study, we consider the test takers’ gender as personal characteristic as well as various formal-technical and content-related item properties and their impact on mode effects. Our results indicate that items requiring long, constructive responses or the use of GeoGebra are more difficult in TBA than in PPA. Content-related item properties, such as the need for competence in the use of representations, also increase difficulty in TBA but are no longer statistically significant when formal-technical item properties are controlled for. Our findings emphasise the importance of carefully designing TBA, especially for mathematics tests, to ensure the comparability of results with PPA.