Can algorithm-based feedback help students to write better? A meta-analysis exploring surface- and deep-level outcomes [Can algorithm-based feedback help students to write better? A meta-analysis exploring surface- and deep-level outcomes]

Scherer, Sina; Graham, Steve; Busse, Vera

Abstract

Against the backdrop of rapid developments of algorithm-based feedback tools — from older tools mainly providing feedback on grammar and spelling to advanced tools based on generative artificial intelligence offering more comprehensive writing support — our meta-analysis examines to what extent algorithm-based feedback improves not only surface- (e.g., grammar and spelling) but also deep-level (e.g., structure, content, coherence) writing outcomes for different learners at secondary school and university. We reviewed experimental and quasi-experimental studies published between 2011 and the end of 2024, covering five European languages. Results from the 33 included studies indicated that algorithm-based feedback was beneficial for improving writing in general (g = 0.36). Specifically, positive effects were observed for surface-level outcomes at posttest (g = 0.31), though no lasting effects were found at maintenance (g =  0.02). In contrast, deep-level writing outcomes showed sustained improvement, with positive effects both at posttest (g = 0.31) and maintenance (g = 0.54). No significant differences between secondary and uni- versity students were observed. However, L2 learners, in general, seemed to profit most from algorithm-based feedback, showing gains in surface- (g = 0.77, bordering on significance), and deep-level outcomes (g = 0.46). While no significant differences were found between the effects of specific types of algorithm-based feedback tools, feedback from Grammarly and Pigai statis- tically enhanced students’ writing, but effects of ChatGPT feedback were non-significant. We discuss implications for future research and educational practice, also in light of the small transfer of learning to new writing tasks.