Measuring Cognitive Engagement in AI-Supported Writing: Introducing the Revision Responsiveness Index (RRI) as a Process-Based Assessment Model

 Abstract

This study examines students’ revision behaviors in AI-feedback-supported environments by analyzing domain- and subskill-level changes in writing, patterns of gains and losses, effort allocation across revisions, and the effectiveness of a process-based metric, the Revision Responsiveness Index. Grounded in cognitive load theory, the study evaluates how reductions in cognitive effort during writing, enabled by artificial intelligence tools, may diminish deeper engagement and learning. A quantitative, within-subjects pretest-posttest design was employed to examine revision outcomes as a writing process. The sample included 1,348 high school students who produced more than 3,800 writing samples across persuasive, informative, and narrative genres. Data were collected in a classroom-based setting using a baseline writing task completed without assistance, followed by a timed revision task supported by structured AI feedback. Revision outcomes were analyzed using the Revision Responsiveness Index (RRI), defined as the change in analytic writing scores between initial and revised drafts, supplemented by domain-level scoring and linguistic analyses. Revision outcomes demonstrated high variability with 48 percent of students improving, 34 percent declining, and 18 percent showed no meaningful change. Students who improved exhibited expected gains across multiple writing domains, while those who declined showed larger and uneven losses. These findings suggest that revision is not inherently beneficial and that ineffective integration of feedback can result in a reduction in overall writing quality.