Jason, I’ve taken part in writing assessment sessions evaluating student-written essays in one of six scoring regions in the state of California using a statewide direct writing assessment strategy with a thorough training session, on-the-spot moderation for recalibrating drifting scorers and mediating greater than one point splits, and qualitative reports back to districts. It was incredible. I’ve also helped design scored middle school portfolios produced in classrooms across 19 states for a multi year project that provided proof of concept for large-scale portfolios. I worked on PACT, a state level assessment consortium that developed a way to use portfolios in teacher prep programs to assess capacity to teach writing. You are generalizing from a narrow slice of experience. I’ve worked with ETS on multiple-choice and constructed response measures. They are pretty good for quick and dirty but indefensible in contrast with NSP, PACT, CAP, Kentucky, Vermont, and all of the other projects. There is a deep body of research with a decided refusal to make use of it. Without defending it in this comment, let me say this: I believe using AI to evaluate student writing is a categorical mistake of immense proportions. And I would never, ever, ever use Texas as a model for assessment unless you want the simplest plain vanilla cut corners ideological measure available. My experience with Texas assessment is extensive.
Jason, I’ve taken part in writing assessment sessions evaluating student-written essays in one of six scoring regions in the state of California using a statewide direct writing assessment strategy with a thorough training session, on-the-spot moderation for recalibrating drifting scorers and mediating greater than one point splits, and qualitative reports back to districts. It was incredible. I’ve also helped design scored middle school portfolios produced in classrooms across 19 states for a multi year project that provided proof of concept for large-scale portfolios. I worked on PACT, a state level assessment consortium that developed a way to use portfolios in teacher prep programs to assess capacity to teach writing. You are generalizing from a narrow slice of experience. I’ve worked with ETS on multiple-choice and constructed response measures. They are pretty good for quick and dirty but indefensible in contrast with NSP, PACT, CAP, Kentucky, Vermont, and all of the other projects. There is a deep body of research with a decided refusal to make use of it. Without defending it in this comment, let me say this: I believe using AI to evaluate student writing is a categorical mistake of immense proportions. And I would never, ever, ever use Texas as a model for assessment unless you want the simplest plain vanilla cut corners ideological measure available. My experience with Texas assessment is extensive.