Interpretability from the Ground Up: Stakeholder-Centric Design of Automated Scoring in Educational Assessments

AI summary

Overview

Research area: Natural Language Processing applied to educational assessment — specifically interpretable automated scoring of constructed (open-ended, text) responses.

Technical level: Intermediate. The framing and principles are accessible to non-specialists, but the evaluation relies on quadratic weighted kappa (QWK), ordinal logistic regression, and LLM fine-tuning/distillation concepts.

Scope: The paper argues that interpretable automated scoring should be designed from stakeholder needs upward, proposes four principles (FGTI), and demonstrates a reference framework, AnalyticScore, on the ASAP-SAS constructed-response dataset.

What This Paper Is About

AI-driven automated scoring can grade complex student writing quickly and at scale, but these systems are hard to inspect, and existing "interpretability" proposals have not been adopted in large-scale, real-world assessments. The authors first analyze what different assessment stakeholders actually need from an explanation, then derive four design principles and build a working scoring framework that follows them. They test whether a system constrained by those principles can still score accurately and whether its internal workings resemble how humans process the same responses.

Key Contributions

1. A stakeholder-needs analysis for automated scoring. The paper examines three groups in large-scale assessment — test takers (distinguishing summative from formative contexts), assessment developers, and test users — and identifies what each needs from scoring explanations and why.
2. The FGTI principles. Four foundational requirements for interpretable automated scoring: Faithful (explanations accurately reflect the model's actual computation), Grounded (initial features are human-understandable, explicitly identifiable elements of student work and the item), Traceable (the model decomposes into subroutines, each a specific evidentiary reasoning step on clearly specified inputs), and Interchangeable (a human can act interchangeably on any subroutine).
3. AnalyticScore, a reference framework. A modular three-phase system — component extraction, featurization, and logically traceable scoring — presented as a concrete baseline for future interpretable scoring research, with an implementation released on GitHub.
4. An empirical evaluation plus a human alignment study. Scoring accuracy is measured in QWK against baselines on ASAP-SAS, and featurization behavior is compared against seven human annotators across three assessment areas.

Main Findings

• Accuracy is close to uninterpretable state of the art. Across 10 items from ASAP-SAS, AnalyticScore is on average within 0.06 QWK of the uninterpretable SOTA — within 0.04 QWK for Science, 0.08 for Reading (Informational Text), and 0.09 for Reading (Literature) items. The AnalyticScore variant with GPT-4.1-mini as featurizer averaged 0.72 ± 0.03 QWK overall (Science 0.78 ± 0.03, Reading Informational Text 0.68 ± 0.06, Reading Literature 0.60 ± 0.01).
• It beats many uninterpretable baselines. AnalyticScore outperformed few-shot prompting with GPT-4.1 (0.63 ± 0.04 average QWK) and the automated scorer baselines AutoSAS (0.56 ± 0.04), ASRRN (0.59 ± 0.02), and NAM (0.56 ± 0.05). Each AnalyticScore variant except the untuned Llama featurizer outperformed these, with p < 0.05 on a Wilcoxon signed-rank test across all items; no area-specific difference was statistically significant due to small n.
• The strongest listed baseline still leads. Fine-tuned Llama-3.1-8b-Instruct with the rubric added as input scored 0.78 ± 0.02 average QWK — the highest average in the comparison table, and the gap the paper describes as 0.06 QWK to uninterpretable SOTA.
• Human scorers remain well ahead. Human scoring accuracy averages 0.88 ± 0.02 QWK overall (Science 0.93 ± 0.01, Reading Informational Text 0.79 ± 0.03, Reading Literature 0.91 ± 0.05).
• Distilling the LLM featurizer into an open-source model works. Fine-tuning Llama-3.1-8b-instruct with QLoRA on 10k sampled (response, component) pairs raised average QWK by 0.13 (Science +0.19, Reading Literature +0.12, Reading Informational Text +0.05), bringing the distilled featurizer to 0.71 ± 0.03 overall — comparable to GPT-4.1-mini's 0.72.
• Featurization behavior aligns with humans. AnalyticScore's featurization reached 0.90, 0.72, 0.81 QWK across assessment areas, with the distilled Llama featurizer showing substantially high agreement in all three areas (QWK CIs of 0.90–0.90 for Science, 0.72–0.73 for Reading Informational Text, and 0.81–0.82 for Reading Literature).
• Humans themselves only moderately agree on the featurization task. Krippendorff's alpha among annotators was (0.718, 0.723) for Science, (0.696, 0.700) for Reading (Informational Text), and (0.672, 0.680) for Reading (Literature), falling in the 0.667 ≤ α < 0.8 range, which the authors treat as acceptable agreement.
• The middle label category is the weak point. F1 was near or above 0.9 for label 0 and moderate-to-high (0.6 to 0.93) for label 2, but only 0.68 down to 0.11 for label 1, which the authors attribute to the ambiguity and rarity of that category in human rating.
• An interpretable system does not automatically make an assessment valid. The authors warn that without deliberate auditing practices and workflows, model interpretations risk creating a false sense of validity and trustworthiness.

Methodology in Plain English

The authors start from stakeholders rather than from algorithms. Drawing on educational assessment literature, they map what test takers, assessment developers, and test users each need from a scoring explanation, then turn those needs into four design principles (FGTI) that any interpretable scorer should satisfy.

They then build AnalyticScore to show the principles can be implemented. It works in three phases:

1. Component extraction. Using the raw student responses (and optionally the item text), an LLM proposes a small set of "analytic components" — here, propositions — which are explicitly identifiable elements of student work. Extraction is capped at 15 components per scoring unit (except one item with a two-part scheme, where 15 were extracted from each part, for 30 total), because too many components would undermine interchangeability.
2. Featurization. Each response is labeled against each component with a hand-defined function: 2 if the response directly paraphrases the component, 1 if it partially paraphrases it, and 0 if it does not. This is implemented with chain-of-thought prompting (the generated "thoughts" are explicitly discarded) and a "first-to-three" aggregation rule. One-hot encodings of every component label are concatenated into a 3k-dimensional binary feature vector. To avoid the linearly growing cost of proprietary LLM calls, the authors fine-tuned Llama-3.1-8b-instruct with QLoRA on 10k sampled (response, component) pairs labeled by o4-mini.
3. Traceable scoring. Features feed an Immediate-Threshold variant of ordinal logistic regression, which sums weights into an interpretable "evidence value" compared against learned thresholds. Because every step is explicit, a human inspector can correct a feature and recompute the score — the paper shows how an "if instead…" explanation is generated this way.

Evaluation uses ASAP-SAS, described as the largest publicly available dataset of English constructed responses from US schoolchildren, covering 10 open-ended questions across three areas: Science, Reading (Informational Text), and Reading (Literature Text), with human raters double-scoring each response on a 3- or 4-point rubric. The original test set was kept, and the public training set split 8:2 into training and validation. Accuracy is measured with QWK against several baselines: few-shot GPT-4.1, fine-tuned BERT, DeBERTa, Llama-3.1-8b and Llama-3.1-8b-Instruct (with and without rubric input), AutoSAS, ASRRN, and NAM. NAM is the only baseline with interpretive aspects; its hand-crafted rubrics were replaced with AnalyticScore's extracted components. Separately, 50 (response, component) pairs per assessment area were scored by seven human annotators — five volunteers from an R1 institution and two of the study's authors, all with PhD-level training — and compared to model outputs using majority-vote human labels, Krippendorff's alpha, QWK, and class-wise F1 with 95% bootstrap confidence intervals.

Why This Matters

The paper reframes interpretable automated scoring from an isolated prediction problem into a stakeholder-centered process spanning assessment design, scoring, and iterative refinement. It also provides a concrete, measurable baseline — an interpretable system that stays close to black-box accuracy — which the field has lacked; the authors note that the 2023 NAEP Math Automated Scoring Challenge run by the US National Center for Education Statistics found no submission met its interpretability criteria even though several approached human-level accuracy.

Real-world applications implied by the paper:

• Summative assessment quality control: traceable explanations let test takers and their representatives inspect scoring decisions, supporting trust and allowing errors to be identified and corrected in a streamlined review process.
• Formative feedback for learners: step-by-step accounts of which features were observed and how they were processed can give students elaborative feedback about the gap between their work and the intended outcome.
• Assessment design and maintenance: developers can see which evidence the scorer detects, decide what parts of a scoring algorithm can be reused across items, spot group-specific biases or model drift, and target new data collection or modified training objectives.
• Validity argumentation for test users: educators, administrators, and policymakers get concrete evidence about whether extracted features and scoring logic reflect the rubric and construct definition, and whether internal score structure matches the construct of interest.

Industry relevance: the framework targets large-scale assessment settings where transparency demands and proprietary LLM costs both matter. Distilling a proprietary LLM featurizer into an open model addresses the linearly growing cost of featurization, and the interchangeable-subroutine design supports auditing workflows that testing organizations and educational technology providers would need to operate. The authors also flag student privacy risks when sending real assessment responses to proprietary LLMs.

Future Directions

• More expressive scoring modules. Ordinal logistic regression provides a strong baseline but may not capture the logical nuances of complex items; the authors suggest exploring traceable, interchangeable alternatives such as LLM workflows or agents whose modules each compute a specific evidentiary reasoning step with human-understandable intermediate outputs.
• More benchmarks. ASAP-SAS represents only a small fraction of assessment items, response types, domains, test-taker populations, and languages, so the authors call for curation of additional large-scale datasets of complex open-ended responses.
• Field validation of the stakeholder analysis. The needs analysis is a theoretical study derived from the assessment literature; the authors call for field studies in authentic assessment environments to validate and iteratively refine the design.
• Construct alignment and less ambiguous featurization. Beyond model-to-human alignment, the paper calls for measuring how analytic components and the featurization task align with the target constructs — requiring access to item design decisions the authors did not have — and for reducing the ambiguity that hurts agreement on the middle label category.

Target Audience

Researchers and practitioners working at the intersection of NLP, educational measurement, and assessment design: automated scoring researchers, psychometricians and validity specialists, assessment developers who build rubrics and items for large-scale testing programs, and interdisciplinary teams building interpretable or concept-based models. It is also useful for policymakers, testing organizations, and educational technology developers who must justify or audit automated scoring decisions, and for graduate students in learning analytics or AI in education looking for a principled framing plus a reproducible baseline.

Authors’ abstract

AI-driven automated scoring systems offer scalable and efficient means of evaluating complex student-generated responses. Yet, despite increasing demand for transparency and interpretability, the field has yet to develop a widely accepted solution for interpretable automated scoring to be used in large-scale real-world assessments. This work takes a principled approach to address this challenge. We analyze the needs and potential benefits of interpretable automated scoring for various assessment stakeholder groups and develop four principles of interpretability -- (F)aithfulness, (G)roundedness, (T)raceability, and (I)nterchangeability (FGTI) -- targeted at those needs. To illustrate the feasibility of implementing these principles, we develop the AnalyticScore framework as a reference framework. When applied to the domain of text-based constructed-response scoring, AnalyticScore outperforms many uninterpretable scoring methods in terms of scoring accuracy and is, on average, within 0.06 QWK of the uninterpretable SOTA across 10 items from the ASAP-SAS dataset. By comparing against human annotators conducting the same featurization task, we further demonstrate that the featurization behavior of AnalyticScore aligns well with that of humans.