A Unified Framework to Elicit Structured Feedback for Interpretable Multi-Trait Essay Scoring

AI summary

Overview

• Research area: Natural Language Processing, specifically Automated Essay Scoring (AES) with large language models and LLM post-training.
• Technical level: Advanced (assumes familiarity with supervised fine-tuning, reinforcement learning, GRPO, and quadratic weighted kappa).
• Scope: The paper introduces HiFTS, a unified autoregressive framework that generates rubric-grounded hierarchical chain-of-thought feedback before predicting trait-level and holistic essay scores, together with CFMS-34, a new 951-essay Chinese multi-trait essay scoring dataset annotated with 34 rubric-based traits.

What This Paper Is About

Multi-trait automated essay scoring requires reasoning across interdependent rubric traits rather than predicting isolated scores, but existing feedback-enhanced methods tend to either decouple feedback from scoring or score each trait independently, weakening score–feedback consistency and rubric alignment. The authors propose HiFTS, which distills hierarchical chain-of-thought feedback from a teacher LLM, trains student models to generate feedback and scores jointly in a single autoregressive process, and aligns them with Group Relative Policy Optimization under a composite reward. They also release CFMS-34 to test whether methods that succeed on the English ASAP++ benchmark transfer to Chinese multi-trait scoring.

Key Contributions

1. CFMS-34 dataset: An expert-annotated, dual-rated Chinese multi-trait AES dataset of 951 primary-school essays with holistic scores and 34 fine-grained rubric-based sub-traits, designed to promote cross-lingual validation of multi-trait AES methods.
2. HiFTS framework: A unified LLM post-training framework that elicits structured hierarchical feedback and multi-trait knowledge in one autoregressive generation, with warmup SFT on teacher-generated CoT followed by GRPO alignment using a composite reward.
3. Prior-guided inference: Coordination of the LLM with a lightweight BERT regressor at inference time, where a coarse holistic prior score is inserted into the system prompt as a soft semantic anchor rather than as the final prediction.
4. Cross-lingual evaluation: Experiments showing that methods performing well on ASAP++ do not necessarily succeed on CFMS-34, while HiFTS improves both holistic and trait-level scoring and feedback quality on both datasets.

Main Findings

• Frontier LLMs struggle on CFMS-34: Using the paper's zero-shot scoring prompt, GPT-4.1 reaches QWK 0.4222, Gemini-3 reaches 0.4363, Qwen3-max reaches 0.4201, DeepSeek-V3 reaches 0.4130, DeepSeek-R1 reaches 0.3271, and Doubao-Seed-1.6 reaches 0.1595, all against a human–human QWK of 0.57. The paper notes the human QWK measures agreement between the two expert raters, while model QWKs are computed against resolved human scores, so the two are not strictly comparable despite being reported together.
• Calibration issues in frontier models: Mean predicted scores range from 2.0743 (Doubao-Seed-1.6, substantial under-prediction) to 3.5792 (DeepSeek-V3, over-prediction) against a human mean of 3.64.
• HiFTS improves scoring on CFMS-34: With Qwen2.5, HiFTS reaches Overall QWK 0.656, Traits QWK 0.439, and MSE 0.957, versus HiFTS-SFT at 0.528, 0.387, and 1.447. With Qwen3, HiFTS reaches the best results reported in the paper: Overall QWK 0.677, Traits QWK 0.453, and MSE 0.741, versus HiFTS-SFT at 0.557, 0.395, and 1.379.
• Baselines degrade on CFMS-34: HISK scores 0.405 Overall / 0.309 Traits, STL-LSTM 0.484 / 0.365, MTL-BiLSTM 0.519 / 0.396, and ArTS 0.264 / 0.149, while RMTS (rationale-enhanced) scores 0.296 / 0.303 with MSE 1.431.
• Results on ASAP++: With Qwen2.5, HiFTS reaches 0.718 prompt-averaged QWK and 0.704 trait QWK (WinRate 0.550); with Qwen3, 0.726 and 0.711 (WinRate 0.613). Traditional baselines on ASAP++ are HISK 0.644 / 0.611, STL-LSTM 0.684 / 0.656, MTL-BiLSTM 0.665 / 0.638, ArTS 0.717 / 0.695, and RMTS 0.720 / 0.704.
• Trait-level gains on ASAP++: In the per-trait table with Qwen2.5, HiFTS attains the best mean QWK of 0.704, with per-trait values including Overall 0.758, Content 0.734, Purpose/Audience 0.749, Language 0.702, Narrative 0.724, Organization 0.681, Conventions 0.684, Word Choice 0.686, Sentence Fluency 0.688, Style 0.715, and Voice 0.626.
• Feedback quality: GRPO-aligned variants are preferred over their SFT counterparts in pairwise WinRate judged by DeepSeek-V3.2. On the rule-based grounding metric, HiFTS-SFT scores 0.60, HiFTS-PPO w/o prior 0.72, HiFTS-PPO 0.74, HiFTS w/o prior 0.74, and full HiFTS 0.78.
• Ablations: The BERT prior alone reaches 0.489 Overall QWK, Qwen zero-shot 0.235 Overall / 0.154 Traits, and Qwen SFT without CoT 0.432 / 0.193. HiFTS-PPO w/o prior reaches 0.572 / 0.417 (WinRate 0.603), HiFTS-PPO 0.647 / 0.407 (0.651), HiFTS w/o prior 0.579 / 0.420 (0.621), while HiFTS with a random prior drops to 0.502 / 0.298 and with a constant prior to 0.411 / 0.252.
• Hyperparameters: α = 0.8 gives the best trade-off between score-oriented and feedback-oriented optimization, and δ = 0.1 is the fixed structural bonus/penalty. Sliding-window sizes give Overall / Traits of 0.612 / 0.422 at h = 16, 0.641 / 0.432 at h = 32, 0.656 / 0.439 at h = 64, and 0.659 / 0.443 at h = 128; h = 64 is used in the main experiments.
• Dataset reliability: 97.4% of samples have the two experts agreeing on the overall score within one point; average agreement across the 34 sub-traits is 0.48 exact match, 0.90 within one point, and 0.41 QWK.

Methodology in Plain English

The authors build two things: a dataset and a method.

For the dataset, they collected 951 Chinese essays written by primary school students during in-class examinations, using textbook unit-level writing prompts under a 50-minute time limit. Two trained experts in Chinese language education independently scored each essay on 34 fine-grained sub-traits grouped into four dimensions (Content, Structure, Expression, Conventions) plus an overall score, all on a 0–5 scale. The data is split 8:1:1 into training, development, and test sets, with the test set deliberately reserving samples where raters fully agreed on the overall score.

For the method, HiFTS has three stages. First, a teacher LLM (Gemini-3) generates hierarchical CoT feedback following a top-down order: global understanding of the essay, then dimension-level analysis, then sub-trait evaluation, conditioned on the essay, rubric definitions, and the labeled scores. These feedback traces are concatenated with the labeled trait and overall scores to form a target sequence, and a student model is fine-tuned on it with standard token-level cross-entropy loss, so the target sequence looks like the natural continuation of an instructional prompt. Second, GRPO aligns the student using a composite reward that combines QWK-based agreement (holistic QWK over a sliding window of recent generations plus trait-level QWK), negative MSE for calibration, cosine similarity between generated and teacher feedback embeddings for feedback quality, and a fixed rule-based structural bonus or penalty. Third, at inference a lightweight BERT regressor estimates a coarse holistic prior score, which is placed in the system prompt as guidance; the paper emphasizes this prior is not the final prediction, and that random or constant priors hurt performance.

Evaluation uses QWK for holistic and average trait agreement, MSE for calibration on CFMS-34, pairwise WinRate judged by DeepSeek-V3.2 (only comparing each SFT model to its GRPO-aligned counterpart), and a rule-based grounding score computed as 0.5 × Coverage + 0.3 × Trait-level Reasoning + 0.2 × Reasoning Density. Models trained are Qwen2.5-7B and Qwen3-4B, with GRPO sampling G = 4 responses per input, KL coefficient β = 0.02, learning rate 1 × 10⁻⁶, decoding at temperature 0.6 and top-p 0.9. Feedback embeddings use bge-small-zh-v1.5 for CFMS-34 and all-MiniLM-L6-v2 for ASAP++. For CFMS-34, 20 core traits with the highest Pearson correlation to the overall score are used for modeling and evaluation, and the two raters' 0–5 annotations are summed into a 0–10 scale for training and evaluation.

Why This Matters

• Research impact: The paper argues that current supervised approaches lack a unified framework that generates multi-trait feedback and scores jointly in a single autoregressive process, and it provides both a method and a non-English benchmark to test whether gains transfer across languages. Its finding that strong ASAP++ performance does not guarantee success on CFMS-34 is a direct challenge to English-only validation of multi-trait AES methods.
• Real-world applications:
  • Formative classroom writing assessment, where teachers and students can inspect rubric-grounded feedback alongside scores.
  • Chinese-language writing instruction, where 34 fine-grained sub-traits map to a pedagogical rubric of Content, Structure, Expression, and Conventions.
  • Cross-lingual or multilingual AES systems that need to transfer between English and Chinese essay settings.
  • Interpretable scoring pipelines that provide evidence for automated judgments instead of black-box score-only outputs.
• Industry relevance: The framework offers a post-training recipe (teacher distillation, SFT, GRPO with a score-and-feedback reward, plus a lightweight BERT prior) that educational technology providers could apply to align LLM scoring with existing rubrics while keeping score prediction calibrated. The paper states the dataset and system are intended for research and educational support, not as the sole basis for high-stakes educational decisions without human oversight.

Future Directions

1. Alternative reasoning organizations: HiFTS uses a predefined global-to-local autoregressive order; the authors suggest exploring alternative generation orders or more flexible feedback organizations.
2. Broader evaluation: Testing on more writing genres, learner populations, and real classroom interactions where students and educators directly engage with the generated feedback.
3. Scalability and educational impact: Establishing how well the approach scales and whether it produces measurable educational benefit beyond benchmark QWK.
4. Trait selection and reasoning length: The paper notes that including too many traits lengthens the reasoning chain and complicates structural constraints, reducing stability in rubric adherence, and that marginal traits may add noise — a trade-off left for further study.

Target Audience

Researchers in automated essay scoring and educational NLP; NLP researchers working on LLM post-training, chain-of-thought distillation, and reinforcement learning from verifiable rewards; Chinese-language education researchers and assessment specialists; and developers building interpretable, rubric-aligned writing feedback systems. Readers need some background in LLM fine-tuning and RL-based alignment to follow the method sections, while the dataset and evaluation portions are accessible to assessment researchers.

Authors’ abstract

Multi-trait Automated Essay Scoring (AES) requires rubric-grounded reasoning across interdependent traits, rather than isolated score prediction. Existing feedback-enhanced methods often decouple feedback from scoring or assess traits independently, weakening score--feedback consistency and rubric alignment. We propose HiFTS, a unified autoregressive framework that generates hierarchical CoT feedback before predicting trait-level and holistic scores. HiFTS distills rubric-grounded hierarchical CoT feedback from a teacher LLM and trains student models to jointly generate feedback and scores. HiFTS further applies Group Relative Policy Optimization with a composite reward balancing score agreement, calibration, feedback quality, and structural validity. At inference, a lightweight global prior provides holistic guidance to reduce drift during long-form reasoning. We also introduce CFMS-34, a Chinese multi-trait AES dataset with 951 essays annotated with holistic scores and 34 rubric-based traits. Experiments on CFMS-34 and ASAP++ show that HiFTS achieves strong holistic and trait-level scoring while producing coherent, rubric-aligned feedback.