KASER: Knowledge-Aligned Student Error Simulator for Open-Ended Coding Tasks

AI summary

Overview

• Research area: Machine learning for education (ML4Ed) / computer science education, combining knowledge tracing, large language model fine-tuning, and reinforcement learning for student simulation.
• Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning, GRPO/PPO-style policy optimization, reward shaping, knowledge tracing, and code-similarity metrics such as CodeBLEU and IoU.
• Scope: The paper introduces KASER, a GRPO-trained simulator that predicts the code and the specific errors a student will produce on an open-ended coding problem, conditioned on an interpretable profile of that student's mastery over knowledge components (KCs).

What This Paper Is About

Training LLMs to imitate how students write code is hard for two reasons: students make errors that come from misconceptions or missing knowledge rather than from clean expert code, and student submissions vary widely in syntax, style, and solution approach. Existing approaches either ignore student knowledge explicitly or suffer from mode collapse, producing correct code when the student would not have written it. KASER's goal is to align predicted student errors with an explicit, interpretable estimate of what the student knows.

Key Contributions

1. A GRPO-based training method that aligns errors in generated student code with an input student knowledge profile. The reward function has three parts: a standard code-similarity reward, a group-level diversity reward to prevent mode collapse, and an error-overlap reward (IoU between predicted and ground-truth error sets) that goes beyond surface code similarity.
2. A student knowledge estimator that converts a student's prior code history into a k-dimensional mastery vector over knowledge components, using a linear layer plus sigmoid on a d-dimensional knowledge-tracing state, and a compensatory model that averages mastery over the KCs associated with a problem.
3. An automatic error annotation pipeline built on OpenAI's o4-mini reasoning model: per-submission error generation across syntax, runtime, and logical categories, clustering of error descriptions using Sentence-BERT embeddings and Hierarchical Agglomerative Clustering with cosine similarity, and LLM-generated representative labels for each cluster. Two human evaluations support the reliability of this LLM-as-a-judge setup.
4. A two-level quantitative evaluation on two real-world datasets (per-student-problem pair and per-problem), plus an ablation study and a qualitative case study showing how predicted errors change with mastery level.

Main Findings

• Per-student-problem results: KASER outperforms all baselines on both datasets with statistical significance (p < 0.05 under paired t-test). On CodeWorkout (Java), KASER reaches CodeBLEU@1 of 0.524, CodeBLEU@5 of 0.599, IoU@1 of 0.157, and IoU@5 of 0.276, versus the strongest fine-tuning baseline, Student SFT, at 0.501, 0.565, 0.115, and 0.244. On FalconCode (Python), KASER reaches 0.668, 0.692, 0.178, and 0.303, versus Student SFT at 0.642, 0.670, 0.153, and 0.270.
• Prompting baselines lag: PersonaPrompt and ICL perform poorly on both datasets. On CodeWorkout, ICL (7B) reaches CodeBLEU@1 of 0.463 and IoU@1 of 0.021; PersonaPrompt (7B) reaches 0.407 and 0.019. The authors attribute this to pretrained models having an overly optimistic bias toward generating correct code even when the student's history shows low mastery.
• Error-type breakdown: At K=1 on CodeWorkout, both KASER and Student SFT achieve perfect precision (1.00) on logical, runtime, and syntax errors. KASER improves recall and F1 across all three categories: logical F1 0.329 versus 0.279, runtime 0.231 versus 0.220, syntax 0.178 versus 0.130.
• Per-problem results: KASER leads on code diversity and error coverage. On CodeWorkout it reaches cosine distance 0.088, CodeBLEU complement max 0.520, error-coverage IoU 0.750, and Chi-squared distance 104.97. On FalconCode it reaches 0.298, 0.643, 0.817, and 45.77.
• Ablation confirms all three rewards matter: Removing the similarity reward causes a 16% decrease on CodeBLEU@5 on CodeWorkout. Removing the error reward causes a 36% decrease on IoU@1 on CodeWorkout and an 11% increase in Chi-squared distance on CodeWorkout. Removing the diversity reward causes a 12% decrease on cosine distance on FalconCode.
• Qualitative knowledge alignment: In a three-student case study on a Java sum problem with conditional logic, KASER predicted an assignment operator instead of a comparison (c=13) and a wrong arithmetic return for a low-mastery student, a missing return statement and omitted final else block for a second low-mastery student, and correct code for a high-mastery student. Student SFT predicted correct code for all three.
• Syntax errors remain hard: IoU values are generally low, around 0.1 against a maximum of 1. KASER predicts logical and runtime errors such as "missing logic branch" well but struggles with syntax errors such as "unbalanced brace," because pretrained LLMs rarely produce syntactically invalid code.
• Judge reliability: Human evaluations on 50 randomly sampled FalconCode problems give an average F1 of 0.749 between LLM and human labels, with inter-rater F1 of 0.814; stratified F1 is 0.830 for syntax, 0.893 for runtime, and 0.917 for logical errors. Error-type classification agreement (Jaccard) is 0.800 for syntax, 0.778 for runtime, and 0.872 for logical errors, on a task with 10 possible errors per problem.

Methodology in Plain English

KASER has two stages. First, a knowledge estimator reads a student's prior code submissions and produces a mastery score between 0 and 1 for each knowledge component the problems test. A linear layer plus sigmoid converts an internal knowledge-tracing state into these interpretable scores, and an average over the KCs required by a problem yields an overall mastery estimate; the model is trained with binary cross-entropy against whether the student's submission was correct.

Second, a large language model (Qwen2.5-Coder 7B Instruct) is fine-tuned to predict the student's next code submission. The prompt contains the problem statement plus the student's mastery level on each relevant KC, so the model sees both the task and a snapshot of what the student knows.

The key step is reinforcement learning with GRPO. For each problem-and-mastery input, the model samples a group of candidate codes. Each candidate gets a reward made of three equally weighted parts on a 0-to-1 scale: (1) CodeBLEU similarity to the actual student code, (2) IoU overlap between the errors in the predicted code and the errors in the real student code, where a judge model extracts the error sets and both-correct predictions score 1, and (3) a diversity term defined as one minus the maximum CodeBLEU to any other code in the same group. The diversity term is measured within the group rather than against a fixed target, which is what stops all samples from collapsing onto the same output. Rewards are z-score normalized within the group into advantages, and the policy is updated with a clipped surrogate objective regularized by a KL penalty toward a frozen reference model.

Evaluation happens at two levels. At the per-student-problem level, the model generates either one or five codes per student-problem pair and the best-scoring one is compared to the ground truth. At the per-problem level, all predicted codes for an unseen problem are pooled and compared against the pooled ground-truth errors and code embeddings (using Qwen3-Embedding). Both datasets are split across students or problems respectively, in 5-fold cross-validation, and the GRPO group size is 5.

Why This Matters

• Research impact: The paper shows that grounding LLM-based student simulation in an explicit, interpretable knowledge profile produces both more accurate code prediction and better error anticipation than fine-tuning alone, and that a group-level diversity reward is an effective antidote to the mode collapse that supervised fine-tuning on real student code exhibits.
• Real-world applications:
  • Teacher-facing diagnostics that highlight which misconceptions and knowledge gaps are likely in a class before students submit.
  • Personalized feedback for students, generated from a simulated prediction of the errors they are likely to make.
  • Practice and tutoring systems that can generate plausible erroneous solutions to test whether feedback or hints fire correctly on student-like code.
  • Curriculum and problem design, by checking which errors an anticipated knowledge profile would produce on a given problem.
• Industry relevance: The work sits directly in the space of intelligent programming tutors and CS education platforms. It also contributes a reusable recipe for post-training code LLMs with hybrid, verifiable rewards, and the authors state that the code will be publicly released at https://github.com/umass-ml4ed/code_personalization.

Future Directions

• Simulate debugging behavior by analyzing the changes across multiple submissions a student makes to the same problem, rather than only the first submission.
• Explore post-training techniques that can make pretrained LLMs produce syntax errors, since KASER currently predicts logical and runtime errors more effectively than syntax errors.
• Investigate weighting the three reward terms, which the authors left fixed at equal weighting and explicitly defer to future work.
• The paper's list of future directions is truncated in the provided content, so any additional directions beyond those above are not reported here.

Target Audience

Researchers and practitioners in machine learning for education, educational data mining, and knowledge tracing who study student modeling on open-ended tasks; CS education researchers and instructors interested in error diagnosis; and NLP/RL engineers working on post-training LLMs with multi-component reward functions for code generation. Readers without a background in RL fine-tuning or knowledge tracing will find the methodology sections demanding.

Authors’ abstract

Open-ended tasks, such as coding problems that are common in computer science education, provide detailed insights into student knowledge. However, training large language models (LLMs) to simulate and predict possible student errors in their responses to these problems can be challenging: they often suffer from mode collapse and fail to fully capture the diversity in syntax, style, and solution approach in student responses. In this work, we present KASER (Knowledge-Aligned Student Error Simulator), a novel approach that aligns errors with student knowledge. We propose a training method based on reinforcement learning using a hybrid reward that reflects three aspects of student code prediction: i) code similarity to the ground-truth, ii) error matching, and iii) code prediction diversity. On two real-world datasets, we perform two levels of evaluation and show that: At the per-student-problem pair level, our method outperforms baselines on code and error prediction; at the per-problem level, our method outperforms baselines on error coverage and simulated code diversity.