Tracing Mathematical Proficiency Through Problem-Solving Processes

AI summary

Overview

Research area: Machine learning for education — specifically Knowledge Tracing (KT), combined with LLM-based structured extraction pipelines (cs.LG).

Technical level: Intermediate. Readers should be comfortable with knowledge tracing baselines, deep sequential models, loss functions, and standard KT evaluation metrics (AUC, ACC), but the paper explains its LLM pipeline in accessible terms.

Scope: The paper reformulates knowledge tracing to include students' written problem-solving processes, releases a first-of-its-kind dataset with those processes, and proposes an LLM pipeline that converts them into interpretable Mathematical Proficiency (MP) signals that improve prediction across ten KT architectures.

What This Paper Is About

Traditional knowledge tracing predicts whether a student will answer a future question correctly using only past correct/incorrect outcomes, question metadata, and knowledge concepts. This collapses the student's actual reasoning into a single binary signal, making predictions hard to explain and blind to what teachers actually care about — how a student solved a problem. The paper's goal is to bring the written problem-solving process (PSP) into the KT objective, turning it into structured, interpretable mathematical proficiency scores that both improve accuracy and justify predictions.

Key Contributions

1. KT-PSP task formulation. A reformulation of knowledge tracing where the interaction sequence becomes (q_t, c_t, r_t, p_t) — question, concept, correctness, and the student's step-by-step solution process. The prediction target stays the same, but the model now has access to reasoning traces.

2. KT-PSP-25 dataset. The first mathematical KT dataset that records real-world PSP for every student–problem interaction: 22,289 sessions, 1,343 students, 2,696 problems, 490 knowledge components, collected from an in-house tablet-based platform between November 2024 and July 2025. Released under CC BY-NC 4.0.

3. StatusKT framework. A teacher–student–teacher three-stage LLM pipeline (GPT-5) that converts raw handwritten PSP into rubric-style Mathematical Proficiency ratios across four dimensions: conceptual understanding, strategic competence, procedural fluency, and adaptive reasoning.

4. Empirical validation with interpretability. StatusKT improves AUC and accuracy over strong DLKT baselines in most configurations, shows statistically significant gains on most architectures, and helps under cold-start conditions where only the first five interactions are available.

Main Findings

• Raw process text is not enough. Concatenating an encoded PSP string ("Baseline + PSP") to standard KT inputs produced inconsistent results, sometimes performing worse than the baseline — the authors attribute this to OCR noise and the mismatch between free-form text and a discrete prediction objective.
• Structured MP signals help. Adding StatusKT's extracted proficiency ratios improved AUC and ACC for most architectures, with the clearest gains in RNN-based (DKT, DKVMN) and transformer-style models (SAINT, AKT, SimpleKT, stableKT). For example, DKVMN rose from 0.6049 to 0.6220 AUC and SAINT from 0.6201 to 0.6401.
• Gains are mostly statistically robust. Paired t-tests across 10-fold splits showed p < 0.05 for the majority of architectures, with moderate to large Cohen's d effect sizes. Only DKT+ and SAKT failed to reach significance — the two with the weakest baselines.
• Cold-start improves broadly. When limited to the first five interactions per student, StatusKT raised ACC across every architecture tested and matched or exceeded baseline AUC in most cases, suggesting process-derived signals act as informative priors when correctness data is sparse.
• MP failures are scope failures. Case analysis found that when MP hurts predictions, it is usually because indicators are over-enriched (e.g., including optional alternative methods) or misaligned with the actual problem requirements, inflating perceived skill demands.
• Proficiency indicators correlate with answer quality. In human evaluation of 590 indicators across 50 items, 55.8% were rated fully appropriate, 35.8% partially valid, and 8.3% irrelevant. For correct student answers, 67.6% of indicators received the top rating versus 44.6% for incorrect answers. Adaptive reasoning scored worst (mean 1.13), indicating LLMs struggle with meta-justification and verification steps.
• OCR is usable but imperfect. Manual review of 100 handwritten solution samples found 85% of OCR outputs preserved mathematical meaning; failures were concentrated in structurally complex handwritten expressions rather than systematic misrecognition.

Methodology in Plain English

The researchers start from a straightforward observation: a teacher does not judge understanding from a right-or-wrong mark alone, but from the work shown. They translate that idea into a machine pipeline.

First, they collect real sessions from a math learning platform, where students write solutions by hand on a tablet. Because the handwriting cannot be released for privacy reasons, they run it through a GPT-5-based OCR step that transcribes the work into LaTeX while ignoring diagrams, and then a second GPT step cleans up the transcription.

Second, they build a three-stage LLM pipeline that mimics how an examiner would work. A "teacher" model reads the problem and its concept and writes a short list of targeted questions — indicators — that a student would need to answer correctly to show mastery (for example, "Did the student recognize this as a partial sum of an arithmetic sequence?"). A "student" model then reads the actual written solution and answers each indicator based only on what the student wrote. A second "teacher" model judges whether each generated answer genuinely satisfies its indicator, yielding a 0 or 1 per indicator. Aggregating these into each of the four proficiency dimensions gives a per-dimension MP ratio between 0 and 1.

Third, these ratios become extra supervision for the KT model. The model predicts both future correctness and the MP ratios for the current problem, trained with a combined loss: binary cross-entropy for correctness plus a weighted mean-squared-error term over the four proficiency dimensions. The MP prediction acts as an auxiliary task that forces the model to internalize proficiency-related patterns rather than just memorizing response sequences.

Finally, they test the approach by plugging it into ten existing KT architectures under three settings: the untouched baseline, a version with raw PSP text embeddings, and StatusKT. They also run a strict cold-start experiment and a qualitative case analysis to understand when MP signals help and when they backfire.

Why This Matters

Impact on research. The paper argues that outcome-centric supervision is a ceiling on KT performance and interpretability. By defining a new task (KT-PSP), releasing a dataset with process traces, and demonstrating a reusable extraction pipeline, it opens a direction where KT models are evaluated on whether they capture reasoning, not just whether they predict the next click. It also gives education researchers a scalable way to operationalize the Findell et al. framework of mathematical proficiency, which previously required human interviews.

Real-world applications:

• Adaptive tutoring platforms can use MP ratios to decide whether a student needs conceptual review versus procedural drill, rather than just re-serving problems they got wrong.
• Early-intervention systems can flag students who answer correctly but show weak reasoning traces — a group invisible to correctness-only models.
• Teacher dashboards can surface proficiency breakdowns per topic, aligning automated assessment with the dimensions teachers already use.
• Automated grading and feedback tools can generate rubric-aligned commentary on written work instead of a single score.

Industry relevance. The dataset and pipeline come from Mathpresso (QANDA), a commercial math education company, making the setting directly applicable to edtech products. The four proficiency dimensions map cleanly onto curriculum design and content tagging. The main practical obstacle is cost: the pipeline requires two LLM calls per student–problem interaction at evaluation time, which the authors acknowledge is heavy compared to lightweight deep KT models.

Future Directions

• Reduce LLM dependence. The authors explicitly plan to explore distillation and task-specific smaller models so the pipeline does not require GPT-5 for every interaction, since indicator extraction can be precomputed but evaluation calls cannot.
• Scope-controlled MP. Separating core indicators from enriched or optional ones could address the MP-hurt failure mode where over-enriched rubrics distort predictions.
• Preserve visual reasoning. The OCR step discards diagrams and graphs, which carry real cognitive signal in geometry and graph-based problems. A multimodal pipeline could recover this.
• Beyond mathematics and single platforms. The dataset covers one subject, one platform, and 1,343 students. Testing transfer to other domains and more diverse learner populations is an open question, as is whether MP indicators generalize across curricula.

Target Audience

This paper is most valuable to educational data mining and knowledge tracing researchers, edtech engineers building adaptive learning systems, and applied ML practitioners interested in LLM pipelines that convert unstructured human work into structured supervision. Learning scientists and assessment researchers will also find the operationalization of mathematical proficiency relevant, though they may want more validation of the indicator rubric. Readers without background in KT metrics or sequence modeling will need to consult the cited baselines to follow the experimental tables.

Authors’ abstract

Knowledge Tracing (KT) aims to model student's knowledge state and predict future performance to enable personalized learning in Intelligent Tutoring Systems. However, traditional KT methods face fundamental limitations in explainability, as they rely solely on the response correctness, neglecting the rich information embedded in students' problem-solving processes. To address this gap, we propose Knowledge Tracing Leveraging Problem-Solving Process (KT-PSP), which incorporates students' problem-solving processes to capture the multidimensional aspects of mathematical proficiency. We also introduce KT-PSP-25, a new dataset specifically designed for KT-PSP. Building on this, we present StatusKT, a KT framework that employs a teacher-student-teacher three-stage LLM pipeline to extract students' Mathematical Proficiency (MP) as intermediate representation. In this pipeline, the teacher LLM first extracts problem-specific proficiency indicators, then a student LLM generates responses based on the student's solution process, and a teacher LLM evaluates these responses to determine mastery of each indicator. The experimental results on KT-PSP-25 demonstrate that StatusKT improves the prediction performance of existing KT methods. Moreover, StatusKT provides interpretable explanations for its predictions by explicitly modeling students' mathematical proficiency. Code is available \href{https://github.com/jungyangpark/KT-PSP-25}{here}.