FoundationalASSIST: An Educational Dataset for Foundational Knowledge Tracing and Pedagogical Grounding of LLMs

AI summary

Overview

Research area: AI in education — knowledge tracing, student modeling, and the pedagogical/psychometric grounding of large language models.

Technical level: Intermediate. The paper is readable without deep machine learning background, but familiarity with knowledge tracing and item response theory helps.

Scope: The paper introduces FoundationalASSIST, an English-language educational dataset built from ASSISTments data containing full question text, actual student responses, distractor selections, and Common Core alignment, and uses it to benchmark four frontier LLMs on two knowledge tracing tasks and four pedagogical grounding tasks.

What This Paper Is About

Existing knowledge tracing datasets represent student interactions as question identifiers plus a binary right/wrong label, which leaves nothing for a language model to reason about in natural language. The authors build FoundationalASSIST to preserve the text that ID-based datasets discard — full problem statements, the exact answers students gave, and which distractors they chose — and then test whether state-of-the-art LLMs can actually model student learning and understand what makes an assessment item effective.

Key Contributions

1. A new educational dataset. FoundationalASSIST contains 1,722,169 student interactions from 5,000 students across 3,395 unique mathematics problems and 462 knowledge components, drawn from student work between January 2019 and July 2024. It is described as the first English educational dataset providing question text, actual student responses, and distractor selection data together.

2. Response-level and distractor-level data. Rather than collapsing outcomes to correctness, the dataset retains which option(s) students selected on roughly 0.5 million multiple-choice interactions and the numerical or textual answers students wrote for the roughly 1.2 million fill-in-the-blank interactions, plus the full set of answer choices for each multiple-choice item.

3. Curriculum grounding. Every problem is drawn from the Illustrative Mathematics curriculum and aligned to Common Core State Standards, which encode grade level and content domain through codes such as "6.NS.C.6" ("Identify Opposites of Integers"). The dataset includes 355,307 skill annotations and spans Common Core mathematics domains from grades 3–8, though it is curated to be overwhelmingly composed of problems for grades 6–8.

4. A six-task benchmark and baseline evaluation. The authors define two Knowledge Tracing tasks (predicting binary correctness; predicting the student's exact response) and four Pedagogical Grounding tasks (difficulty comparison, discrimination comparison, and predicting the most- and least-selected distractors), then evaluate GPT-OSS-120B, Llama-3.3-70B-Instruct, and two Qwen3-Next-80B variants (Instruct and Thinking) on them.

Main Findings

• LLMs barely beat the trivial baseline on knowledge tracing. The evaluation set was sampled to have a 51.3% correctness rate, so always predicting "correct" yields 51.3% accuracy. The strongest model surpassed that baseline by only five percentage points, while other models achieved gains of merely one percentage point.

• Models show a strong optimistic bias. Llama-3.3-70B predicted correctness with 85.4% accuracy when students actually answered correctly but only 12.6% accuracy when they answered incorrectly — a bias the authors note would cause a tutor to systematically fail to recognize struggling students.

• Difficulty judgment is the one area of partial competence. The abstract reports models judging relative difficulty with up to 68.6% accuracy, while the introduction reports up to 80.0% accuracy when the two problems differ substantially. The authors describe this as partial success that only highlights the gaps elsewhere.

• Every model fails at discrimination. All four models perform below random chance (50%) on identifying which of two problems better distinguishes high-ability from low-ability students. The Qwen3-Next-80B "Thinking" variant had the highest discrimination accuracy at 46.9%, still under 50%.

• Distractor prediction is asymmetric. Models reached 47.9% accuracy at identifying which wrong answer students choose most often, beating the 35.8% random baseline, but fell 15 percentage points below chance at predicting which errors students rarely select.

• Extended reasoning has mixed effects. The Qwen3-Next-80B "Thinking" variant, which generates explicit reasoning chains, produced the highest discrimination accuracy (46.9%) while simultaneously producing the lowest distractor prediction accuracy (20.5%). The authors suggest explicit reasoning may help with abstract pedagogical judgments but cause overthinking on more concrete predictions.

• Accuracy varies by problem type. Single-select multiple choice items show 70.5% correctness (helped by guessing), fill-in-the-blank items show 61.7%, and multi-select and ordering items hover around 47–48%.

• Fill-in problems dominate and are harder to predict. Fill-in-the-blank items are 64.4% of unique problems but 69.7% of interactions (2,188 problems; 1,199,593 interactions). Multiple choice (select one) accounts for 795 problems (23.4%) and 328,329 interactions (19.1%); multi-select accounts for 380 problems (11.2%) and 210,927 interactions (12.2%); ordering problems account for 32 problems (0.9%) and 14,761 interactions (0.9%).

• Student histories are long relative to context windows. Students completed between 211 and 421 interactions each, averaging 344.4, with full histories approaching 32K tokens when encoded as prompts.

Methodology in Plain English

The authors started from the ASSISTments platform, which for over a decade has supplied standard knowledge tracing benchmarks but stripped out question text and detailed responses. They rebuilt a dataset in which a model sees that a student was asked how many batches of spice mix can be made with 9 cups of chili powder when one batch needs 3/4 cup, and that the student answered "27" instead of the correct "12" — an error that reveals a specific misconception. HTML and MathML in the problem text are converted to readable text through preprocessing code they release.

For knowledge tracing, each student's interactions are formatted chronologically as a history containing question text, problem type, the student's response, the correct answer, answer options, Common Core skill labels, correctness, hint count, whether the answer was revealed, and timestamps. The model then receives a new question and must predict either whether the student will answer correctly or the exact answer the student will give. Prompts ask for JSON output, and the reported evaluation covers question-level correctness and exact answer prediction — skill-level mastery prediction was excluded because the authors judged LLMs better at surface-level than latent constructs. Exact match is used for multiple choice and a 1% relative-error tolerance for fill-in items.

For pedagogical grounding, ground truth comes from Item Response Theory. The authors fit a two-parameter logistic model to student responses using Bayesian inference (py-irt), filtering to items with at least 50 responses to ensure reliable estimates; the paper reports fitting across 2,548 problems and states that 2,788 items were included in the IRT model. Models are then asked to choose which of two problems is harder, which is more discriminating, and which distractor students select most and least often. Difficulty pairs were stratified by the size of the difficulty gap (small 0.1–0.5, medium 0.5–1.0, large over 1.0), and discrimination pairs by gaps of 0.1–0.5 and 0.5–1.0, with equal numbers drawn from each stratum. The multiple-choice answer-prediction baseline was computed as the weighted mean of the 1/k guessing probabilities across problems.

Why This Matters

Impact on research. The paper argues that existing benchmarks such as ASSIST 2009, 2012, and 2017, EdNet, and Junyi Academy cannot support language-model research because they contain only identifiers and binary labels. FoundationalASSIST is positioned as the first dataset combining question text, actual responses, distractor data, and English text — the closest prior work, XES3G5M, offers question text but only binary correctness and is Chinese-only, while Eedi provides only images rather than text of questions and is limited to multiple-choice problems. The dataset is intended to serve as a foundational benchmark in the way the original ASSISTments datasets did for over a decade of knowledge tracing research.

Real-world applications:

• Computerized adaptive testing: selecting items that efficiently separate students by ability level, which requires the difficulty and discrimination judgments models currently handle poorly.
• LLM-based tutoring and feedback: targeting misconceptions requires anticipating which wrong answers students find tempting, but models fall well below chance at predicting rarely chosen distractors.
• Automated distractor generation: generating plausible wrong answers for practice items depends on the same knowledge of student errors that the benchmark shows is weak.
• Identifying at-risk students: the systematic optimistic bias found here (85.4% accuracy on correct answers versus 12.6% on incorrect ones for Llama-3.3-70B) would cause deployed systems to miss students who are falling behind.
• Item difficulty estimation: models show partial competence here, which the authors frame as the strongest but still incomplete capability.

Industry relevance. Commercial systems already use LLMs to generate problems, explain strategies, and hold Socratic dialogues with students, and products such as Khan Academy's Khanmigo and Duolingo's AI features demonstrate these capabilities at scale. The paper's results indicate that substantial advances are needed before LLMs can reliably support personalized learning at scale.

Future Directions

1. Fine-tuning and prompt engineering. The authors state that LLMs require significant development, prompt engineering, or fine-tuning before performing well on these tasks, leaving model adaptation as an open line of work.

2. Diagnosing the discrimination gap. Because every model performs below chance at judging which items are diagnostic, a natural question is whether training on the dataset's item text and response distributions can teach models what makes an item discriminating rather than merely difficult.

3. Understanding when reasoning helps. The mixed effects of the Qwen3-Next-80B Thinking variant — best at discrimination (46.9%), worst at distractor prediction (20.5%) — raise the question of which educational tasks benefit from explicit reasoning chains and which are harmed by them.

4. Extending beyond the evaluated tasks. The dataset supports directions the authors describe as previously impossible, including distractor analysis, cognitive student modeling of exact responses, analysis by Common Core grade level and domain, and analysis across problem formats — including the fill-in versus multiple-choice comparison the authors connect to prior work finding fill-in problems may promote deeper learning. The truncated paper text does not include the discussion, limitations, or conclusion sections (Sections 7 and 8), so any additional next steps proposed there are not reported here.

Target Audience

Researchers in educational data mining and knowledge tracing who need natural-language data for LLM-based student modeling; AI researchers studying whether LLMs reason about learners rather than just content; psychometricians interested in whether LLMs understand item difficulty and discrimination; and developers of adaptive testing, intelligent tutoring, or automated content generation systems who need to know where current models fall short before deployment.

Authors’ abstract

Can Large Language Models understand how students learn? As LLMs are deployed for adaptive testing and personalized tutoring, this question becomes urgent -- yet we cannot answer it with existing resources. Current educational datasets provide only question identifiers and binary correctness labels, rendering them opaque to LLMs that reason in natural language. We address this gap with FoundationalASSIST, the first English educational dataset providing the complete information needed for research on LLMs in education: full question text, actual student responses (not just right/wrong), records of which wrong answers students chose, and alignment to Common Core K-12 standards. These 1.7 million interactions from 5,000 students enable research directions that were previously impossible to pursue, from fine-tuning student models to analyzing misconception patterns. To demonstrate the dataset's utility, we evaluate four frontier models (GPT-OSS-120B, Llama-3.3-70B, Qwen3-Next-80B variants) on two complementary task families: Knowledge Tracing, testing whether LLMs can predict student performance on questions, and the exact answer a student will give; and \textbf{Pedagogical Grounding}, testing whether LLMs understand the properties that make assessment items effective. Our evaluation reveals significant gaps in current LLM capabilities. Every model barely achieves a trivial baseline on knowledge tracing. All models fall below random chance on item discrimination, indicating that LLMs do not understand what makes one problem more diagnostic than another. Models do show competence at judging relative difficulty (up to 68.6%), but this partial success only highlights the gaps elsewhere. These results establish that substantial advances are needed before LLMs can reliably support personalized learning at scale. We release FoundationalASSIST to support progress on these foundational challenges.