EDU-CIRCUIT-HW: Evaluating Multimodal Large Language Models on Real-World University-Level STEM Student Handwritten Solutions

AI summary

Overview

Research area: Computer Vision / multimodal large language model evaluation, applied to AI-enabled education (automated grading of handwritten STEM work).

Technical level: Intermediate. The paper assumes familiarity with multimodal LLMs, auto-grading pipelines, and standard evaluation metrics (precision/recall/F1, agreement rates), but the problem setup and findings are explained without deep technical machinery.

Scope in one sentence: The paper releases EDU-CIRCUIT-HW, a dataset of 1,334 real university-level handwritten circuit-analysis solutions, and uses it to measure both how accurately multimodal LLMs read that handwriting and how those reading errors cascade into automated grading.

What This Paper Is About

Multimodal LLMs are increasingly proposed as tools to reduce teachers' grading workload, but it is unclear whether they can actually read messy, real student handwriting — interleaved equations, hand-drawn diagrams, and prose reasoning — before they grade it. Existing benchmarks mostly test isolated expressions, K–12-level math, or judge models only by downstream task outcomes, which can hide recognition errors that never happen to affect the grade. The authors build an authentic university STEM dataset with expert-verified transcriptions and grading reports so that recognition fidelity and grading quality can be evaluated separately and together.

Key Contributions

1. The EDU-CIRCUIT-HW dataset: 1,334 authentic handwritten solutions from 29 students in an undergraduate circuit analysis course, with 62 unique questions, expert-labeled reference grades along five rubric dimensions, and expert-proofread near-verbatim transcriptions for a 513-sample subset.

2. A diagnostic evaluation workflow that separates upstream handwriting recognition from downstream auto-grading, using a validated LLM-as-a-judge pipeline to detect recognition errors and a four-category taxonomy (Symbolic & Character, Structural & Notational, Diagrammatic, Textual & Logical) to classify them.

3. Quantification of error cascading via the Error Impact Rate (EIR), showing how many recognition errors actually propagate into grading discrepancies, and how this changes across five closed-source models (Gemini-3-Pro-Preview, Gemini-2.5-Pro, Qwen3-VL-PLUS, Claude-4.5-Sonnet, GPT-5.1) and one open-source model (Qwen3-VL-8B-Thinking).

4. A deployment-oriented case study demonstrating a human-in-the-loop regrading module that uses identified error patterns to flag likely recognition failures, routing a small fraction of assignments to a human grader and improving grading robustness on the held-out test set.

Main Findings

• Recognition errors are pervasive but partly invisible. Gemini-3-Preview reaches a Sample Error Rate (SER) of 37.62% yet an EIR of only 7.60%, meaning most of its recognition mistakes do not change the grade. As recognition degrades, the EIR rises from 7.60% (Gemini-3-Preview) to 19.60% (Qwen3-VL-8B-Thinking).

• Fine-grained grading exposes the damage. Going from Gemini-3-Preview to Qwen3-VL-8B-Thinking, the grading gap widens from 12.86% in binary agreement to 17.16% (type agreement) and 17.35% (point agreement). Rows in Table 5 show Gemini-3-Preview at 87.91 / 78.17 / 74.27 (binary / type / point) versus Qwen3-VL-8B-Thinking at 75.05 / 61.01 / 56.92.

• Humans beat models on strict criteria, but not on coarse ones. The graduate teaching assistant scores 83.63 (binary), 82.46 (type), and 81.29 (point). The Gemini series exceeds the human on binary agreement (e.g., 87.91), but every evaluated MLLM falls behind the human on type and point agreement.

• Perfect recognition is better than any recognizer tested. When a human expert supplies the recognized text, the GPT-5.1 grader reaches 89.47 (binary), 78.36 (type), and 74.46 (point).

• Surface errors hurt grading most. Symbolic & Character errors have EIRs consistently around 20%, while Diagrammatic and Textual & Logical errors frequently fall below 10% — suggesting current auto-graders rely heavily on symbolic matching rather than structural or diagrammatic reasoning.

• Model strengths vary by error type. Qwen3-VL-8B-Thinking records 98 Diagrammatic errors, slightly better than the larger Gemini-2.5-Pro at 103, while commercial models generally lead on Textual & Logical errors.

• The LLM-judge is reliable. Across 186 handwritten solutions and more than 5,000 items, sample-level accuracy exceeds 0.95 for all three tested models (GPT-5.1: 0.9516; Claude-4.5-Sonnet: 0.9839; Qwen3-VL-Plus: 0.9516; total 0.9624), with item-level precision, recall, and F1 close to or above 0.9 (total precision 0.9193, recall 0.9080, F1 0.9136).

• Targeted regrading works with minimal human effort. On the 821-sample test set, regrading raises Gemini-2.5-Pro from 85.02 / 74.91 / 69.91 to 86.48 / 77.34 / 74.42, and GPT-5.1 from 82.34 / 72.23 / 66.87 to 86.60 / 78.93 / 75.76. This requires human regrading for only 3.3% of assignments (Gemini-2.5-Pro recognizer) or 4.4% (GPT-5.1 recognizer), with the rest regraded by an LLM (20.6% and 25.1% respectively).

• Dataset composition. The observation set has 513 solutions from 11 students (62 questions), with verified recognition and ground-truth grades, and a student solution accuracy of 72.12%. The test set has 821 samples from 18 students (62 questions), with ground-truth grades but no verified recognition, and a student solution accuracy of 71.86%.

Methodology in Plain English

The authors collected homework from a Spring 2025 undergraduate circuit analysis course taught from a standard textbook (Svoboda and Dorf, 2013), removed student names and university IDs, and split the data into an observation set (with expert-proofread transcriptions) and a test set (grades only).

For recognition, they prompted each MLLM to describe a handwritten solution in Markdown, transcribing equations and text verbatim and describing diagrams in natural language. Experts then corrected the recognition output by hand, producing ground-truth transcriptions. To detect errors automatically, they used an LLM judge (Gemini-2.5-Pro) that compares a model's recognition against the verified transcription and lists every discrepant item — a sentence or an equation. They validated this judge against human annotations, then categorized each error into one of four types.

For grading, an LLM grader (GPT-5.1 in all settings) received the model-recognized text, the problem description, a reference solution, and a problem-specific rubric with five error dimensions: Equation (E), Method (M), Unit (U), Calculation (C), and Incomplete Solution (NC). Grading was scored at three levels of strictness: binary (any mistake detected), type (correct error types flagged), and point (exact match of types and point deductions). They introduced the Error Impact Rate to measure what fraction of recognition errors actually caused a grading discrepancy.

Finally, they summarized error patterns from the observation set, used them to flag likely recognition errors in the unseen test set with a confidence label ("high" or "low"), routed low-confidence cases to a teaching assistant, and regraded the rest with an LLM.

Why This Matters

This work shows that a model can appear to grade well while quietly misreading student work, and that the masking effect grows weaker as grading becomes more detailed. That matters for research because it argues against evaluating handwriting understanding purely through downstream task success, and it provides a benchmark with expert-verified transcriptions to support the alternative.

Real-world applications:

• Automated grading of handwritten assignments in university STEM courses, with calibrated human oversight.
• Step-by-step feedback generation, where finer rubric granularity makes hidden recognition errors more consequential.
• Structured downstream conversions, such as circuit-to-netlist transformation, where the paper notes diagrammatic recognition errors that are harmless for simple grading could be catastrophic.
• Handwriting recognition correction pipelines, using identified error patterns to preemptively detect and suppress recognition failures before they reach graders.

Industry relevance: vendors of AI grading and educational assessment platforms can use the taxonomy, metrics, and human-in-the-loop design to set realistic reliability expectations and to decide which fraction of submissions genuinely needs human review — the paper reports expert-comparable grading with human intervention on fewer than 5% of solutions.

Future Directions

• Extend the benchmark beyond circuit diagrams to other diagram modalities, such as complex geometric graphs, which the authors identify as underrepresented.
• Evaluate error propagation into downstream tasks other than auto-grading — visual question answering is named as an unexplored alternative that may show different sensitivity patterns.
• Broaden subject coverage, since the dataset is grounded in a single course, even though it draws on calculus, differential equations, linear algebra, and physics.
• Address subjectivity in grading rubrics and reference grades, which were produced by a small number of doctoral experts and may carry subtle biases.

Target Audience

Researchers working on multimodal LLM evaluation and document/handwriting understanding; AI-in-education researchers and developers building auto-grading or feedback systems; instructors and teaching assistants in university STEM courses considering automated assessment; and benchmarking or dataset curators interested in expert-verified, education-domain multimodal data.

Authors’ abstract

Multimodal Large Language Models (MLLMs) hold significant promise for revolutionizing traditional education and reducing teachers' workload. However, accurately interpreting unconstrained STEM student handwritten solutions with intertwined mathematical formulas, diagrams, and textual reasoning poses a significant challenge due to the lack of authentic and domain-specific benchmarks. Additionally, current evaluation paradigms predominantly rely on the outcomes of downstream tasks (e.g., auto-grading), which often probe only a subset of the recognized content, thereby failing to capture the MLLMs' understanding of complex handwritten logic as a whole. To bridge this gap, we release EDU-CIRCUIT-HW, a dataset consisting of 1,300+ authentic student handwritten solutions from a university-level STEM course. Utilizing the expert-verified verbatim transcriptions and grading reports of student solutions, we simultaneously evaluate various MLLMs' upstream recognition fidelity and downstream auto-grading performance. Our evaluation uncovers an astonishing scale of latent failures within MLLM-recognized student handwritten content, highlighting the models' insufficient reliability for auto-grading and other understanding-oriented applications in high-stakes educational settings. As a potential solution, we present a case study demonstrating that leveraging identified error patterns to preemptively detect and correct recognition errors, while requiring only minimal human intervention (e.g., routing 3.3% of assignments to human graders and the remainder to the GPT-5.1 grader), can effectively enhance the robustness of the deployed AI-enabled grading system. Code and dataset are available in this GitHub repo: https://gt-learning-innovation.github.io/CIRCUIT_EDU_HW_ACL.