SketchJudge: A Diagnostic Benchmark for Grading Hand-drawn Diagrams with Multimodal Large Language Models

AI summary

Overview

Research area: Multimodal large language model (MLLM) evaluation, with a focus on visual grading of hand-drawn STEM diagrams.

Technical level: Intermediate — readers benefit from familiarity with multimodal benchmarks, classification metrics (accuracy, FNR, FPR, F1), and prompt-engineering concepts.

Scope: The paper introduces SketchJudge, a diagnostic benchmark of 1,015 hand-drawn student diagram answers across geometry, physics, charts, and flowcharts, and uses it to evaluate 16 MLLMs as graders rather than as problem solvers.

What This Paper Is About

Most multimodal benchmarks assume clean, standardized images and cast models as solvers whose answers are matched against ground truth. This paper argues that real educational settings demand a different role — a grader who must parse messy freehand sketches, judge correctness, and diagnose specific conceptual errors. SketchJudge is built to test that grading capability, pairing natural-language questions, clean reference diagrams, and noisy student sketches annotated with correctness labels and fine-grained error types.

Key Contributions

1. The authors introduce what they describe as the first diagnostic benchmark for grading hand-drawn diagrammatic answers in educational contexts, spanning four task types (geometry, physics, charts, flowcharts).
2. A fine-grained error taxonomy is proposed — 23 domain-specific error types in total — enabling structured diagnosis of model mistakes beyond simple right/wrong judgments.
3. Systematic zero-shot experiments across 16 state-of-the-art MLLMs (open-source and proprietary) are presented, revealing persistent limitations in diagram-level reasoning.
4. The benchmark supports two evaluation settings — a reference-based setting (WithRef) that supplies the clean reference diagram and a reference-free setting (NoRef) that withholds it — plus three prompting styles (Baseline, Rubric, Chain-of-Thought).

Main Findings

• Humans outperform all models. Human raters achieve an overall accuracy of 83.33% and an overall example-based F1 (eb F1) of 66.42. The human baseline was measured on a stratified subset of 160 student answers (40 per domain, balanced over correct/incorrect), graded by three non-expert raters following the same instructions as models.
• GPT-5 leads the leaderboard but still trails humans. GPT-5 reaches 78.42% overall accuracy and 61.13 eb F1, followed by Gemini-2.5-Flash at 77.74% / 58.30 and o3 at 74.58% / 55.91. Claude-3.7-Sonnet reaches 73.00% / 57.11.
• A clear open-source vs. closed-source gap. Multiple closed-source models exceed 73% overall accuracy, whereas most open-source models fall in the 53%–69% range with larger cross-domain variance, even at larger scales.
• The strongest model varies by domain. Grading performance is domain-sensitive; per-domain results show the best model differing across physics, geometry, charts, and flowcharts.
• Grading bias varies sharply. In the FNR–FPR analysis, several smaller open-source models are strongly over-strict with very high FNR (e.g., Qwen2.5-VL-3B-Instruct), while others are over-lenient with low FNR but much higher FPR (e.g., Gemma-3-4B-it). Stronger closed-source models such as GPT-5 and Gemini-2.5-Flash lie closer to the balanced region.
• Reference diagrams mainly help correctness, not error typing. Providing a reference improves accuracy for most models, but its effect on eb F1 is less uniform — suggesting references aid correctness verification more than fine-grained error attribution.
• Prompting does not reliably help. Rubric prompting yields only modest, model-dependent changes (it slightly improves accuracy and eb F1 for GLM-4.6V and Qwen2.5-VL-72B-Instruct but decreases both for Gemini-2.5-Flash). Chain-of-Thought prompting consistently hurts all three tested models; for Gemini-2.5-Flash it reduces accuracy from 77.74% to 76.16% while increasing FNR from 0.263 to 0.351.
• Local cues are easier than global structure. Per-error-type recall shows connection-related errors in physics and chart-type mistakes in charts reach the highest recall, while categories depending on global structure or latent intent (concept-target and scaling errors in physics, strategy-level mistakes in geometry, presentation or omission errors in charts) remain difficult.

Methodology in Plain English

The researchers assembled instances of the form {Q, R, A}: a natural-language question (Q), a clean reference diagram (R), and one or more hand-drawn student answers (A). The dataset totals 300 problems and 1,015 hand-drawn answers, averaging 3.38 answers per problem, with 1,462 total images (147 / 300 / 1,015 for Q / R / A). Questions containing images make up 147 of 300 (49%), and all 300 problems have reference diagrams. Of the answers, 470 (46%) are correct and 545 (54%) incorrect.

Geometry and physics questions came primarily from past exams and textbooks; chart and flowchart tasks included a small synthetic subset generated with GPT-4o via multi-turn prompting, later manually verified. Reference diagrams were obtained by retrieving official solutions, programmatically rendering charts from underlying data, or manually authoring and validating them; three independent raters verified all problem statements and references. Student answers — 1–9 per question, contributed by 18 individuals — were digitized as images while preserving realistic noise such as construction traces, uneven strokes, and minor skew.

Annotation proceeded in two stages. In Stage 1, three annotators independently wrote error descriptions for all incorrect sketches; 50 representative descriptions per domain were sampled, and GPT-4o was prompted to cluster them into draft categories. In Stage 2, subject-matter experts (including frontline physics and math teachers) refined the taxonomies, after which GPT-4o mapped each written description to error types with the sketch withheld, and human annotators verified and corrected the labels. Correct responses were labeled error-free.

Evaluation was strictly zero-shot with deterministic decoding (temperature = 0). Correctness was measured as binary classification using accuracy, FNR, and FPR; error-type recognition used example-based F1 computed only on instances where both the gold annotation and the model prediction indicated incorrectness. Domain-specific taxonomies were unified into a namespaced global label set.

Why This Matters

Impact on research. The paper targets a gap its comparison table makes explicit: DocVQA, ChartQA, MathVista, JudgeBench, and HallusionBench do not jointly cover freehand inputs, student-style responses, grading/judging, fine-grained diagnostic types, and provided references — SketchJudge claims all five. The results indicate that current vision-language alignment is fragile in symbolic and noisy contexts, positioning diagram grading as a distinct research frontier rather than an extension of question answering.

Real-world applications:

• Scalable formative assessment of student work in STEM classrooms.
• Personalized feedback that identifies which specific conceptual or procedural error a student made.
• Human–AI collaborative grading workflows where models flag candidate errors for teacher review.
• Tool-augmented or agentic systems that construct, generate, or verify diagrams as part of reasoning.

Industry relevance. The findings matter to educational technology and automated assessment vendors: grading bias profiles (over-strict vs. over-lenient) and the failure of Chain-of-Thought prompting are directly actionable signals about where model-assisted grading can and cannot be trusted. The paper explicitly cautions that models evaluated on SketchJudge should be viewed as research tools rather than replacements for human educators, and that high-stakes use requires validation and oversight.

Future Directions

• Tool-augmented and agentic reasoning. The authors propose SketchJudge as a testbed where models must plan, externalize intermediate steps, and interact with tools to construct, generate, or verify diagrams.
• Better grounding of error diagnoses. Since reference availability mainly aids correctness verification while fine-grained error typing remains weak, future work is needed on structured reasoning and supervision strategies that ground diagnoses in diagram semantics.
• Robustness to stylistic variation. Consistent attribution of conceptual errors and sensitivity to structural and topological distinctions remain unsolved competencies that the paper identifies as core grading skills.
• Expanding and updating the benchmark. The authors note that the labor-intensive annotation pipeline constrains dataset expansion and taxonomy updates, and that extending grading models to broader real-world settings will require further attention to robustness, calibration, and user-facing reliability.

Target Audience

Researchers and practitioners in multimodal LLM evaluation, computer vision, and AI in education — particularly those working on visual reasoning benchmarks, automated assessment, or educational feedback systems. The paper is also useful for dataset builders interested in error taxonomies and human-verified annotation protocols, and for educators and edtech developers evaluating whether current MLLMs are ready for grading hand-drawn student work.

Authors’ abstract

While Multimodal Large Language Models (MLLMs) have achieved remarkable progress in visual understanding, they often struggle when faced with the unstructured and ambiguous nature of human-generated sketches. This limitation is particularly pronounced in the underexplored task of visual grading, where models should not only solve a problem but also diagnose errors in hand-drawn diagrams. Such diagnostic capabilities depend on complex structural, semantic, and metacognitive reasoning. To bridge this gap, we introduce SketchJudge, a novel benchmark tailored for evaluating MLLMs as graders of hand-drawn STEM diagrams. SketchJudge encompasses 1,015 hand-drawn student responses across four domains: geometry, physics, charts, and flowcharts, featuring diverse stylistic variations and distinct error types. Evaluations on SketchJudge demonstrate that even advanced MLLMs lag significantly behind humans, validating the benchmark's effectiveness in exposing the fragility of current vision-language alignment in symbolic and noisy contexts. All data, code, and evaluation scripts are publicly available at https://github.com/yuhangsu82/SketchJudge.