VEHME: A Vision-Language Model For Evaluating Handwritten Mathematics Expressions

AI summary

Overview

Research area: Automated assessment of handwritten mathematics using vision-language models (VLMs), intersecting educational technology, multimodal reasoning, and reinforcement learning.

Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning, GRPO-style reinforcement learning, reward shaping, and modern VLM architectures.

Scope: The paper introduces VEHME, an end-to-end training pipeline that grades open-form handwritten K–12 mathematics solutions by detecting and localizing errors.

What This Paper Is About

Grading handwritten mathematics is labor-intensive for teachers and hard to automate because student work is unstructured, visually noisy, and varies widely in layout and handwriting. Existing automated systems rely on rigid templates, OCR pipelines, or typed text, and they break down on real handwritten multi-expression solutions. The paper's goal is to build a VLM that takes a question, a reference answer, and a student's handwritten answer image, then predicts correctness and pinpoints where any error occurs, with interpretable reasoning traces.

Key Contributions

1. VEHME framework. The authors present a novel framework for evaluating handwritten mathematical expressions, which they state is the first end-to-end training pipeline for grading handwritten mathematical expressions.

2. Dual-phase training regime. A pipeline combining supervised fine-tuning on synthesized reasoning data distilled from QwQ-32B, followed by reinforcement learning with Group Relative Policy Optimization (GRPO) using a composite reward for correctness, reasoning quality, and error localization.

3. Data synthesis pipeline. To address the scarcity of training data that pairs handwritten expressions with detailed error annotations, the authors construct a reasoning-augmented dataset pairing handwritten expressions with error detection and localization traces.

4. Expression-Aware Visual Prompting Module (EVPM). To handle spatial layout and visual noise, the authors add a module that predicts oriented bounding boxes around individual mathematical expressions, trained on a synthesized dataset of complex, heterogeneous, multi-expression handwritten math.

5. Small models beating larger ones. The training pipeline lets small open-source VLMs surpass larger open-source counterparts and approach state-of-the-art proprietary models.

Main Findings

• AIHub results. VEHME-Qwen2.5-VL-7B achieves 73.01% ED accuracy (49.22% F1) and 61.13% EL accuracy (58.18% F1), substantially outperforming Qwen2.5-VL-7B-Instruct (46.68%/31.48% ED; 38.00%/33.75% EL), Pixtral-12B (52.67%/31.69% ED; 32.20%/38.35% EL), Phi-4-multimodal-instruct, and Llama-3.2-11B-Vision-Instruct.

• FERMAT results. VEHME leads open-source models with 62.61% ED accuracy (29.81% F1) and 31.90% EL accuracy (44.36% F1). Pixtral-12B shows an unusual split with higher ED accuracy (41.39%) but very low EL performance (10.53% Acc/3.52% F1).

• Closed-source comparison. Gemini-2.0-Flash remains the top overall performer (75.40%/50.30% ED and 67.40%/65.73% EL on AIHub; 75.22%/43.13% ED and 64.24%/75.13% EL on FERMAT), followed by GPT-4o and GPT-4o-mini. VEHME's Qwen2.5-VL-7B backbone contains only 7 billion parameters versus the much larger proprietary systems.

• Component ablation (AIHub). Removing EVPM causes a modest ED drop (73.01% to 71.98%) and a larger EL F1 decline (58.18% to 56.59%). Removing SFT degrades both tasks (ED 63.24%/41.47% F1; EL 52.67%/33.41% F1). Removing RL causes the largest collapse, particularly EL F1 (58.18% to 29.59%).

• Heavy-rotation subset. On the 100 most heavily rotated AIHub samples (mean 21.81 degrees), removing EVPM causes a substantial drop (ED 75% to 62%, EL 66% to 62%). VEHME reaches 75% Acc/50.24% F1 ED and 66% Acc/65.32% F1 EL, surpassing GPT-4o (63%/62.36% EL) and Gemini-2.5-Flash-Preview (59%/58.80% EL) on localization.

• Resolution ablation. Increasing input resolution from 224×224 to 448×448 improves both tasks (ED 73.01% to 75.66% Acc; EL 61.13% to 63.87% Acc), while going to 768×768 yields only marginal changes (ED 75.29% Acc; EL 64.58% Acc).

• Dataset skew differences. AIHub was balanced after curation to 81,394 training and 9,062 test samples, whereas FERMAT has an approximately 85:15 incorrect-to-correct ratio and 2,244 manually curated solutions across eight mathematical domains.

Methodology in Plain English

The researchers framed grading as a vision-language problem: given a question image, a reference answer, and a student's handwritten answer image, the model must output a correctness label and, if wrong, a description of where the error is.

They trained in two phases. First, supervised fine-tuning: they asked QwQ-32B (a large math-focused model) to generate structured feedback with a thinking process, a correctness label, and an error localization for each example. When outputs exceeded the token budget, they repaired truncated responses and used grammar-constrained decoding to keep outputs valid. The VLM (Qwen2.5-VL-7B-Instruct) then learned to imitate these structured responses.

Second, reinforcement learning using GRPO. For each input, the model samples a group of completions; each is scored by a composite reward combining correctness match, error-localization correctness (judged by QwQ-32B as an automated judge), a length term, a cosine reward, and a repetition penalty. Advantages are computed by standardizing rewards within the group.

Separately, to help the model handle cluttered handwritten layouts, they built the EVPM: a YOLOv11-based detector trained on synthetic multi-line handwritten expressions that are randomly rotated and placed on blank canvases, producing oriented bounding boxes that serve as visual prompts to the downstream VLM.

Why This Matters

Impact on research. The paper shows that targeted data synthesis, spatial prompting, and RL can close much of the gap between 7B open-source VLMs and much larger proprietary systems on a hard educational grading task. It also extends process-reward-style supervision into the handwritten multimodal domain, where it had not been applied before.

Real-world applications:

• Automatic grading and feedback in digital learning platforms for K–12 mathematics.
• Diagnostic tools that help teachers quickly identify where students make procedural or conceptual errors.
• Step-level feedback systems that generate interpretable error explanations for learners.
• Error analysis at scale across educational datasets and curricula.

Industry relevance. EdTech companies, assessment providers, and platforms serving large class sizes could deploy a lightweight 7B model rather than relying on expensive proprietary APIs, while gaining interpretable reasoning traces and spatial error localization.

Future Directions

• Data accessibility. Because AIHub sharing restrictions prevent public release of original problems and responses outside Korea, the authors identify this as a reproducibility limitation and call for more accessible datasets.
• Robustness. Handling extremely illegible handwriting, low-quality images, and unconventional notation remains an open problem.
• Bias and reward design. QwQ-32B may propagate length-based tendencies or anchoring effects into the evaluation pipeline; addressing deeper preference and anchoring biases is left for future work.
• Training stability. The RL setup is sensitive to reward design and may introduce instability, motivating investigation of improved robustness and more interpretable learning signals.

Target Audience

Researchers and practitioners in multimodal machine learning, educational data mining, and automated assessment, as well as EdTech engineers interested in deploying vision-language models for grading. Readers should be comfortable with reinforcement learning for language models and VLM training pipelines.

Authors’ abstract

Automatically assessing handwritten mathematical solutions is an important problem in educational technology with practical applications, but it remains a significant challenge due to the diverse formats, unstructured layouts, and symbolic complexity of student work. To address this challenge, we introduce VEHME-a Vision-Language Model for Evaluating Handwritten Mathematics Expressions-designed to assess open-form handwritten math responses with high accuracy and interpretable reasoning traces. VEHME integrates a two-phase training pipeline: (i) supervised fine-tuning using structured reasoning data, and (ii) reinforcement learning that aligns model outputs with multi-dimensional grading objectives, including correctness, reasoning depth, and error localization. To enhance spatial understanding, we propose an Expression-Aware Visual Prompting Module, trained on our synthesized multi-line math expressions dataset to robustly guide attention in visually heterogeneous inputs. Evaluated on AIHub and FERMAT datasets, VEHME achieves state-of-the-art performance among open-source models and approaches the accuracy of proprietary systems, demonstrating its potential as a scalable and accessible tool for automated math assessment. Our training and experiment code is publicly available at our GitHub repository.