GryphOne: Symbol-Aware Masked Diffusion for Structural Refinement in Offline Handwritten Mathematical Expression Recognition

AI summary

Overview

Research area: Computer Vision / document understanding — specifically offline handwritten mathematical expression recognition (HMER), the task of converting an image of handwritten mathematics into LaTeX markup.

Technical level: Advanced. The paper assumes familiarity with transformer encoder–decoder architectures, autoregressive vs. non-autoregressive decoding, discrete/masked diffusion models, and LaTeX syntax structure.

Scope (one sentence): The authors replace sequential autoregressive LaTeX generation with a symbol-aware masked diffusion process that iteratively unmasks and remasks a fixed-length symbol sequence, evaluated on MathWriting and CROHME 2014–2023.

What This Paper Is About

Handwritten math recognition is hard because of two ambiguities: symbols that look alike (a cursive "z" resembling "2") and layouts that change meaning (a fraction misread as a plain product). Existing autoregressive models generate LaTeX tokens left to right, so an early mistake propagates and cannot be revised, while non-autoregressive models make a single one-shot prediction and cannot refine wrong symbols or structures.

GryphOne's goal is to unify structural consistency and robustness to ambiguity by reformulating HMER as an iterative symbolic refinement problem using a masked diffusion model — starting from a fully masked sequence and progressively refining all symbols in parallel. The authors state that, to the best of their knowledge, this is the first application of masked diffusion to HMER.

Key Contributions

1. Masked diffusion formulation of HMER. HMER is cast as a discrete masked diffusion process (with explicit forward corruption and reverse refinement algorithms), enabling iterative symbolic refinement without left-to-right exposure bias.
2. Symbol-aware tokenization (SAT). LaTeX tokens are decomposed into a sequence of visible symbols and a sequence of invisible modifiers (carets, underscores, braces) aligned one-to-one to symbols, preserving syntax while keeping each structural edit local — the paper notes that removing a superscript in x_1^{y_2} to get x_1y_2 otherwise eliminates five tokens including a caret and four bracket tokens.
3. Random-masking mutual learning (RMML). Two independently masked views of the same ground-truth sequence are decoded by a shared-parameter decoder, trained with two cross-entropy losses plus two Kullback–Leibler divergence terms between the two predictions, with no temperature scaling, to improve invariance to masking patterns.
4. Empirical validation. Evaluation on MathWriting and held-out CROHME (2014, 2016, 2019, 2023) shows the best recognition performance and improved structural stability, with an explicit accuracy–latency trade-off controlled by the number of diffusion steps.

Main Findings

• MathWriting test results: GryphOne-50 reaches 5.51% CER and 59.9% EM (ExpRate) on the test split, with 78.1% of expressions having at most one token error. On the validation split it reaches 4.63% CER, 71.0% EM, and 85.0% for the ≤1 metric.
• Gains over the strongest reimplemented baseline: Compared with ICAL on the validation split, EM improves from 63.4 to 71.0 and the ≤1 metric from 74.5 to 85.0. On the test split, the ≤1 metric improves by 9.9 points.
• Speed: GryphOne-10 runs at 73.7 FPS, faster than all autoregressive baselines in Table 1, while preserving over 98% of the recognition performance of T=50. GryphOne-50 runs at 21.2 FPS. Mask-predict (MP) decoding outperforms autoregressive baselines but consistently underperforms GryphOne-10.
• Cross-dataset generalization (CROHME, ExpRate EM): GryphOne-50 achieves 65.2 (2014), 61.4 (2016), 61.8 (2019), and 61.2 (2023), with ≤1 scores of 75.9, 74.3, 75.7, and 74.8 respectively. NAMER (literature-only scores, marked with an asterisk) reports 60.5, 60.2, and 61.7 for 2014–2019 and no value for 2023.
• Ablation: Combining SAT and RMML gives the lowest CER at every tested diffusion depth (T = 2, 5, 10, 50): 5.01, 4.78, 4.70, 4.63, versus 5.42, 4.97, 4.84, 4.75 with neither component. For SER, RMML yields greater improvements at larger T, while SAT provides strong early gains at small T; the two are described as complementary. RMML alone gives the lowest SER values in the table (e.g., 0.59 at T=50), while SAT+RMML gives 0.84 at T=50.
• Refinement dynamics: Step-wise CER, SER, and EM all decrease/improve steadily with convergence around steps 40–50; improvements become marginal after approximately 30 steps, suggesting truncated diffusion schedules can retain most accuracy at lower inference cost.
• Accuracy–latency control: The refinement depth T acts as a length-independent control knob. At T=1 the model runs at 164 FPS (valid) / 161 FPS (test) with 5.23 / 6.02 CER; at T=50 it runs at 21.2 / 21.1 FPS with 4.63 / 5.51 CER.
• Error recovery: When fraction, square root, integral, and summation operators are removed and reconstruction is initialized mid-schedule, recovery difficulty shows a single extremum at roughly t ≈ 20. CER is higher with SAT in this specific analysis, which the authors attribute to braces contributing to CER and being removed early; SER consistently improves with SAT.
• Diversity: Across 10 independent decoding runs per input, output diversity remains low for most expressions, and both SAT and RMML further reduce variability. A qualitative case study (an ambiguous fraction numerator) yields x=5+\frac{14}{x}, x=5+\frac{17}{x}, and x=5+\frac{4}{x}, which the authors interpret as reflecting genuine visual ambiguity rather than arbitrary noise.
• Abstract-level claim: GryphOne is reported to outperform all reimplemented models in the matched setting as well as the commercial HMER system; Table 1 lists an OCR baseline from the MathWriting paper (6.50 CER / 64.0 EM on valid; 7.17 CER / 53.0 EM on test) with no FPS reported.

Methodology in Plain English

The model has two parts: a vision transformer encoder (DINO, patch size 8, hidden dimension 384) that reads a 224×224 rasterized image of the handwriting, and a masked-diffusion decoder made of five transformer blocks that produces the LaTeX-equivalent symbol sequence.

Instead of writing the answer one token at a time, training corrupts a ground-truth symbol sequence by randomly replacing each symbol with a MASK token with probability t/T, where the diffusion time t is sampled uniformly from (0, T). The decoder is trained with a cross-entropy loss over all positions — masked and unmasked — to recover the original symbols. At inference, decoding starts from a sequence of all MASK tokens of fixed length, and for T steps the decoder predicts all positions in parallel, then randomly remasks a subset of tokens according to the same corruption schedule. This unmask-and-remask loop is what lets the model revise earlier decisions as global context improves.

SAT is the representation trick that makes this work on LaTeX. LaTeX structural markers (carets, underscores, braces) are pulled out of the main sequence and aligned to the symbols they modify — carets, underscores, and open braces go to the leftmost symbol to their right; close braces go to the rightmost symbol to their left; modifiers on the same symbol are merged. Each symbol embedding is then summed with its aligned modifier embedding, so each token corresponds one-to-one with a handwritten glyph and a local structural edit only touches a single token.

RMML adds a training-time regularizer: two differently masked versions of the same ground truth are decoded, each with its own cross-entropy loss, plus symmetric KL divergences between the two prediction distributions, so the model becomes less sensitive to any particular masking pattern. It is used only during training and adds no inference cost.

Implementation details: images are 224×224 with 1-pixel stroke width; sequences are padded to a maximum length of 150; attention uses 8 heads with 30% dropout; T=50 by default; 60 epochs, batch size 32, AdamW with learning rate 10⁻⁴ and weight decay 10⁻³; FPS measured on a single NVIDIA V100; results averaged over 10 inference runs; no data augmentation. Training data is MathWriting (230k handwritten samples plus 400k synthetic samples), with CROHME used as a held-out benchmark because its limited size is insufficient for diffusion training.

Why This Matters

Impact on research: The work reframes HMER from sequential generation to iterative refinement, showing that discrete diffusion can handle outputs with hierarchical, interdependent syntax rather than flat token streams. It also proposes a representation-level fix (SAT) for the mismatch between diffusion's locality assumption and LaTeX's non-local structural edits, and it introduces an explicit accuracy–latency control knob that is independent of expression length — something autoregressive latency, which scales linearly with length, cannot offer. The held-out CROHME 2014–2023 results suggest the refinement approach generalizes beyond the training distribution.

Real-world applications:

• Digitizing handwritten STEM homework, exams, and lecture notes into editable LaTeX for grading or archival.
• Assistive input tools that let users write mathematics by hand and receive typeset output for papers, slides, or code.
• Conversion of scanned historical or archival mathematical manuscripts into searchable, machine-readable markup.
• Front ends for computer algebra systems, where a handwritten formula is recognized and passed directly to a solver.

Industry relevance: The paper reports performance competitive with a commercial HMER system in Table 1 and reports throughput (up to 164 FPS at T=1, 73.7 FPS at T=10 on a V100), which matters for deployment on document-processing pipelines. The T parameter gives deployers a direct dial between accuracy and compute cost without retraining, and the authors note that training requires a large, diverse dataset such as MathWriting, which is a practical constraint for smaller organizations.

Future Directions

• Latency: The iterative refinement requires multiple decoder passes; the authors list more efficient diffusion algorithms as future work, noting that the number of steps is often much smaller than sequence length in practice.
• Data efficiency: Diffusion training is described as data-hungry and unable to reach competitive performance when trained solely on small-scale datasets such as CROHME; hybrid training schemes are suggested as a remedy.
• Input flexibility: The current ViT operates on fixed-resolution rasterized images, which the authors say may limit handling of extreme aspect ratios or very long expressions; adaptive-resolution encoders are proposed as a direction.
• Open questions raised by the results: Whether structural recovery can be made more robust at intermediate corruption levels (the t ≈ 20 extremum), why CER in the operator-removal analysis is higher with SAT, and whether truncated schedules can be tuned further, given that improvements become marginal after roughly 30 steps.

Target Audience

Researchers and practitioners working on document understanding, offline handwriting recognition, and mathematical expression recognition; engineers building optical character recognition or handwriting-to-LaTeX systems; and machine learning researchers interested in discrete diffusion and masked generative models applied to structured, syntax-heavy outputs. Readers need a working knowledge of transformer architectures and sequence decoding to follow the method, but the high-level contribution — replacing sequential decoding with iterative symbolic refinement — is accessible without it.

Authors’ abstract

Handwritten mathematical expression recognition (HMER) requires reasoning over diverse symbols and structures, yet autoregressive models struggle with exposure bias and syntax inconsistency. We present GryphOne, a discrete diffusion framework which reformulates HMER as iterative symbolic refinement instead of sequential generation. GryphOne progressively refines symbols and relations, removing autoregression and improving consistency. Symbol-aware tokenization and random-masking mutual learning further enhance robustness to handwriting diversity. On the MathWriting benchmark, GryphOne achieves 5.51% CER and 59.9% EM (ExpRate), outperforming all reimplemented models in the matched setting as well as the commercial HMER system. Held-out evaluation on CROHME 2014-2023 further shows strong cross-dataset generalization.