Bridging Online and Offline Handwriting via Differentiable Physical Rendering

AI summary

Overview

• Research area: Computer vision / handwriting generation, spanning online trajectory modeling, offline image synthesis, differentiable rendering, physical brush modeling, and robotic calligraphy.
• Technical level: Advanced. The paper assumes familiarity with transformer sequence models, diffusion models, differentiable rendering, and classical computer-graphics brush simulation.
• Scope: The paper proposes a compact six-parameter physical brush model and a differentiable renderer that together unify online (trajectory) and offline (image) handwriting generation in a single framework that can also be executed by a robot arm.

What This Paper Is About

Handwriting research has split into two separate camps: online methods that predict the time-ordered pen trajectory of writing but miss fine visual texture, and offline methods that synthesize realistic handwriting images but discard stroke order and motion. The paper builds a bridge between the two by defining a differentiable physical brush model that turns stroke motion into pixels, then wrapping it in a four-module framework that generates strokes from text and style images, estimates brush parameters from style references, renders a handwriting image, and optionally refines it with a pretrained diffusion model.

Key Contributions

1. A compact physical brush model with six core parameters (w_base, k_spread, ρ_ink, σ_sharp, p_min, p_max) that explicitly connects stroke kinematics to visual text appearance.
2. A differentiable rendering module that converts sequential stroke trajectories into pixel-level images, enabling gradient backpropagation across the motion and appearance domains.
3. A unified handwriting generation framework that jointly models online trajectories and offline images via differentiable physical rendering, combining a text-to-stroke generator, a brush parameter observer, the renderer, and a zero-shot image refiner.
4. Paired data synthesis from existing online datasets. Because no dataset provides both raster images and aligned stroke trajectories, the renderer synthesizes offline images from online stroke datasets, reducing dependence on paired data.
5. Experimental and robotic validation, including real-world calligraphy demonstrations with a UFACTORY Lite 6 robot arm.

Main Findings

• Online trajectory accuracy: On DTW error, the proposed method reports 0.2936 (multi-letter) and 0.3462 (single-letter), versus SDT at 0.8155 (multi-letter) and 0.6056 (single-letter), outperforming the state-of-the-art single-character model in both scenarios.
• Renderer fidelity on the paired dataset: "Our Renderer" records the lowest FID (11.81), BFID (11.03), HWD (1.0833), and LPIPS (0.001) among the compared entries, compared with VATr++ (74.28 / 33.44 / 2.5990 / 0.622), DiffPen (89.06 / 118.04 / 2.0030 / 0.602), One-DM (52.26 / 19.88 / 2.3650 / 0.539), and Emuru (89.97 / 60.88 / 2.6220 / 0.583) on FID, BFID, HWD, and LPIPS respectively. On the same table, One-DM reports the best CER (0.229).
• Offline model generalization gap: Standard offline handwriting models struggle when conditioned on the paper's paired dataset — attributed to over-reliance on pixel-level distributions and an inability to capture the underlying motion.
• Prerender-guided diffusion improves realism: Attaching the renderer to two diffusion backbones improves FID and BFID on both datasets. On IAM Words, One-DM+Ours reaches FID 25.89 and BFID 8.15 versus One-DM's 29.00 and 16.35; on CVL Words, One-DM+Ours reaches FID 14.45 and BFID 10.47 versus One-DM's 19.45 and 15.18. DiffPen+Ours improves on DiffPen on FID and BFID on both datasets, but HWD degrades (e.g., 2.3116 vs 1.6275 on IAM Words).
• Backbone-dependent trade-offs: Metrics such as HWD and CER show backbone- and dataset-dependent behavior, and the noise-injection start step must be tuned per setting: 50 for DiffPen+Ours on both IAM and CVL, 900 for One-DM+Ours on IAM, and 200 for One-DM+Ours on CVL.
• One-DM is the strongest standalone baseline: It achieves the best FID, BFID, and CER on both IAM and CVL; DiffPen yields the lowest HWD on both datasets; Emuru performs noticeably worse across most metrics, including CER of 3.0583 on IAM and 3.5744 on CVL.
• Style is preserved by mixing physics with diffusion: The renderer preserves stroke thickness and ink density, while diffusion adds texture and realism. The paper states our framework acts as a complementary guidance module whose effect depends on the backbone.
• Perceptual user study: A ranking-based user study for perceptual style similarity gives One-DM+Ours the best average rank of 2.840, improving over One-DM at 3.406.
• Robotic executability: Predicted trajectories ran on a UFACTORY Lite 6 robot arm with a fixed writing tool (0-DoF rotation), using the 2D stroke coordinates as the (x, y) path and the pressure proxy mapped to the vertical z position, with no additional optimization or post-processing.

Methodology in Plain English

The authors start from classical computer-graphics brush simulators, but instead of building a full physical simulator they distill the visually important behavior into six numbers: base stroke width, how much width grows with pressure, ink density, edge softness, and the lower and upper bounds of a pressure range. Since online datasets have no real pressure sensor data, the system infers pressure from writing speed — faster strokes are treated as lighter — then smooths that signal over time.

Rendering works by walking along the stroke, computing each pixel's distance to the stroke segment, and turning that distance into a soft footprint. The footprint width grows with the pressure proxy, and the opacity of the deposited ink grows with ink density and pressure. Rather than adding strokes up (which would darken self-overlapping curves), the renderer takes the maximum coverage per pixel and subtracts it from white. Every step of this is differentiable, so gradients flow from image errors back to trajectory and brush parameters.

Because no dataset pairs handwriting images with their stroke sequences, the authors use the renderer to manufacture one: take existing online trajectory datasets, randomly sample brush parameters from predefined distributions, render an image, then composite that ink over random real-world background textures such as notebook paper, cardboard, and wood boards. The result is a synthetic paired dataset of text, strokes, images, and the brush parameters used to make them.

Four modules then form the pipeline. A transformer-based stroke generator predicts the trajectory from the target text plus style images; it was extended from single-character to word level by replacing the single content token with a sequence of character embeddings and adding a cross-attention layer in the decoder, outputting a 20-component 2D Gaussian mixture for pen position plus pen-state logits. A brush parameter observer uses DINOv3 features with a learnable style token and a transformer aggregator that mixes within-image and across-image attention to regress the six brush parameters, trained with both parameter-space MSE and rendering-consistency MSE. The differentiable renderer turns strokes and parameters into an image. Finally, a zero-shot image enhancer takes an off-the-shelf handwriting diffusion model, leaves it completely unchanged, encodes the rendered image into its latent space, and starts the denoising process from a noised version of that latent — preserving the predicted layout while letting the model add paper texture and scanner-like artifacts.

Training used IAM-OnDB and CASIA-OLHWDB rendered into 155,840 word-level samples, split by writer into 303 training and 78 testing writers. Each trajectory averages about 60 points and is truncated at a maximum of 250 points; the maximum word length is 9 characters with an average of 3.5, and the dataset contains 25,912 unique word types. Offline evaluation used the test sets of IAM (161 writers) and CVL (283 writers), discarding words longer than 9 characters and resizing all rendered images to a fixed height of 64 pixels. Implementation details, evaluation metrics, and further experimental settings are deferred to the Supplementary Material, which is not included in the provided content.

Why This Matters

The paper argues that handwriting is a single physical act, but research has treated the motion (trajectory) and the trace (image) as two unrelated generative tasks. That split creates a causal gap: image models can produce structurally inconsistent glyphs because they are not grounded in the writing process, while trajectory-only models cannot produce realistic visual appearance. Connecting them lets one system output both a realistic image and a physically executable motion plan, and it also lets researchers generate images from trajectory-only datasets.

Real-world applications named in the paper:

• Personalized font design and artistic expression, including customized fonts for individuals.
• Training-data synthesis for OCR/text recognition, where realistic handwriting images are needed at scale.
• Biometric authentication, where handwriting style is a signature of identity.
• Robotic calligraphy and physical AI, where a manipulator holding a brush or pen executes a generated trajectory.
• Assistive technologies for people with physical impairments who cannot write by hand.

Industry relevance: The framework is positioned for physical AI systems, digital note-taking, and graphics pipelines. The controllable, interpretable parameter estimates can be used to select an appropriate physical writing tool — the paper selects among pencil, sharpie, and marker pen — and the predicted trajectories execute directly as Cartesian motion without extra optimization, which matters for deploying generative models on real hardware.

Future Directions

• Calibrating surrogate parameters to physical units. The paper notes the brush parameters are renderer-level surrogates, not quantities calibrated to real-world physical units, so transferring predictions to physical writing tools may require additional calibration.
• Extending from words to long-form text. The current architecture is optimized for word-level generation; coherent long-form sentence generation is stated as an important direction for future work.
• Making the noise-injection step adaptive. HWD and CER trade-offs vary by backbone and dataset, and the start step had to be hand-tuned (50, 900, 200) per setting, suggesting a need for automatic selection.
• Reducing backbone dependence. The framework acts as complementary guidance whose benefit depends on the diffusion backbone, leaving open the question of how to make the improvement consistent across models.

Target Audience

Researchers and practitioners in computer vision, handwriting generation, and human-computer interaction; graphics researchers interested in differentiable physically-based stroke rendering; robotics engineers working on calligraphy or physically grounded generative manipulation; and machine-learning engineers who need synthetic handwriting data for OCR or font design. Readers should be comfortable with sequence models, diffusion sampling, and rendering math, making this most valuable to those with intermediate-to-advanced machine learning background.

Authors’ abstract

Realistic handwritten text generation plays an important role in numerous applications, such as font design, biometric authentication, and robotic calligraphy. Existing methods are typically divided into two independent paradigms: online approaches that estimate handwriting trajectories and offline approaches that synthesize realistic handwriting images. While online models capture structural and temporal dynamics, they often lack fine-grained textures, whereas offline models reproduce realistic appearance but discard stroke order. However, unifying online and offline models remains challenging due to (1) the lack of an explicit physical model linking stroke kinematics to pixel-level appearance and (2) the absence of paired trajectory-image datasets. Moreover, enabling end-to-end learning requires a differentiable rendering process across motion and appearance domains. To address these challenges, we propose a compact physical brush model that bridges stroke dynamics and visual appearance, together with a differentiable rendering module that converts stroke trajectories into stylized images. By integrating these components, we propose a unified online-offline handwriting generation framework via differentiable brush rendering. The proposed framework consists of four core modules: 1) a text-to-stroke generator that predicts the target stroke conditioned on the given text and style image, 2) a brush parameter observer that extracts brush model parameters from style references, 3) a differentiable brush renderer that maps a stroke sequence and physical brush parameters into a handwritten image, and 4) a zero-shot image refiner that refines rendered images via diffusion models. Extensive experiments and real-world robotic calligraphy demonstrations validate our approach, achieving both structural and visual fidelity.