Render-of-Thought: Rendering Textual Chain-of-Thought as Images for Visual Latent Reasoning

AI summary

Overview

Research area: Natural Language Processing / LLM reasoning efficiency, spanning Chain-of-Thought prompting, latent (implicit) reasoning, and Vision-Language Models.

Technical level: Intermediate — the paper assumes familiarity with Chain-of-Thought prompting, LLM hidden states, and multimodal vision encoders, but the core idea is explained conceptually.

Scope: The paper proposes Render-of-Thought (RoT), a framework that renders textual reasoning steps into images and uses a frozen vision encoder as a "semantic anchor" so an LLM can perform reasoning in a compressed visual latent space instead of generating verbose text.

Authors and venues: Yifan Wang, Shiyu Li, Peiming Li, Xiaochen Yang, Yang Tang, and Zheng Wei, with affiliations at Tencent BAC, the Shenzhen International Graduate School at Tsinghua University, Peking University, and the University of Glasgow. Posted as arXiv:2601.14750v4 [cs.CL]. Code is listed at https://github.com/TencentBAC/RoT.

What This Paper Is About

Chain-of-Thought prompting makes LLMs better at reasoning, but it produces long textual outputs that slow inference and consume memory. A newer line of work compresses reasoning into dense latent vectors, but those vectors are opaque "black boxes" and are usually trained only on final-answer correctness, with no supervision on the intermediate reasoning. Render-of-Thought tackles both problems at once: it converts each textual reasoning step into an image, encodes that image with an existing frozen vision encoder, and trains the LLM to generate hidden states that match those visual embeddings — making the reasoning compact, faster, and inspectable.

Key Contributions

1. A new paradigm for compressing reasoning: Render-of-Thought is presented as the first framework to reify the reasoning chain by rendering textual steps into images, making latent reasoning explicit and traceable rather than opaque.

2. Vision encoders as semantic anchors: Instead of learning reasoning tokens from scratch, the method freezes a pre-trained vision encoder and aligns the LLM's latent states with its visual embeddings, enabling plug-and-play use without additional pre-training.

3. A two-stage training framework: Stage I aligns LLM hidden states to visual embeddings via a projection head (with MSE alignment loss plus cross-entropy), and Stage II fine-tunes the LLM with LoRA to autoregressively generate the latent reasoning trajectory followed by the final answer.

4. Empirical validation of visual latent reasoning: Experiments show 3-4x token compression and substantial inference acceleration versus explicit CoT, while maintaining competitive accuracy across mathematical and logical benchmarks.

Main Findings

• Token compression and accuracy trade-off: On Qwen3-VL-4B-Instruct, RoT reaches 55.4% average accuracy using only 32 latent tokens, compared to 79.3% accuracy with 108.4 tokens for explicit CoT (SFT-CoT) and 47.8% for SFT without CoT.

• Near-parity on simpler tasks: On MultiArith, RoT achieves near-parity performance with a 1.8x reduction in token consumption (97.2% Pass@1 at 32 tokens vs 98.3% at 59.1 tokens for SFT-CoT).

• Outperforms LLM-based latent methods: Against baselines reproduced on Qwen3-4B-Instruct, RoT's 55.4% average accuracy exceeds the best LLM-based latent method, CoLaR-2, by 8.1% (47.3% average). CoLaR-2 is slightly higher on GSM8k-Aug (40.0% vs 37.8%), which the authors attribute to stronger out-of-domain generalization from the pre-trained visual encoder.

• Performance on hard tasks (MATH): On Qwen3-VL-4B-Instruct, explicit CoT achieves 55.8% accuracy at 291.5 tokens, while RoT achieves 33.2% Pass@1 using only 64 latent tokens, surpassing the 29.4% w/o CoT baseline.

• Cross-architecture consistency: The same pattern holds on Qwen3-VL-2B-Instruct (RoT average 37.0% at 32 tokens vs SFT-CoT 63.8% at 117.5 tokens) and LLaVa-V1.6-Mistral-7B (RoT 34.3% vs SFT-CoT 48.3% at 130.8 tokens).

• Inference speed: On the challenging GSM-Hard dataset, per-sample inference time drops from 8.55s to 1.84s. All timing experiments ran on a single NVIDIA H20 GPU with batch size 1.

• Both training stages are necessary: Removing Stage I drops GSM8k-Aug accuracy from 37.8% to 24.8%; removing Stage II drops MATH performance from 33.2% to 26.2%.

• Rendering layout matters: Single-line rendering with dynamic width, 32 px height, 20 px font size, and 4 px padding (37.8% Pass@1 on GSM8k-Aug) converges faster and scores better than fixed-size 1024 px x 1024 px square images with multi-line text wrapping.

• Fixed token budgets beat dynamic termination: Dynamic termination via the <|img_end|> special token yields only 3.87% (GSM8k-Aug) and 2.20% (MATH). Fixed budgets perform best at 32 tokens on GSM8k-Aug (37.8%) and 64 tokens on MATH (33.2%); budgets of 8, 16, 128, and 256 tokens all perform worse.

• Latent tokens saturate: The authors observe that generated latent tokens become increasingly homogeneous after a certain position — the token similarity matrix approaches 1.0 and feature activation heatmaps become nearly identical — suggesting the core reasoning is encoded early and later tokens maintain semantic context.

• Projection head design: SwiGLU (37.8% / 33.2% on GSM8k-Aug / MATH) beats GELU (35.1% / 30.8%) and ReLU (33.2% / 28.6%), and a hidden dimension of 4096 beats 2048 (34.5% / 30.1%).

• Unified backbone comparison: When latent baselines are adapted to Qwen3-VL-4B-Instruct, RoT still leads with 55.4% average Pass@1 vs CoLaR-2 at 45.3%, Coconut at 30.5%, and CODI at 9.38%.

• Non-mathematical generalization: On GPQA and ProsQA, RoT reaches 60.5 and 100.0 Pass@1, compared to CoLaR-2 at 59.4 and 99.4, CODI at 51.4 and 98.6, and Coconut at 50.2 and 99.6.

• Training cost: RoT training totals 56.1 hours (19.0h Stage I + 37.1h Stage II) versus 36.9h for SFT-CoT — roughly 1.5x — but inference is where the 3-4x speedup is realized.

Methodology in Plain English

The approach rests on a simple but unusual idea: treat the written reasoning steps as an image, and let a vision model read them.

1. Render the reasoning text. The textual Chain-of-Thought for each training problem is drawn as a single-line image — black text on a white background, fixed height (32 px), dynamic width, 20 px font, 4 px padding. Keeping it single-line means image patches are extracted strictly left-to-right, which matches the natural order of the text and avoids spatial ambiguity.

2. Encode the image. An existing, frozen vision encoder (the native module from Qwen3-VL) turns that image into a sequence of visual embeddings. These become the target the model must reproduce.

3. Stage I — Visual Alignment. The LLM backbone and vision encoder are both frozen. Only a small projection head — a two-layer MLP with SwiGLU activation — is trained. It maps the LLM's hidden state at the output side into the vision encoder's embedding space, trained with a Mean Squared Error alignment loss plus cross-entropy loss on the answer and the <|img_end|> token. This is the inverse of how multimodal LLMs usually work, where visual features are projected into the LLM's input space.

4. Stage II — Latent Supervised Fine-Tuning. The vision encoder and now-aligned projection head are frozen, and the LLM backbone is fine-tuned with LoRA. The model learns to autoregressively emit a sequence of latent visual tokens followed by the <|img_end|> token and the final textual answer. The loss here is just the answer/token prediction loss — no explicit visual regression — so the model optimizes purely for answer accuracy within the aligned latent space.

5. Inference. Rendering and the vision encoder are discarded entirely. The model runs a forward pass through the fine-tuned LLM backbone and projection head, generating a fixed budget of latent tokens before switching to text. Dynamic stopping via the special token was tested but proved unstable, so fixed budgets were used.

Training ran for 3 epochs total (1 in Stage I, 2 in Stage II) with AdamW, learning rate 2e-5, weight decay 1e-2, batch size 16, LoRA with alpha=32 and r=16, and an alignment loss weight of 10.0. Evaluation used five random seeds with 95% confidence intervals.

Datasets: GSM8k-Aug-NL (approximately 385k training samples, over 1k test samples), GSM-Hard (over 1k test samples), SVAMP, MultiArith, and MATH (7.5k training, 0.5k test). For MATH, since no official validation set exists, 10% of the shuffled training set was used for validation, following the CoLaR protocol.

Why This Matters

Impact on research. The paper reframes a tension in latent reasoning research: compression usually costs interpretability, because latent vectors are opaque. By grounding latent states in visual embeddings of rendered text, RoT offers intermediate reasoning that can be visualized and inspected — a partial answer to the "black box" problem. It also opens a bridge between two literatures that have largely stayed separate: "text-as-image" input compression and implicit Chain-of-Thought reasoning. The paper positions itself as the first to apply visual rendering to compressing reasoning steps rather than just inputs.

Real-world applications:

• Cost reduction in deployed reasoning models: A 3-4x token compression plus the reported GSM-Hard inference drop from 8.55s to 1.84s per sample translates directly into lower serving costs and faster response times for reasoning-heavy products.
• Auditable AI systems: Domains like education, finance, and healthcare need to inspect how a model reached an answer. Visual latent tokens that can be rendered and inspected offer a middle ground between unusably verbose text traces and fully opaque vectors.
• Edge and latency-sensitive deployment: Compressing reasoning into 32 or 64 latent tokens instead of hundreds of text tokens makes on-device or low-latency reasoning pipelines more feasible.
• Existing VLM upgrades: Because the method is plug-and-play using the model's own frozen vision encoder, organizations with deployed VLMs like Qwen3-VL or LLaVa could adapt them without building a new model from scratch or running extra pre-training.

Industry relevance. The work comes from Tencent BAC with academic collaborators and ships with a public code repository, suggesting an intent for practical adoption. The use of LoRA, frozen backbones, and existing vision encoders keeps the barrier to experimentation low — the main new component is a small projection head.

Future Directions

• Language and domain coverage. The authors state plainly that evaluation is limited to English mathematical and logical reasoning. Commonsense reasoning, causal inference, and multilingual settings remain unexplored.

• Adaptive token budgets. The optimal latent budget must currently be tuned per task — 32 tokens for GSM8k-Aug versus 64 for MATH. The paper suggests developing adaptive mechanisms that adjust based on problem difficulty, or learning budget predictors from problem characteristics.

• Fixing dynamic termination. Self-regulated stopping via special tokens is unstable in continuous latent spaces (3.87% and 2.20% Pass@1), a phenomenon the authors link to prior work and explicitly defer to future work.

• Reducing training overhead. Rendering CoT to images and running the vision encoder makes training about 1.5x slower than text-only SFT-CoT (56.1h vs 36.9h). Offline rendering strategies or caching mechanisms are proposed as mitigations, since the cost disappears at inference.

• Understanding the saturation plateau. The observation that latent tokens become highly similar after an early phase raises an open question: what exactly are the redundant tokens doing, and could they be pruned to compress further?

Target Audience

This paper is best suited to machine learning researchers and engineers working on LLM reasoning efficiency, latent/implicit Chain-of-Thought, or multimodal model architectures. Practitioners who deploy reasoning models and care about inference cost, latency, or the auditability of model outputs will also find it relevant. It is accessible to readers with a working understanding of how LLMs generate tokens and what Chain-of-Thought prompting does, but the two-stage alignment formulation and latent-space discussion assume some background in model training and representation learning.

Authors’ abstract

Chain-of-Thought (CoT) prompting has achieved remarkable success in unlocking the reasoning capabilities of Large Language Models (LLMs). Although CoT prompting enhances reasoning, its verbosity imposes substantial computational overhead. Recent works often focus exclusively on outcome alignment and lack supervision on the intermediate reasoning process. These deficiencies obscure the analyzability of the latent reasoning chain. To address these challenges, we introduce Render-of-Thought (RoT), the first framework to reify the reasoning chain by rendering textual steps into images, making the latent rationale explicit and traceable. Specifically, we leverage the vision encoders of existing Vision Language Models (VLMs) as semantic anchors to align the vision embeddings with the textual space. This design ensures plug-and-play implementation without incurring additional pre-training overhead. Extensive experiments on mathematical and logical reasoning benchmarks demonstrate that our method achieves 3-4x token compression and substantial inference acceleration compared to explicit CoT. Furthermore, it maintains competitive performance against other methods, validating the feasibility of this paradigm. Our code is available at https://github.com/TencentBAC/RoT