Self-Conditioning Enables Deep Sequential Reasoning in Flow Matching

AI summary

Overview

Research area: Generative visual reasoning with diffusion/flow-matching image generators, analyzed through circuit complexity theory.

Technical level: Advanced — the paper combines formal circuit-complexity proofs (AC⁰, TC⁰, NC¹, P/poly, PSPACE/poly) with large-scale image-generation experiments.

Scope: The paper proves that vanilla flow matching's many sampling steps add no expressive power for constant-depth backbones, and shows that self-conditioning removes this barrier both in theory and in controlled experiments.

What This Paper Is About

Models can now solve visual puzzles by drawing the answer with an image generator, and it seems natural that a flow-matching sampler's K steps act like K reasoning steps. This paper argues that intuition is wrong: it proves that a vanilla sampler's whole trajectory can be compressed into a constant number of parallel model-evaluation rounds, so more steps buy no extra reasoning depth. The authors then show that a simple change — feeding the model its own previous guess of the final image (self-conditioning) — restores a genuine per-step scratchpad and lifts expressive power dramatically.

Key Contributions

1. Self-conditioning as an internal scratchpad. The paper reinterprets self-conditioning not as an empirical regularizer but as a mechanism that carries a rich, abruptly-changing computational state (an endpoint prediction) from one sampling step to the next, enabling deep sequential computation.

2. A circuit-complexity collapse theorem for vanilla flow matching. Under mild regularity conditions (B-bounded, L-Lipschitz velocity model, polynomial state dimension, and precision in a specified range), the languages decided by vanilla flow matching stay inside the backbone's own circuit class (e.g., AC⁰ or TC⁰), regardless of the number of sampling steps.

3. An expressivity lower bound for self-conditioned flow matching. The authors prove that an AC⁰-computable, 4-bounded, 8-Lipschitz model with self-conditioning can decide any P/poly language with polynomially many steps and any PSPACE/poly language with up to exponentially many steps — a formal separation between the two samplers.

4. Empirical validation across synthetic and structured tasks. Controlled experiments on tasks with known circuit complexity (NC¹-complete line tracing; TC⁰-complete long multiplication and long text rendering) plus five structured visual-reasoning tasks show consistent large gains for self-conditioning, including a jump from 65.9% to 76.7% average accuracy on the controlled suite.

Main Findings

• Vanilla flow matching has an "illusion" of depth. Theorem 1 shows that if the velocity model is B-bounded and L-Lipschitz with L constant, the entire K-step Euler trajectory can be reproduced by O(L) sequential rounds of at most K(m) parallel model calls (Lemma 1), so the decided language remains in the backbone's class. Precision must satisfy (1+Ω(1))·log₂K(m) ≤ b(m) ≤ O(log m).

• The proof technique is a parallel Picard iteration. A naive re-simulation would need e^{O(L)} steps because the ODE can amplify error by e^L; updating all K positions in parallel makes the error decay factorially across rounds, giving O(L) rounds. Lemma 1 requires b(m) ≥ log₂K(m) + (log₂e)·L(m) + c₀.

• Self-conditioning breaks the barrier. Feeding back z_i = x_{i−1} + (1 − t_{i−1})·v_{i−1} (with z₀ = 0ⁿ) lets information change abruptly between steps instead of by an O(1/K) increment, so the constant-depth collapse no longer applies.

• A sharp separation. Theorem 2 gives AC⁰ models that decide P/poly with polynomial steps and PSPACE/poly with at most exponential steps. For D ∈ {AC⁰, TC⁰}, the samplers separate across P/poly \ D; the AC⁰ case holds unconditionally (AC⁰ ⊊ TC⁰, Furst et al. 1984), while the TC⁰ case relies on the conjecture TC⁰ ⊊ NC¹.

• The O(L) round bound is tight. Any polynomial-size majority circuit of depth d can be simulated under vanilla flow matching in O(d) rounds by an O(d)-Lipschitz TC⁰ model, so the bound cannot be improved to o(L(m)) rounds without collapsing the depth hierarchy.

• Robustness of the theory. Extensions cover growing Lipschitz constants (L = poly(log m) gives an NC upper bound; L = O(log^k m) gives TC^k), ℓ_p norms instead of ℓ_∞, and fixed explicit Runge–Kutta solvers such as Heun's method.

• Line tracing: 50 vs. near 8. At 100 NFEs, SelfCondFM reaches the maximum mean longest correct prefix of 50 at 1.10 × 10¹⁸ training FLOPs, while VanillaFM plateaus near 8.

• Long multiplication and long text rendering. SelfCondFM exceeds 90% accuracy within half the training budget on long multiplication, a level VanillaFM never reaches; on long text rendering SelfCondFM's accuracy rises with compute while VanillaFM stays near zero for most of training.

• Steps translate into depth only for SelfCondFM. On line tracing, both methods start at mean LCP 0 at NFE = 1; SelfCondFM reaches 49.84 at NFE = 10, whereas VanillaFM reaches 7.66 at NFE = 6 and stays below 8.0 through NFE = 50.

• Structured visual-reasoning gains. On FLUX.2-klein-base-4B LoRA adapters with matched training FLOPs and NFE, SelfCondFM beats VanillaFM on every task: VSP-Super 92.0 vs. 89.0, Maze 70.7 vs. 64.3, TSP 66.7 vs. 51.0, Sudoku 80.0 vs. 60.5, Jigsaw 74.3 vs. 64.7, average 76.7 vs. 65.9.

• Large-scale results. With a 20B Qwen-Image-Edit-2511 LoRA, task-specific adapters average 90.0% versus DiffThinker's 75.8% and EndoCoT's 88.3%; a single unified adapter trained on all four tasks reaches 90.2%, versus 66.1% (DiffThinker) and 76.6% (EndoCoT) under unified training. (Jigsaw is reported separately because EndoCoT does not evaluate it.)

• Trained models fit the theory's assumptions. The estimated Lipschitz constant of a VanillaFM model on mixed-length line tracing levels off at roughly 400 for m ≥ 30, consistent with the bounded-Lipschitz regime.

• The model actually uses the scratchpad. Attention analysis shows the endpoint prediction z_i receives a nontrivial attention share throughout generation, with a larger share early on when x_i is still noisy.

• Intrinsic advantages are preserved. SelfCondFM retains flow matching's ability to form multiple local results in parallel within one trajectory and to correct a wrong mid-trajectory generation in place.

Methodology in Plain English

The argument has two halves.

Theory. The authors model the sampler as a finite-precision computer: numbers have b fractional bits, states have dimension n, and the step count K can grow with problem size m. They require the velocity model to be bounded and Lipschitz continuous — small input changes cause only proportionally small output changes, with a constant factor. Under these conditions each Euler step moves the state by only O(1/K), so the entire continuous trajectory can be re-derived by iteratively refining all K time points at once, like solving a differential equation by repeated parallel correction rather than marching forward. Because this Picard-style iteration converges in O(L) rounds, a constant-depth backbone never escapes its own circuit class. Self-conditioning changes the picture by adding a second input channel — an extrapolated guess of the clean image — which is not constrained by the step size and can jump arbitrarily between steps. That freedom lets the sampler simulate a Turing machine, storing the tape and machine state in the endpoint prediction.

Experiments. To test whether trained models realize this theoretical gap, the authors keep everything identical between the two methods — same backbone, data, optimizer, noise schedule — and change only whether the endpoint prediction is passed along. Because SelfCondFM needs two forward passes per training example, they match cumulative training FLOPs rather than gradient steps, and compare at equal numbers of function evaluations (default 50 NFEs) since each NFE is one backbone call. Tasks are chosen so their circuit complexity sits exactly on the separation barrier: line tracing (NC¹-complete), long multiplication and long text rendering (TC⁰-complete). They then scale up to five structured visual-reasoning tasks from He et al. (2025) and finally to a 20B-parameter backbone compared against published baselines.

Why This Matters

Impact on research. The paper reframes a widely held intuition — that more diffusion steps mean more reasoning — as a provable misconception, and gives the field a concrete complexity-theoretic vocabulary for comparing generative reasoning methods. It also retroactively explains why self-conditioning, usually treated as a small quality trick, matters so much here, and it supplies a design principle: carry more information between sampling steps.

Real-world applications:

• Spatial planning and routing — mazes and traveling-salesperson-style layout problems solved by generating a plan image directly.
• Document and worksheet automation — step-by-step arithmetic and structured rendering tasks such as long multiplication worksheets or laying out multi-word paragraphs onto a fixed-width canvas.
• Puzzle and constraint solving — Sudoku and jigsaw assembly, where the model must commit partial solutions and revise them.
• Design and layout tooling — generating structured visual outputs where intermediate drafts need to be inspected and corrected in place.

Industry relevance. Self-conditioning is a minimal change — one extra conditioning input, one extra forward pass during training — that plugs into existing image-conditioned backbones such as FLUX.2 and Qwen-Image-Edit. For teams building visual reasoning products on diffusion models, it offers a way to convert inference compute into actual capability rather than into wasted sampling steps.

Future Directions

• Which inter-step channel is best? The paper shows the endpoint prediction works and is unconstrained by step size; it leaves open whether richer or differently structured scratchpads (beyond a single extrapolated image estimate) yield even greater depth.

• Closing the gap between theory and training. The separation is proven for models satisfying boundedness and Lipschitz conditions; the measured constant of roughly 400 for m ≥ 30 is suggestive but not a proof that trained models across all tasks stay in that regime.

• Extending to other solvers and norms. The analysis covers Euler, Heun's method, and fixed explicit Runge–Kutta schemes, and generalizes to ℓ_p norms; adaptive or implicit solvers, and growing Lipschitz constants beyond poly(log m), remain open.

• Scaling laws for reasoning depth. SelfCondFM keeps improving with NFE while VanillaFM saturates; characterizing how far that scaling holds at larger model and task scales, and how it interacts with the difficulty levels used in the large-scale comparison, is a natural next study.

Target Audience

Researchers in generative modeling and multimodal reasoning who want a rigorous account of why iterative image generators do or do not reason; complexity theorists interested in attention-based and diffusion-based computational models; and practitioners building visual-reasoning systems on flow-matching backbones, who can adopt self-conditioning as a low-cost change. Readers should be comfortable with diffusion/flow-matching basics and with complexity classes such as AC⁰, TC⁰ and NC¹.

Authors’ abstract

Recent work has explored visual reasoning through image generation, where models solve visual-reasoning problems by directly generating solution images. Intuitively, iterative samplers such as flow matching are well suited to this task, since each sampling step may resemble one reasoning step. However, in this work, from a circuit complexity perspective, we show that this intuition is misleading. We prove that as long as the model outputs satisfy certain mild regularity conditions, the trajectory of the vanilla flow-matching sampler can be simulated by a constant number of sequential rounds of parallel model evaluations. Consequently, additional sampling steps cannot increase the expressive power of a constant-depth backbone model under the standard complexity-theoretic model of transformers. We then reveal that self-conditioning, which conditions the model on the previous step's estimate of the final image, plays a previously unrecognized role in removing this bottleneck. Controlled experiments on diverse visual-reasoning tasks show that flow matching with self-conditioning achieves higher accuracy and greater reasoning depth than vanilla flow matching, and its performance improves with more sampling steps. Theoretically, we prove that with self-conditioning, a constant-depth backbone can achieve expressive power equivalent to that of space-bounded Turing machines. Our study suggests a design principle for future visual reasoning research: carry more information between sampling steps, so that additional steps translate into deeper reasoning.