CFG-Ctrl: Control-Based Classifier-Free Diffusion Guidance

AI summary

Overview

Research area: Generative computer vision — specifically guidance mechanisms for flow-matching diffusion models used in text-to-image synthesis. The paper bridges generative modeling with classical control theory.

Technical level: Advanced. The paper assumes familiarity with Classifier-Free Guidance (CFG), flow matching / velocity fields, ordinary differential equations, and it introduces sliding mode control and Lyapunov stability analysis.

One-sentence scope: The paper reframes Classifier-Free Guidance as a feedback controller on a continuous-time generative flow, then proposes a nonlinear sliding-mode variant (SMC-CFG) and validates it on three large text-to-image backbones.

What This Paper Is About

Classifier-Free Guidance improves how well generated images match text prompts by amplifying the difference between a model's conditional and unconditional velocity predictions, but this amplification is linear, and at large guidance scales it produces oversaturated colors, warped structures, and unstable trajectories. The authors' central claim is that this discrepancy behaves like an error signal in a control loop, so guidance can be redesigned using control-theoretic tools rather than ad hoc extrapolation rules. Their goal is a guidance method that converges stably and keeps semantic fidelity even at high guidance scales.

Key Contributions

1. CFG-Ctrl, a unified control-theoretic framework. The paper models flow-matching sampling as a control-affine ODE and decomposes any guidance update into a guidance schedule K_t (strength) and a direction operator Π_t (direction) acting on the semantic error e(t) = v_θ(x_t, t, c) − v_θ(x_t, t, ∅). Under this lens, standard CFG is a proportional controller with fixed gain, and follow-up methods are re-cast as different control laws (see their Table 1: weight scheduling, projection-based feedback control, model predictive control).

2. SMC-CFG (Sliding Mode Control CFG). A nonlinear guidance mechanism that defines an exponential sliding surface s(t) = ė(t) + λe(t) over the semantic prediction error and adds a switching control term Δe(t) = −k·sign(s_t) that drives the trajectory onto that surface.

3. Lyapunov stability analysis. Assuming a lower-bounded minimum singular value for the coefficient matrix (σ_min(Γ_s) ≥ b_min > 0) and a bounded disturbance term (‖Φ_s‖ ≤ δ), the paper shows V̇ = sᵀṡ ≤ −η‖s‖ with η = k·b_min − δ > 0 when k·b_min > δ, which supports finite-time convergence of ‖s(t)‖ to zero.

4. Empirical validation across three backbones. Experiments on SD3.5, Flux-dev, and Qwen-Image (8B, 12B, and 20B parameters respectively) on a 5,000 image-text-pair subset of MS-COCO, against standard CFG, CFG-Zero⋆, and Rectified-CFG++.

Main Findings

• CFG is mathematically a proportional controller. Substituting K_t = w and Π_t = I into the general state-feedback law recovers the standard CFG update exactly, so the guidance scale w plays the role of the proportional gain.

• Existing variants are alternative control laws, not fundamentally new mechanisms. Weight scheduling is time-varying gain scheduling (K_t = w(t), Π_t = I); APG is projection-based feedback control with a matrix gain w[I ηI] and a projection operator built from P_t; Rectified-CFG++ is characterized as Model Predictive Control; CFG-Zero⋆ is also classed as projection-based feedback control.

• SMC-CFG achieves the lowest FID on all three backbones among guided methods. SD3.5: 20.044 (vs. CFG 21.421, CFG-Zero⋆ 20.317, Rect-CFG++ 20.550). Flux-dev: 26.398 (vs. CFG 27.323, CFG-Zero⋆ 26.901, Rect-CFG++ 27.219). Qwen-Image: 33.371 (vs. CFG 35.431, CFG-Zero⋆ 35.391, Rect-CFG++ 34.371). The unguided Qwen-Image model has the lowest FID overall (24.894), which the paper attributes to its preservation of natural image statistics.

• SMC-CFG gets the highest CLIP score on all three backbones. SD3.5: 0.3694; Flux-dev: 0.3743; Qwen-Image: 0.3856 — each above CFG, CFG-Zero⋆, and Rect-CFG++.

• Highest ImageReward, HPSv2.1, and MPS across all three backbones. ImageReward: 0.9486 (SD3.5), 1.0558 (Flux-dev), 1.2035 (Qwen-Image). HPSv2.1: 0.2875, 0.3021, 0.3105. MPS: 7.5719, 8.2307, 8.4320.

• Some metrics are matched rather than dominated. On SD3.5, CFG-Zero⋆ has a higher Aesthetic score (5.6124 vs. 5.5790). On Flux-dev, CFG-Zero⋆ has a marginally higher HPSv2 (0.2987 vs. 0.2986). On Qwen-Image, CFG-Zero⋆ has higher Aesthetic (5.6598 vs. 5.6289) and PickScore (0.2279 vs. 0.2275). The paper characterizes these as "comparable or better" results.

• Qualitative improvements on hard prompts. Figures 2 and 3 report sharper details, more coherent object structures, better positional relationships and text generation, and fewer irrational outputs on prompts involving relative positions, clothing styles, and human actions.

• Fixed hyperparameters per model. The two SMC hyperparameters (λ and the switching gain K) are held constant within each model and shared across all datasets and conditions. The exact values are not reported in the visible content — they are deferred to the supplementary material.

Methodology in Plain English

The authors start from an observation about how diffusion/flow models behave during sampling: the model is asked to make two predictions at each step, one with the text condition and one without. Early in sampling these two predictions differ a lot; late in sampling they nearly agree. Standard CFG simply multiplies that gap by a fixed number and adds it back into the trajectory.

The authors instead treat the gap as an error signal in a control loop and ask what a control engineer would do with it. Their answer: define a target "surface" through the (error, error-rate) plane that the system should ride on. This surface is s = ė + λe, a simple line through the origin. If the state sits on the surface, the error decays exponentially at a rate set by λ. The method computes its position relative to the surface each step and pushes the error back toward it with a hard switching term proportional to −k·sign(s) — a correction that does not depend on the size of the deviation, only its direction, which is what gives sliding mode control its robustness.

They then prove the push actually works: by defining an "energy" V = ½‖s‖², they show that under standard boundedness assumptions, the correction makes the energy decrease monotonically at a guaranteed rate and drives the state to the surface in finite time. On top of this they add a time-varying component (an exponential sliding mode surface variant) to the formulation.

Finally they plug SMC-CFG into three off-the-shelf pretrained text-to-image models, using each model's default CFG scale, and compare against standard CFG and two flow-matching-specific guidance baselines on 5,000 MS-COCO image-text pairs, measuring FID, CLIP, and five human-preference/aesthetic metrics. All experiments run on a single NVIDIA A100 GPU (40GB).

Why This Matters

Impact on research: The paper claims to unify a scattered family of CFG improvements — weight schedulers, orthogonal projections, predictive correctors — under a single state-feedback formalism, which makes it possible to compare and design guidance rules using control-theoretic criteria (stability, convergence rate, disturbance rejection) instead of empirical tuning. It also imports a well-established nonlinear control technique into generative modeling, suggesting that other robust control paradigms could be brought to bear.

Real-world applications:

• Text-to-image products where users crank up the guidance scale to force prompt adherence and currently get oversaturated or structurally broken images.
• Advertising and marketing content pipelines that need reliable adherence to detailed product or brand descriptions.
• Game, film, and industrial design prototyping, where prompts often specify spatial relationships, clothing styles, and human actions — the categories the paper highlights in its qualitative comparisons.
• Accessibility tooling that turns detailed textual instructions into images, where positional and attribute fidelity matter more than aesthetics alone.

Industry relevance: The method is described as model-agnostic and was validated across three different backbones at three different parameter scales (8B, 12B, 20B), which is the kind of generality needed to ship a guidance change across a product's model lineup. The paper does not report the computational overhead of the added switching term or its memory cost, so the practical cost of adoption is not established here.

Future Directions

• Characterizing the overhead. The paper does not report inference latency, step-count, or memory cost for SMC-CFG relative to CFG; establishing whether the extra switching term is essentially free would strongly affect adoption.
• Sensitivity to λ and k. The exact values and selection procedure for the sliding-surface parameter λ and the switching gain k are not given in the visible content, and finite-time convergence requires k·b_min > δ. How these are chosen in practice, and how robust the method is when that condition is only approximately met, remains open.
• Extending CFG-Ctrl beyond guidance. The framework is explicitly built for the CFG error signal in flow matching; whether the same control formalism can be applied to other controllable generation settings, or to training-time objectives, is untested.
• Broader evaluation. The main quantitative study covers one 5,000-pair MS-COCO subset and three backbones, with additional benchmarks and models deferred to supplementary material. Independent validation on other datasets, other modalities such as video, and non-flow diffusion formulations would test the generality of the claims.

Target Audience

This paper is best suited for researchers and engineers already working on diffusion or flow-matching generative models who want to understand or redesign guidance mechanisms, and for readers with a control-systems background interested in seeing sliding mode control and Lyapunov analysis applied to generative modeling. A reader without comfort in ODEs, matrix projections, and stability proofs would find the theoretical sections dense, though the framing of CFG as a proportional controller is accessible on its own. Practitioners evaluating whether to swap out CFG in a production text-to-image pipeline will find the benchmark table directly useful, but will need the supplementary material for the hyperparameter settings and overhead figures that are not present in the paper body.

Authors’ abstract

Classifier-Free Guidance (CFG) has emerged as a central approach for enhancing semantic alignment in flow-based diffusion models. In this paper, we explore a unified framework called CFG-Ctrl, which reinterprets CFG as a control applied to the first-order continuous-time generative flow, using the conditional-unconditional discrepancy as an error signal to adjust the velocity field. From this perspective, we summarize vanilla CFG as a proportional controller (P-control) with fixed gain, and typical follow-up variants develop extended control-law designs derived from it. However, existing methods mainly rely on linear control, inherently leading to instability, overshooting, and degraded semantic fidelity especially on large guidance scales. To address this, we introduce Sliding Mode Control CFG (SMC-CFG), which enforces the generative flow toward a rapidly convergent sliding manifold. Specifically, we define an exponential sliding mode surface over the semantic prediction error and introduce a switching control term to establish nonlinear feedback-guided correction. Moreover, we provide a Lyapunov stability analysis to theoretically support finite-time convergence. Experiments across text-to-image generation models including Stable Diffusion 3.5, Flux, and Qwen-Image demonstrate that SMC-CFG outperforms standard CFG in semantic alignment and enhances robustness across a wide range of guidance scales. Project Page: https://hanyang-21.github.io/CFG-Ctrl