LinCa: Accelerating Diffusion Models via Learnable Decomposed Feature Caching

AI summary

Overview

Research area: Computer vision and generative AI — specifically inference acceleration for diffusion transformers (DiTs) used in image generation, image editing, and video generation.

Technical level: Intermediate to Advanced. The caching concept and empirical results are accessible to anyone familiar with diffusion sampling; the invertible-network construction (orthogonal convolutions, additive coupling layers) and the Hermite-extrapolation math require some background in differentiable architectures.

One-sentence scope: The paper introduces LinCa, a feature-caching framework that uses tiny learnable invertible networks to decompose cached diffusion features into sub-components with distinct temporal behavior, then applies prediction rules matched to each component, achieving 5–7× speedups with near-lossless output quality.

What This Paper Is About

Diffusion models generate images and video by running a large neural network dozens of times in sequence, which makes inference slow and expensive. A popular shortcut called feature caching runs the full network only once every N steps and guesses the intermediate features for the skipped steps based on previously cached ones. Existing caching methods guess with a single fixed rule applied to every timestep, every model, and every feature dimension. This paper shows that this assumption is wrong — feature behavior varies dramatically across timesteps, across models, and even across individual dimensions within a single layer — and proposes a learned, decomposing alternative that adapts to those differences.

Key Contributions

1. A diagnosis of "dynamics mismatch" at two levels. Through PCA visualization of denoising trajectories, the authors demonstrate that (a) feature evolution patterns differ substantially across timesteps and across models, and (b) different feature dimensions within the same layer at the same timestep have markedly different continuity — some are stable and predictable, others mutate abruptly. Critically, these two kinds of dimensions are interleaved in the original feature space and cannot be separated by simple index-based partitioning.

2. The LinCa framework: a learnable Decompose–Predict–Reconstruct pipeline. A lightweight invertible network projects cached features into M sub-components (M = 3 in practice) whose continuity properties match the prediction order applied to them. The zeroth-order component (unstable) is copied from the nearest cache; higher-order components are extrapolated with Hermite interpolation of matching order. Strict invertibility of the network guarantees that predicted sub-components can be mapped back to the original feature space with zero reconstruction error.

3. Model- and segment-specific training at negligible cost. The denoising trajectory is split into segments, and an isomorphic but independently parameterized predictor is trained per segment, allowing adaptation to each model's dynamics. Total overhead is under 0.2% of the base model's parameters, training requires only 100–200 pre-generated feature samples, and completes in under an hour on a single 12GB GPU — without ever loading the diffusion model's weights.

4. Broad empirical validation. LinCa is tested on FLUX.1-dev, Qwen-Image, Qwen-Image-Edit, HunyuanVideo, and on distilled or quantized variants (FLUX.1-lite-8B, FLUX.1-schnell, FLUX.1-dev-int8), consistently outperforming training-free baselines at matched acceleration ratios.

Main Findings

• Feature dynamics are genuinely heterogeneous. PCA trajectories show stable, continuous segments alongside abrupt mutations both across timesteps within one model and across models (FLUX.1-dev vs. Qwen-Image). Dimension-level analysis shows low-continuity dimensions with sharp changes and high-continuity dimensions that are well-behaved — justifying the decomposition approach.

• Best speed-quality trade-off on FLUX.1-dev. LinCa reaches 3.32× acceleration at N = 4 with an ImageReward of 1.0175, 4.52× at N = 6 with 1.0228, and 5.51× at N = 8 with 1.0162 — all above the original 50-step baseline's 0.9930. Comparable baselines degrade sharply at these ratios: FORA drops to 0.8276, DBCache to 0.5106, and TeaCache to 0.8421 at their highest settings.

• Strong results on Qwen-Image. At N = 10, LinCa sustains an ImageReward of 1.0524 at 6.95× speedup (reported as 5.46× on latency and 6.95× on FLOPs), while TaylorSeer falls to 0.7318 and DuCa to 0.4104. At lower ratios (N = 3), LinCa achieves 1.2329 versus TaylorSeer's 1.0674 and the 50-step baseline's 1.2532.

• Near-lossless video generation. On HunyuanVideo at N = 6, LinCa delivers 5.50× speedup with a VBench score of 80.16, against 80.66 for the unaccelerated 50-step baseline — a 0.6% drop. Clusca (79.99) and Speca (79.98) score lower with less speedup.

• Editing quality can exceed the original model. On Qwen-Image-Edit at N = 7, LinCa scores 7.45 (Chinese) and 7.56 (English) overall on GEdit-Bench, surpassing the 50-step baseline's 7.41 and 7.54. At N = 10 it still holds 7.27/7.40, whereas DuCa and TaylorSeer collapse to roughly 6.4.

• Compatibility with existing compression techniques. Applied on top of FLUX.1-lite-8B, LinCa adds 2.32× speedup with ImageReward rising to 0.9070 (from 0.8936). On the 4-step FLUX.1-schnell it adds 1.99× (0.9843 vs. 0.9692), and on INT8-quantized FLUX.1-dev it adds 2.63× (1.0036 vs. 0.9744).

• Invertibility and differentiated orders both matter. Ablations show the learnable invertible network consistently beats both a non-invertible learned MLP and an untrained mapping, indicating that lossless reconstruction avoids information loss during projection. Combined 0th + 1st + 2nd order prediction outperforms any single-order strategy.

Methodology in Plain English

The starting point. The method caches the cumulative residual feature at the final layer of the diffusion model. Normally, the model does a full forward pass every N steps and caches those features; on the skipped steps, a prediction function reconstructs them from cached history. Prior work uses one fixed function (identity reuse, Taylor expansion, Hermite interpolation) for everything.

Step 1 — Decompose. Instead of predicting the raw cached feature, LinCa pushes it through a small learned network that splits the feature dimension into three groups. The network is trained so that dimensions which behave similarly over time end up in the same group, even though they were interleaved in the original space.

Step 2 — Predict with matched rules. Group 0 holds the jumpy, hard-to-predict dimensions; for these, the method simply reuses the closest cached value. Groups 1 and 2 hold smoother dimensions and are extrapolated using first- and second-order Hermite interpolation from historical caches. These extrapolations are closed-form and essentially free computationally.

Step 3 — Reconstruct. The predicted groups are concatenated and passed through the inverse of the learned mapping to return to the original feature space. Because the mapping is built from strictly invertible pieces — an invertible 1×1 convolution (parameterized by an orthogonal matrix) for channel mixing, followed by an additive coupling layer with two small MLPs — the inverse exists exactly, so reconstruction introduces no additional error. Each layer's inverse is just subtraction plus multiplication by the transposed mixing matrix.

Step 4 — Train per model and per segment. The denoising trajectory is divided into several segments, and a separate copy of this small network is trained for each segment, so the predictor can match the distinct dynamics of early, middle, and late denoising. Training uses only 100–200 pre-generated feature samples and a loss combining the end-to-end prediction error in the original feature space with a per-component error that encourages the network to cluster suitable dimensions together.

Why this is cheap. The added parameters are under 0.2% of the base model. Since training operates purely on pre-generated features, the diffusion model itself is never loaded — training fits on a single 12GB GPU in under an hour, and inference runs at essentially the same speed as training-free caching methods.

Why This Matters

Impact on research. Feature caching has largely been a hand-designed, training-free heuristic. LinCa reframes it as a small learning problem: identify the structure of feature dynamics and learn a lossless reorganization that exposes it. The finding that a strictly invertible mapping outperforms a more expressive non-invertible one is a useful, transferable insight — reconstruction fidelity can matter more than representational capacity. The analysis also provides a clean vocabulary (cross-timestep, cross-model, and cross-dimension dynamics mismatch) that future caching work can build on.

Real-world applications:

• Interactive image generation and editing tools where users iterate on prompts and need sub-second responses; the 6.95× and 7.08× results on Qwen-Image and Qwen-Image-Edit map directly to this.
• Video generation and synthesis pipelines, where per-clip cost is dominated by denoising over many frames; a 5.50× speedup on HunyuanVideo changes what is economically viable.
• Cost reduction in cloud inference serving. Since the method stacks cleanly with quantization and distillation, a serving fleet can compound 2–3× compression gains with 5–7× caching gains.
• Deployment on constrained hardware, including single-GPU workstations and edge or terminal devices, given the low training memory footprint and compatibility with INT8 models.

Industry relevance. The method is a drop-in module: it does not modify the diffusion model architecture, does not require retraining the base model, and its training is cheap enough to be done per-deployment. That combination — no architectural surgery, minimal training data, tiny parameter overhead, multiplicative compatibility with existing compression — is what makes it practical rather than merely academic.

Future Directions

• Adaptive decomposition and order selection. M is fixed at 3 and the per-segment structure is chosen by hand. Automatically deciding how many sub-components to use, how many denoising segments to split into, and which prediction order each component deserves could remove the remaining manual tuning.
• Generalization beyond the training distribution. The predictors are trained on 100–200 pre-generated samples. How they behave on out-of-distribution prompts, unusual aspect ratios, or novel styles, and whether periodic lightweight adaptation would help, remains open.
• Combination with complementary acceleration axes. The method already layers onto quantization, model distillation, and step distillation. Combining it with token pruning or merging, attention-level caching, or spatial clustering methods such as Clusca points toward strictly larger compounded speedups.
• Broader modality and architecture coverage. Testing on audio diffusion, 3D or 4D generation, larger video models, and autoregressive or hybrid architectures would clarify whether the "heterogeneous dynamics" observation is universal or specific to the transformer-based visual generators studied here.
• Theoretical grounding. There is no formal account yet of why dimensions cluster into distinct continuity classes, or of how prediction error propagates and accumulates over a long denoising trajectory. Such analysis could yield principled bounds on the achievable acceleration ratio.

Target Audience

Graduate students and researchers working on diffusion model efficiency, generative model inference, or efficient deep learning more broadly. ML systems and infrastructure engineers responsible for deploying image or video generation models at scale, who need to reduce GPU cost without retraining or degrading output quality. Practitioners building interactive creative tools who need faster iteration. The paper is most valuable to readers who already understand the diffusion sampling loop and are looking for a concrete, low-overhead acceleration mechanism that composes with other compression techniques; readers without any diffusion background will find the empirical results clear but the methodology section demanding.

Authors’ abstract

Diffusion models have achieved remarkable success in image and video generation, yet the high computational cost of iterative sampling remains a critical bottleneck for practical deployment. Feature caching has emerged as a promising acceleration paradigm by reusing or predicting intermediate features across timesteps. However, existing training-free methods apply uniform prediction strategies that cannot adapt to the heterogeneous feature dynamics, causing significant quality degradation under high acceleration ratios. We propose LinCa, a feature caching framework based on learnable invertible networks. LinCa decomposes cached features into sub-components with distinct continuity properties via a lightweight invertible network and applies differentiated prediction orders matched to each component. The strict invertibility guarantees lossless reconstruction back to the original feature space, forming a unified Decompose-Predict-Reconstruct pipeline. By training separate predictors for different models and timestep segments, LinCa adapts to heterogeneous feature dynamics. Experiments on FLUX, Qwen-Image, and HunyuanVideo demonstrate that LinCa, with less than 0.2% additional parameters, significantly outperforms existing methods and maintains near-lossless quality at 5-7x speedup. Code: https://github.com/QHR69/LinCa