Rate-distortion optimization for full-reference image quality metrics via stochastic Hessian estimates

AI summary

Overview

• Research area: Image and video compression, specifically perceptual rate-distortion optimization (RDO) inside block-based codecs; subfield of end-to-end coding, computer vision.
• Technical level: Advanced. The paper assumes familiarity with rate-distortion theory, hybrid video codecs (VVC/VTM), Hessian and Gauss-Newton approximations, and full-reference image quality metrics.
• Scope: The paper derives a block-wise, Hessian-based quadratic approximation of arbitrary differentiable full-reference image quality metrics and integrates it into a standard VVC encoder's RDO, reporting BD-rate gains and complexity for five metrics on Kodak and CLIC.

What This Paper Is About

Standard video codecs minimize the sum of squared errors (SSE) because it decomposes block-wise and makes RDO tractable, but perceptual metrics such as SSIM, MS-SSIM, LPIPS, DISTS, and Wasserstein Distortion (WD) align better with human vision and cannot be used directly in-loop: they do not decompose block-wise and require the fully decoded image. The paper's goal is to make RDO for these metrics possible inside a standard encoder by replacing SSE with an input-dependent quadratic distortion (IDQD) built from the metric's Hessian, without changing the decoder.

Key Contributions

1. IDQD-based RDO for any differentiable FR-IQA metric. Building on existing metric quadratization results, the paper approximates a broad class of full-reference metrics by the quadratic form (x - x̂)ᵀ H(x) (x - x̂), where H(x) is the Hessian of the metric evaluated at the source image, and uses it in place of SSE in partitioning, prediction, transform, and RDOQ decisions.
2. Two block-wise Hessian approximations with stochastic estimators. The paper keeps either the block-diagonal (tiles of n_t pixels) or only the diagonal of the Hessian, and estimates both from Hessian-vector products with Rademacher probes computed by automatic differentiation.
3. A lower-variance Gauss-Newton estimator. For metrics that are squared norms of a residual (LPIPS, DISTS, WD), the Hessian at the input takes the Gauss-Newton form 2 J(x)ᵀ J(x), so both estimators can be computed from vector-Jacobian products using Gaussian probes; these are positive semidefinite by construction and have diagonal entries with relative standard deviation sqrt(2/m).
4. Metric smoothing and a modified RDO Lagrangian. A Gaussian-smoothed Hessian is estimated with R noise draws, and the IDQD Lagrangian is derived from the SSE Lagrangian as λ_m = λ_s · E_x[tr Q(x)] / n_p with Q(x) = H(x) + τ I, to control the PSNR-versus-target-metric trade-off and fit standard rate control.

Main Findings

• Overall BD-rate gains: Across five metrics on Kodak and CLIC in VVC, IDQD-RDO achieves 14.2–36.7 % BD-rate savings under the target metric with no decoder changes and 10–30 % encoding complexity overhead. The introduction reports the block-diagonal estimator saving 15.5–23.6 % (Kodak) and 14.2–36.7 % (CLIC), up to 21.0 % more than PerceptQPA.
• Block estimator beats diagonal and PerceptQPA: In Table 2, for each RDO target the block estimator (B) outperforms the diagonal estimator (D) on the target metric; the block estimator outperforms PerceptQPA on every target metric, while the diagonal estimator outperforms PerceptQPA in 6 out of 10 cases. Best Kodak values per column, interpolated to +8.1 % Y-PSNR cost, are −23.6 % (SSIM), −16.8 % (MS-SSIM), −15.5 % (LPIPS), −17.3 % (DISTS), −19.1 % (WD). Best CLIC values, at +9.7 % Y-PSNR cost, are −36.7 % (SSIM), −16.9 % (MS-SSIM), −14.9 % (LPIPS), −14.2 % (DISTS), −16.0 % (WD).
• Locality holds strongly for some metrics and weakly for others: Averaged over rows and 3 Kodak images, the fraction of Hessian-row energy inside the containing 8×8 tile is 99 % for SSIM and MS-SSIM, 63 % for WD, 18 % for LPIPS, and 0.3 % for DISTS. Restricting to the diagonal entry alone gives 11 %, 42 %, 15 %, 4 %, and 0.02 % respectively.
• Metric map interactions: SSIM and DISTS maps degrade each other; the authors conjecture the two metrics measure different properties.
• Gauss-Newton and smoothing help neural-network metrics: In Table 3 (Kodak, target-metric BD-rate at +8.1 % Y-PSNR cost, interpolated), block Gauss-Newton with smoothing σ=3 gives −15.5 % (LPIPS), −17.3 % (DISTS), −19.1 % (WD), beating diagonal Gauss-Newton, unsmoothed Gauss-Newton, and the block Hessian-vector-product estimator (−9.5 %, −10.6 %, −15.0 %). Using σ=6 and 10 was worse.
• Comparison with a QP-offset method: Under the configuration of Yang & Bajić [29] on Kodak, the method matches their RGB-PSNR cost and improves the optimized metric: optimizing MS-SSIM gives +0.98 % RGB-PSNR and −17.53 % MS-SSIM (versus +0.98 % and −11.88 % for [29]); optimizing LPIPS gives −0.10 % RGB-PSNR and −17.63 % LPIPS (versus −0.10 % and −8.33 %). The authors first reproduce PerceptQPA in that environment, obtaining +2.52 % RGB-PSNR, −11.03 % MS-SSIM, −12.87 % LPIPS, close to the values reported in [29].
• Complexity: The diagonal estimator adds negligible encoding overhead; the block-diagonal estimator adds 10–30 %. Hessian estimation takes 1–10 s per image at m=256 probes. Using m=64 retains at least 83 % of the gain for every metric at 1/4 of the estimation time, and the neural-network-based metrics saturate first while SSIM and MS-SSIM continue to improve up to m=256.
• Statistical support on Kodak: Bootstrap 95 % intervals of the mean difference (block versus PerceptQPA) always exclude zero.

Methodology in Plain English

The authors start from the idea that any well-behaved metric, near the original image, can be approximated by a quadratic form whose curvature is given by the Hessian of the metric evaluated at the source image. That quadratic form, the IDQD, is what the encoder minimizes instead of SSE, and it is controlled by a regularizer τ that balances PSNR against the target metric.

The obstacle is that the Hessian has n_p² entries, is not available in closed form, and does not split into blocks. The authors make two simplifications: keep only the diagonal blocks of size n_t tiles (so each tile's distortion uses only its own block, and the total distortion is a sum over tiles), or keep only the diagonal (per-pixel weights). Because FR-IQA metrics are built from local operations to varying degrees, pixels far apart are weakly coupled, so these restrictions are reasonable for some metrics.

Rather than form the Hessian, they estimate it stochastically: random Rademacher probes are multiplied by the Hessian through Hessian-vector products obtained by automatic differentiation (one forward and two backward passes), and the diagonal is the average of probe times result, the block-diagonal the average of outer products within each tile. For metrics written as squared residual norms, a Gauss-Newton form replaces this with vector-Jacobian products and Gaussian probes, giving positive semidefinite estimates with lower variance. Because neural-network gradients vary rapidly under small input perturbations, they additionally smooth the metric with Gaussian noise and average Hessian estimates over R noise draws. Finally they derive how to scale the SSE Lagrangian into the IDQD Lagrangian using the trace of the Hessian, so rate control still works.

Experiments use VTM 23.8 all-intra at 8-bit 4:2:0 with QP in {22, 27, 32, 37}, m = 256 probes, 8×8 tiles, and σ = 3 with R = m/16 for LPIPS-VGG, DISTS, and WD. Settings are selected on Kodak and applied to the CLIC professional validation set (41 images, cropped to multiples of 8). SSIM and MS-SSIM use luma; LPIPS (VGG version), DISTS, and WD use RGB. Three values of τ are swept and interpolated to PerceptQPA's Y-PSNR BD-rate. IDQD replaces SSE in quadtree, binary, and ternary partitioning, intra luma and chroma mode selection, and transform choices, while RDOQ uses a weighted SSE in the transform domain. Runtime is measured on an NVIDIA A100 GPU and an Intel Xeon E5-2667 CPU.

Why This Matters

• Research impact: The work turns perceptual RDO from a metric-specific engineering problem (prior methods target only SSIM or WPSNR, or modify only per-block QP offsets) into a general procedure that accepts any differentiable full-reference metric, and it connects metric quadratization theory to practical encoder decisions. It also gives an empirical characterization, via Hessian energy fractions, of how local each metric is.
• Real-world applications:
  • Streaming and on-demand video services that want to encode for a chosen perceptual quality metric rather than PSNR.
  • Content delivery pipelines where bitrate must be reduced at a fixed perceptual quality target without changing deployed decoders.
  • Compression for learned or task-oriented pipelines that already use LPIPS, DISTS, or WD as their quality signal.
  • Codec tuning tools that need to trade off PSNR compliance and perceptual quality in a controllable way.
• Industry relevance: Decoder complexity is unchanged, and the encoder overhead (10–30 % for the block estimator through an offline/once-per-image Hessian estimate) is the kind of cost that can be absorbed in offline or high-quality encoding workflows. Compatibility with standard rate control via the modified Lagrangian makes adoption by codec implementers plausible.

Future Directions

• Reducing the cost and improving the accuracy of the Hessian estimate for metrics with wide receptive fields; the paper shows only 18 % (LPIPS) and 0.3 % (DISTS) of Hessian-row energy stays inside an 8×8 tile, so the block-diagonal assumption is weakest exactly there.
• Exploring larger or adaptive tile sizes, or structures between diagonal and block-diagonal, since the estimator quality and the locality assumption depend heavily on the metric.
• Extending beyond all-intra coding; the experiments use only VTM 23.8 all-intra with QP in {22, 27, 32, 37}, and temporal prediction raises questions about whether the Hessian should be recomputed per frame and how errors propagate.
• Better probe budgets and variance reduction, given that estimation takes 1–10 s per image at m = 256 and that m = 64 already retains at least 83 % of the gain; the paper does not report results for the probe counts between the measured 16, 32, 64, 128, and 256 beyond the saturation behavior.

Target Audience

• Codec and compression engineers working on RDO, rate control, or perceptual quality inside VVC, HEVC, or similar hybrid codecs.
• Researchers in perceptual image and video quality interested in how full-reference metrics behave locally and how their Hessians can be used.
• Graduate students and applied scientists in image processing and optimization comfortable with Hessian/Gauss-Newton approximations and automatic differentiation.
• Practitioners in streaming and media delivery who need to hit perceptual quality targets at fixed bitrates without touching decoders.

Authors’ abstract

Block-based video codecs select coding parameters based on the input by optimizing a rate-distortion trade-off. The conventional distortion choice, the sum of squared errors (SSE), simplifies parameter selection: the SSE is the sum of block-wise SSEs, so rate-distortion optimization (RDO) can treat blocks independently. Alternatively, full-reference image quality assessment (FR-IQA) metrics such as MS-SSIM or LPIPS often align better with the human visual system than SSE, but they cannot be used in-loop: they do not decompose block-wise and typically require the fully decoded image as input. Building on existing results in metric quadratization, we approximate a broad class of FR-IQA metrics by an input-dependent quadratic distortion (IDQD), whose quadratic form matrix is derived from the Hessian of the metric evaluated at the source video. To make the distortion computable block-wise, we propose two approximations of the Hessian matrix: 1) keeping the block-diagonal, and 2) keeping only its diagonal. We propose estimators for both that require only matrix-vector products with the Hessian obtained by automatic differentiation. Across five metrics for Kodak and CLIC in VVC, IDQD-RDO achieves 14.2-36.7 % BD-rate savings under the target metric with no decoder changes and incurs 10-30 % encoding complexity overhead.