JLD: Perceptual Distance Through A Jacobian Lens

AI summary

Overview

Research area: Computer Vision — full-reference perceptual image and video quality assessment (perceptual distance metrics).

Technical level: Intermediate. The paper assumes familiarity with vision encoders (ViT), Jacobians, and benchmark correlation metrics (SRCC), but its core idea — weighting early features by the encoder's own output sensitivity — is explained geometrically and can be followed without deep background.

Scope: The paper introduces the Jacobian Lens Distance (JLD), a label-free perceptual distance derived from the output Jacobian of a frozen DINOv2-S encoder, and evaluates it against 15 full-reference distances on image benchmarks, resolution rescoring, cost, and video.

What This Paper Is About

Measuring whether two images "look" different to a person is fundamental to compression, restoration and generation, but pixel error such as MSE or PSNR treats all pixel changes equally while human vision does not. The most accurate perceptual distances are fitted to human judgments, which ties them to a fixed dataset and resolution — for example, when TID2013 is rescored from 256 to 512 pixels, DISTS drops from 0.815 to 0.717 SRCC. The paper's goal is a perceptual distance whose geometry comes from a frozen vision encoder rather than human labels, and which stays stable when image resolution changes.

Key Contributions

1. A label-free perceptual distance from encoder sensitivity. JLD derives its geometry from the output sensitivity of a frozen vision encoder (DINOv2-S), not from human labels. Its "lens term" defines a pseudometric whose local form is a pullback metric in pixel space.

2. Direct geometric analysis on real images. The authors show the iso-distance contours of JLD agree with the ellipses predicted by the encoder, and that its sensitivity decreases with local contrast (masking, SRCC −0.86 against local contrast).

3. State-of-the-art evaluation against 15 full-reference distances. JLD achieves the best mean SRCC over the first four databases among all evaluated distances, beats every learned deep distance, and remains stable as resolution doubles. JLD-fast is four times faster than LPIPS-VGG.

4. Transferability. A single fitted lens transfers across 14 frozen encoders (including DINOv3, CLIP and VGG16), improving over raw features from the same block in all 14 cases, and the construction extends to video via a frozen VideoMAE-Base encoder.

Main Findings

• State-of-the-art mean correlation: JLD reaches a mean SRCC of 0.926 over TID2013, CSIQ, LIVE and KADID-10k test, versus 0.915 for VSI, 0.913 for DeepWSD, 0.905 for DeepDC and 0.900 for DISTS. Every gap has a 95% interval over resampled references that excludes zero.

• Beats human-fitted deep distances without human labels: JLD exceeds every learned deep distance (LPIPS-VGG, DISTS, PieAPP, DreamSim) on all four datasets, and is first or second on each dataset. On PIPAL validation it reaches 0.624, above LPIPS-VGG and every classical distance.

• Pairwise agreement: Across 699,534 same-reference pairs with different human scores, JLD places the preferred image closer to the reference in 90.0% of cases, ahead of DeepDC (89.4%) and DISTS (88.4%). Among the 24,623 pairs whose PSNR differs by less than 0.25 dB, where PSNR is at chance (48.7%), JLD agrees with people on 76.0%, compared with 74.4% for DISTS and 65.3% for LPIPS-VGG.

• PSNR blind spot quantified: Across all TID2013 pairs with equal PSNR, PSNR selects the human-preferred image in only 51.1% of cases, whereas JLD does so in 79.9%.

• Resolution stability: From 256 to 512 pixel short side, the TID2013 lens-term correlation goes 0.850, 0.860, 0.845, while unweighted block-1 features fall from 0.784 to 0.656, DISTS from 0.815 to 0.717 and LPIPS-VGG from 0.757 to 0.661. On CSIQ the lens rises from 0.923 to 0.958.

• Choosing the right directions matters more than how many: All 384 raw block-1 features give a four-set mean of 0.772, and 64 random directions give the same value; 64 PCA directions reach 0.851, while the 64 lens directions reach 0.905. Adding the chroma pre-filter and output term raises the mean to 0.911 and then 0.926.

• The lens is strongly anisotropic: For block 1 of DINOv2-S, only 64 of its 384 eigendirections account for 76.5% of its trace.

• Geometry is inspectable: At least 89% of pixel directions leave the lens term unchanged to first order at every image, since rank G(x) ≤ 64N while the image contains 588N pixel values. Around TID2013 image I19, MSE produces nearly circular contours (radius ratio 1.06), whereas JLD produces ellipses 2.33 times longer along chroma than blur; LPIPS-VGG and DISTS give ratios of 1.61 and 1.64. The restricted pullback has anisotropy 2.07, its fixed axis lies 3° from blur, and its ellipse matches the innermost contour within 9.2% in radius.

• Equal pixel error hides large perceptual differences: Across three rings of equal MSE on the Figure 2 planes, JLD varies by factors of 2.4, 1.5 and 4.4. At level 5 on I19, JPEG and colour noise have nearly identical PSNR (21.6 and 21.3 dB) but very different MOS values (1.66 and 4.47); the lens retains 14.3% of the JPEG displacement but only 6.4% of the noise displacement, making JLD four times larger for JPEG (0.516 versus 0.129).

• The lens discards most feature motion: Across all 120 distortions of I19 the lens retains 9.6% of the displacement energy; across all 24 distortion types the median is 10.6%, well below the isotropic share of k/d = 64/384 = 16.7%.

• Cost: JLD costs 118.5 GFLOPs and 12.2 ms per pair at 512×384 on an A100; JLD-fast costs 10.7 GFLOPs and 2.5 ms, 4.3 times faster than LPIPS-VGG (10.9 ms) and 4.0 times faster than DISTS (9.9 ms). IEM requires 725.7 ms per pair at 256×256, compared with 12.2 ms for JLD.

• Fit is cheap and stable: The lens is fitted once from 100 unlabeled DIV2K images (four 224×224 crops each, eight probes per crop) in about 35 seconds on one A100 GPU. Independent refits with fresh crops and probes recover the same subspace with overlap 0.962.

• Video extension: JLD-video uses a 16-direction lens on block 2 of a frozen VideoMAE-Base fitted on 50 unlabelled clips. On 240 Waterloo IVC 4K pairs it reaches 0.786 SRCC versus 0.694 for MS-SSIM, 0.611 for VMAF and 0.562 for PSNR, a 0.175 lead over VMAF. On 120 held-out AVT-VQDB-UHD-1 pairs it reaches 0.912, within 0.020 of VMAF (0.932), which is trained on human video scores.

• Human study confirms disagreement cases: In a ten-person study on trials where methods disagree, participants side with JLD on 24 of 28 decided trials and with DISTS on 14 of 26.

Methodology in Plain English

The researchers start from a frozen DINOv2-S (ViT-S/14) vision encoder pretrained without any quality judgments. They take the patch tokens after block 1 (384-dimensional vectors, with a patch size of 14 pixels), because after one attention block each patch has interacted once with its neighbours, retaining local structure while incorporating context.

Their key move is to ask how strongly the encoder's final output responds to small changes in those early tokens. That relationship is the Jacobian. Rather than forming full Jacobian matrices, they backpropagate a random Gaussian probe on the output, which gives — for every patch in one backward pass — an unbiased sample of the squared Jacobian response. Averaging over 100 unlabeled DIV2K images, four crops each and eight probes per crop produces one fixed 384×384 sensitivity matrix, fitted in about 35 seconds.

The leading 64 eigenvectors of that matrix form what they call the Jacobian lens: a fixed orthonormal projection that retains 76.5% of the matrix's trace, mapping every image and every resolution to the same subspace. Because the lens is a fixed linear map, the geometry it induces can be measured and drawn directly on real images, independently of any benchmark.

The distance itself has two parts. The "lens term" is the root-mean-square of the projected patch-token differences between two images, after applying a fixed chroma pre-filter that blurs colour channels while preserving luminance, following S-CIELAB. The "output term" is half of one minus the cosine between the final class tokens, catching whole-image changes such as global colour shifts that spread too thinly over patches. A fast variant stops the encoder after block 1; a linear variant applies the lens to the plain patch embedding. Scoring needs two forward passes, no learned parameters beyond the frozen encoder and fixed lens, and is fully differentiable, so JLD can also serve as a loss.

Human data enter only in one place: choosing among 245 configurations on CSIQ and KADID-10k development splits. LIVE, KADID-10k test references and PIPAL validation are held out.

Why This Matters

Perceptual metrics are currently split between hand-designed early-vision models that are interpretable but less accurate, and learned deep distances that are accurate but fitted to human labels and brittle to resolution change. JLD shows that a network which has only learned to see already contains a useful perceptual geometry, and that this geometry can be read out and inspected through contour maps and per-patch lens maps whose root mean square equals the lens term. Because the fit takes seconds and requires no labels, the construction can be transferred to a new encoder, domain or modality without collecting human ratings.

Real-world applications:

• Image and video compression: perceptual metrics are used to tune lossy codecs; a label-free metric that is stable across resolutions and extends to video could reduce dependence on expensive subjective testing.
• Restoration and generative model training: JLD is differentiable, so it can be used as a training loss where a well-defined, interpretable geometry matters.
• Evaluating generative and restoration outputs: its 0.624 SRCC on PIPAL's restoration and GAN outputs, above LPIPS-VGG and every classical distance, is directly relevant to judging GAN-generated content.
• Quality assessment across display resolutions: the lens term's near-constant correlation from 256 to 512 pixels (0.850 to 0.845 on TID2013) targets a known failure mode of label-fitted metrics.
• Content platforms and streaming: the video result on Waterloo IVC 4K and AVT-VQDB-UHD-1, and JLD-fast at 2.5 ms per pair, are relevant to large-scale quality monitoring.

Industry relevance: The work is co-authored from Google (Balu Adsumilli) alongside UT Austin and the University of Colorado Boulder, and it targets exactly the metric bottleneck that compression, streaming and generative imaging pipelines face.

Future Directions

• Extending beyond vision: the authors state that any differentiable encoder, including those for audio, medical imaging or 3D, can in principle define such a lens, leaving that transfer untested here.
• Improving JLD as a training loss: the discussion suggests that closing the lens's null space with the full metric tensor could further strengthen its use as a loss; the paper reports that treating the selected directions equally outperforms weighting them by the metric tensor (0.916 versus 0.897).
• The output term's metric-theoretic status: the lens term satisfies the triangle inequality on all 171.4 million same-reference triplets examined, but the full distance violates it on at most 107 triplets per million, with a maximum violation of 3.6%. The paper notes the output term need not satisfy the triangle inequality and verifies empirically that violations are rare, rather than proving a bound.
• Video scope: only one video lens configuration is reported (a 16-direction lens on block 2 of VideoMAE-Base, fitted on 50 unlabelled clips), leaving open whether the depth and rank choices that worked for images carry over systematically to the video setting.

Target Audience

Researchers and engineers working on perceptual image and video quality assessment, learned perceptual metrics, and image/video compression and restoration. It is also relevant to practitioners building generative or restoration models who need a differentiable perceptual loss, and to anyone interested in the geometry of representations in frozen vision encoders.

Authors’ abstract

Image compression, restoration, and generation all require a way to measure how different two images look to a person. Pixel error ignores how people see, while the most accurate perceptual distances are typically fitted to human judgments, tying them to a fixed data and resolution. For example, when image resolution is doubled, the correlation of DISTS with human scores on TID2013 drops from 0.815 to 0.717. We introduce the Jacobian Lens Distance (JLD), which derives its perceptual geometry from a frozen vision encoder rather than from human labels. JLD combines the locality of early patch features with the perceptual sensitivity captured by later encoder representations. Specifically, we use the encoder Jacobian to identify directions in the early feature space that most strongly affect the encoder output, producing a fixed metric tensor, $E[J^\top J]$, which we call the Jacobian lens. The lens is fitted only once from 100 unlabeled images, taking about 35 seconds. Locally, this construction defines a pullback metric in pixel space, giving JLD a clear geometric interpretation that can be directly analyzed on real images. Across four standard perceptual databases, JLD achieves state-of-the-art performance and consistently outperforms LPIPS, DISTS, PieAPP, and DreamSim. JLD is also robust to changes in image resolution, on TID2013, its lens-term correlation remains nearly unchanged when the resolution is doubled, decreasing only from 0.850 to 0.845. We further introduce JLD-fast, which is $4\times$ faster than LPIPS-VGG while achieving a mean correlation of 0.911. Finally, JLD naturally extends to video, reaching a correlation of 0.786 on Waterloo IVC 4K compared with 0.611 for VMAF.