Conformal Calibration Transfer

AI summary

Overview

Research area: Conformal prediction, distribution shift, and cross-domain calibration transfer in machine learning.

Technical level: Intermediate (the core construction is intuitive; the impossibility and coverage theorems in Section 3.5 require familiarity with conformal prediction and importance weighting).

Scope: The paper formalizes "conformal calibration transfer" — producing valid target-domain prediction sets when labeled calibration data exist only in a source space linked to the target by unlabeled paired observations — and proposes a transport-plus-correction framework (TCC) with two label-free correction mechanisms and accompanying coverage theory.

What This Paper Is About

Standard conformal prediction guarantees that prediction sets contain the true label with probability at least 1 − α, but only if the labeled calibration data are exchangeable with deployment data. This paper targets a case where that fails: labeled examples exist only in a source space (for example, a legacy sensor or one imaging modality), yet prediction sets are needed in a different target space (an upgraded sensor, a different modality), with the two spaces linked only by unlabeled paired observations of the same underlying instances. The goal is to transfer calibration across that gap and still deliver target-domain coverage without ever collecting a labeled target calibration set.

Key Contributions

1. Formalizes conformal calibration transfer. A new setting where the inputs, not just the distributions, differ between calibration and deployment: labeled source calibration $\mathcal{C}_s$, a fixed target predictor $g_t$, and unlabeled paired data $\mathcal{U} = {(x_s^{(j)}, x_t^{(j)})}$, but no labeled target calibration.

2. Proposes Transported Conformal Calibration (TCC). A transport map $f: \mathcal{X}_s \to \mathcal{X}_t$ is learned from the unlabeled pairs, labeled source calibration is mapped into the target space, and residual post-transport mismatch is corrected using only unlabeled target inputs. Two complementary corrections are introduced: TCC-KS, which uses a label-free uncertainty surrogate and a one-sided Kolmogorov–Smirnov discrepancy to inflate the level to $\alpha^\star = \max(0, \alpha - \delta^+)$, and weighted-TCC, which performs transport-then-reweight using density ratios between transported and real target inputs.

3. Provides theory, including an impossibility result. Theorem 3.1 shows that without additional structural assumptions, any procedure guaranteeing distribution-free target coverage using only unlabeled target inputs must be near-vacuous (worst-case expected set size at least $(1-\alpha)|\mathcal{Y}|$). Theorem 3.2 gives a high-probability coverage bound for TCC-KS of $1 - (\alpha^\star + \delta^+ + \varepsilon)$ under an explicit "approximate surrogate control" assumption, and Proposition 3.3 gives oracle-weight validity for weighted-TCC under covariate shift.

4. Delivers deployable, label-free diagnostics. The KS inflation term $\delta^+$ and the effective sample size (ESS%) of the residual weights can be computed before deployment and indicate both whether correction is needed and which correction to trust — validated across 328 configurations on CIFAR-100-C, Tiny-ImageNet-C, and SEN12MS (SAR → RGB).

Main Findings

• Transport alone is often sufficient on average, but breaks under severe shift. Averaged over 20 corruption conditions on CIFAR-100-C, plain Transported CP matches the target-labeled Oracle almost exactly (0.89 coverage at α = 0.1, versus 0.89 for Oracle) with comparable set sizes. In the severe-shift stress test (shot noise and motion blur at severities 4–5), Transported CP and weighted-TCC slip below nominal (0.88 at α = 0.1), while TCC-KS restores coverage to 0.91.

• TCC-KS acts as a reliable safety margin at moderate cost. It consistently over-covers relative to nominal — 0.92 on CIFAR-100-C and 0.93 on Tiny-ImageNet-C at α = 0.1 — while keeping set sizes well below the no-transport weighted baseline (WCP). On SAR → RGB it reaches 0.99 coverage with sets of average size ≈ 2.51, which is non-vacuous; WCP reaches 1.00 but with larger sets (2.87).

• weighted-TCC is the efficiency option when weights are stable. On CIFAR-100-C with the KL-aligned transport at epoch 150, weighted-TCC matches Transported CP's set size (37.15 versus 37.18) at the same 0.89 coverage, making it essentially free when the residual shift is small and weights are well-behaved.

• The label-free diagnostics predict behavior. As transport quality improves, $\delta^+$ falls and ESS% rises. On Tiny-ImageNet-C, adding the predictive KL term and training to epoch 150 drops $\delta^+$ from 0.1006 to 0.0148 and raises ESS% from 48.4 to 91.8 — exactly the regime where weighted-TCC becomes both efficient and near nominal (0.900 coverage, size 69.71), while TCC-KS remains conservative (0.935 coverage, size 86.33).

• Diagnostics also flag when to distrust a method. On SAR → RGB, $\delta^+$ stays large even at epoch 150 with KL alignment (0.3200), and correspondingly Transported CP and weighted-TCC can under-cover while TCC-KS keeps acting as a guardrail — a case where the diagnostic correctly signals persistent residual mismatch.

• Predictive alignment during transport learning helps both mechanisms. Adding a label-free KL term that matches the target predictor's output distribution on paired inputs reduces measured mismatch and stabilizes weights, improving both TCC-KS and weighted-TCC.

• A fundamental limit motivates the assumptions. The impossibility result makes clear that no method can be simultaneously label-free, distribution-free valid, and informative in the worst case — so TCC's value is in making the required structural assumption explicit (Assumption A2), providing a diagnostic that assesses when it is plausible, and degrading gracefully when it is not.

• Correction is one-sided. TCC-KS only targets under-coverage caused by the target domain being "harder" than the transported one; the opposite deviation means transported calibration is already conservative and needs no adjustment.

Methodology in Plain English

The approach has three stages.

Stage 1 — Learn a transport map. From the unlabeled paired observations, train a mapping $f$ that turns a source input into its target-space counterpart. This can be a plain reconstruction-style objective, or additionally include a KL term that encourages the target classifier to produce similar softmax outputs on $f(x_s)$ and on the true paired $x_t$. That extra term is label-free, since it only uses the classifier's outputs.

Stage 2 — Transport the labeled calibration. Push the labeled source inputs through $f$, keeping their labels, to obtain pseudo-target calibration examples $\widetilde{\mathcal{C}}_t = {(f(x_s^{(i)}), y^{(i)})}$. Compute nonconformity scores on these (using the standard least-ambiguous-classification score $1 - \hat{p}_t(y \mid x)$) and take the usual split-conformal quantile. This is the naive "Transported CP" baseline.

Stage 3 — Correct the residual mismatch without target labels. Transport is imperfect, so the transported inputs $f(X_s)$ do not exactly match real target inputs $X_t$, which can shift the score distribution and cause under-coverage. Two corrections operate on this residual gap:

• TCC-KS picks a surrogate statistic $T$ that needs no labels, such as $T(x) = 1 - \max_y \hat{p}_t(y \mid x)$. It compares the empirical distributions of $T$ on real target inputs versus transported inputs using a one-sided KS gap, inflates that gap by a DKW finite-sample bound to get $\delta^+$, and then runs conformal calibration at the tightened level $\alpha^\star = \max(0, \alpha - \delta^+)$. Small mismatch leaves the procedure essentially at nominal; large mismatch deliberately makes it conservative.

• weighted-TCC reweights the transported calibration scores toward the real target distribution, estimating the density ratio $p_{X_t}(x) / p_{\tilde{X}_t}(x)$ from unlabeled samples (e.g., via a domain classifier). Because the residual shift after transport is usually smaller than the original source-target gap, these weights are more tractable than direct source-to-target reweighting — and this is the point of doing reweighting after transport rather than before.

Sample splitting or cross-fitting keeps the transport learning, the KS estimate, and the weight estimate on disjoint data, which the coverage guarantee requires.

Why This Matters

Impact on research. The paper opens a distinct problem setting in the conformal prediction literature: previous shift-aware conformal work (importance-weighted CP under covariate shift, online/adaptive conformal) assumes calibration and deployment share a single input space. Here the spaces themselves differ, with paired observations as the bridge. The impossibility theorem sharpens the discussion by clarifying that label-free validity requires some structural assumption and by naming exactly which one TCC uses.

Real-world applications:

• Sensor and hardware upgrades. A perception system calibrated on a legacy sensor deployed on upgraded hardware, where co-registered or synchronized captures from both sensors are easy to collect but expert labels are not.
• Medical imaging across modalities or scanners. Transferring calibrated prediction sets from one acquisition modality or scanner to another using co-registered scans, where re-annotation by radiologists is slow and expensive.
• Satellite remote sensing. The paper's own SEN12MS experiment moves from SAR to RGB optical imagery — relevant for disaster response and cloud-covered regions where optical data are unavailable but SAR is.
• Thermal-to-RGB and simulation-to-reality perception. Robotics and automotive systems that must transfer calibration between camera types or from a simulator to a real vehicle using paired logs.

Industry relevance. Maintaining a separate labeled calibration set for every deployment distribution is a real operational cost in regulated and safety-critical domains. TCC reduces this to collecting unlabeled pairs plus running a cheap diagnostic, which fits naturally into MLOps pipelines. The label-free diagnostics ($\delta^+$ and ESS%) are particularly practical: they let an engineering team decide in advance whether to fall back to a conservative setting or trust a more efficient one, without needing target labels to make that call. The explicit conservative/efficient trade-off between TCC-KS and weighted-TCC maps well onto risk-sensitive deployment decisions.

Future Directions

• Weakening or testing the surrogate control assumption. Assumption A2 ($\delta_S \le \delta_T + \varepsilon$) is what makes nontrivial label-free guarantees possible. How to verify it in practice, or replace it with weaker conditions, is the main open theoretical question the paper itself raises.
• Extending beyond classification. The framework is instantiated with classification scores; regression, segmentation, detection, and other structured outputs each need their own surrogate statistics and mismatch certificates.
• Handling imperfect pairing. The method assumes clean paired observations linking the two spaces. Noisy, partial, or temporally misaligned pairs — common in real sensor transitions — are not addressed.
• Interpolating between the two corrections. TCC-KS is conservative, weighted-TCC is efficient but fragile when weights concentrate. A principled way to select or blend them adaptively — beyond using ESS% as a manual signal — could capture the best of both.
• Online and drifting targets. Combining TCC with online conformal procedures would address settings where the target distribution itself keeps changing after deployment.

Target Audience

Machine learning researchers working on conformal prediction, uncertainty quantification, and distribution shift will find the theoretical framing (especially the impossibility result and the explicit role of structural assumptions) most valuable. Practitioners deploying models across sensor changes, modality switches, or hardware upgrades — in medical imaging, remote sensing, robotics, and autonomous systems — will benefit from the practical recipe and the deployable diagnostics. Engineers building calibration pipelines for safety-critical systems are a natural secondary audience, since the method reduces the labeling burden of maintaining per-deployment calibration sets.

Authors’ abstract

Conformal prediction converts point predictions into set-valued predictions with coverage guarantees under exchangeability between calibration and deployment data. We study conformal calibration transfer, where this requirement fails because labeled calibration is available only in a source space, while prediction sets are needed in a target space linked to the source through unlabeled paired observations (e.g., paired modalities or sensor changes). We propose Transported Conformal Calibration (TCC): we transport labeled source calibration into the target space using the paired data, and then correct residual post-transport mismatch using only unlabeled target inputs. We instantiate this correction with two complementary methods: TCC-KS, which uses a label-free uncertainty surrogate to detect mismatch and adjust calibration conservatively, and weighted-TCC, which reweights transported calibration toward the target domain for improved efficiency when weights are stable. We provide finite-sample target-domain coverage guarantees that adapt to an observable measure of mismatch. Across CIFAR-100-C, Tiny-ImageNet-C, and SEN12MS, we show reliable target-domain coverage transfer without labeled target calibration data, with label-free diagnostics that predict when correction is needed.