A Controlled Evaluation of Curvature Inspired Scalar Feature Scaling for Domain Generalization

AI summary

Overview

Research area: Domain generalization (DG) in machine learning, specifically the empirical evaluation of a curvature-inspired training method against a controlled baseline.

Technical level: Intermediate. The paper is readable without deep mathematics, but it assumes familiarity with terms such as ERM (empirical risk minimization), BatchNorm, leave-one-domain-out evaluation, and model-selection protocols.

Scope: A six-condition, four-dataset, four-target, four-seed controlled study (384 runs) that audits and evaluates one narrow scalar feature-scaling implementation, called HyperDG, against native DomainBed ERM, and does not test genuine hyperbolic representation learning.

What This Paper Is About

The paper examines a method that claims inspiration from domain-specific curvature in domain generalization, where a model trains on several source environments and is tested on an unseen target environment. Rather than implementing curved geometry, the code under audit multiplies Euclidean encoder features by learned positive scalars during training and then predicts with unscaled features and an ordinary linear classifier. The authors' goal is to specify exactly what the implementation computes, to test whether the full package beats a matched ERM baseline, and to test whether learning three separate source scales beats learning one shared scale.

Key Contributions

1. An implementation-level specification that separates the intended geometric interpretation from the executed computation, documenting an inactive local-gradient term, an algebraically scale-cancelling norm proxy, and a train-inference scaling mismatch.
2. A completed six-condition, four-dataset, four-target, four-seed evaluation (384 of 384 planned runs), including native DomainBed ERM and a fully replicated shared-learnable condition.
3. Source-only model selection with target-clustered uncertainty, reporting every named target and run seed separately rather than treating seed-by-target cells as independent unit-level observations.
4. A narrow negative conclusion: under the tested protocol, the full package shows no adjusted evidence of improvement over ERM, and separate source scales are macro-equivalent to one shared learned scale within a stated revision-stage margin.

Main Findings

• The method is not hyperbolic. In this paper, HyperDG denotes the exact scalar-scaling formulation described in Section 3: a Euclidean feature h_θ(x) is divided by sqrt(c_i) during the training loss, where c_i = softplus(a_i) + 10^-4 is indexed by a source-loader slot. Prediction uses the unscaled Euclidean feature. The paper states it implements no Lorentz or Poincaré representation, no geodesic prediction, and no Sharpness-Aware Minimization inner loop.

• Full package versus ERM is negative but statistically unconfirmed. The equal-dataset macro estimate for FullPerDomain minus StandardERM is -1.61 percentage points, with a 95% CI of [-3.17, -0.04]. All four dataset estimates favor ERM: -1.01, -0.77, -1.69, and -2.96 percentage points on PACS, VLCS, OfficeHome, and TerraIncognita respectively. The primary four-dataset exact sign-flip test gives p = .125, Holm-adjusted p = .250, so the authors claim no adjusted evidence of a difference and also do not claim equivalence.

• Separate source scales do not beat one shared scale. The cleaner contrast, NoVariance minus SharedLearnable, is -0.13 percentage points on the suite macro with a 90% CI of [-0.43, 0.17]. It passes a revision-stage ±1 percentage point TOST equivalence test (p = .003) and also sits inside the ±0.5 percentage point sensitivity margin (TOST p = .031). Per-dataset effects are -0.27, -0.39, +0.19, and -0.05 percentage points.

• Accuracy table. Source-validation-selected target accuracies (in percent) were: StandardERM 85.88 (PACS), 77.19 (VLCS), 70.21 (OfficeHome), 47.13 (TerraIncognita); MatchedControl 84.22, 75.38, 70.05, 44.14; FullPerDomain 84.87, 76.42, 68.52, 44.17; Frozen 86.14, 75.84, 67.81, 44.20; NoVariance 85.28, 76.17, 68.36, 43.70; SharedLearnable 85.55, 76.56, 68.17, 43.75.

• Auxiliary terms are partly inert. The local-gradient regularizer is identically zero in every primary-run log because autodiff returns None from a re-indexed view and it is converted to zero. The term presented as a norm/sharpness proxy collapses algebraically to the variance of raw feature norms and therefore does not depend on c_i, so it is not curvature-sensitive and is not SAM.

• Checkpoint endpoint matters. Source selection improves over the update-4999 endpoint in 20 of 24 condition-by-dataset cells, by 0.97 percentage points on average. At update 4999, NoVariance minus SharedLearnable is -0.32 percentage points with a 90% CI of [-1.13, 0.50] and TOST p = .072, so equivalence is not established there. The equivalence conclusion is restricted to the declared source-selected endpoint.

• The ERM comparison is not a clean curvature-only test. HyperDG calls the feature extractor six times per update (three objective forwards plus three no_grad() logging forwards that still run in training-mode BatchNorm), while native ERM concatenates the three source minibatches and runs one forward pass over 96 examples. The FullPerDomain minus StandardERM contrast therefore estimates the whole implementation including this BatchNorm-update asymmetry.

• Relative to the matched control, the pattern is mixed. FullPerDomain changes the four dataset means by +0.65, +1.04, -1.53, and +0.03 percentage points relative to MatchedControl.

Methodology in Plain English

The researchers took a single, deliberately narrow implementation and audited what its code actually does, then evaluated it under a tightly matched protocol. Six conditions were run: native DomainBed ERM; a matched control that uses the same update route with fixed scales and all auxiliary coefficients set to zero; the full method with three learned local scales and one learned global scale; a frozen-scale variant; a no-dispersion-penalty variant with three learned local scales; and a shared-learnable variant with one learned scale across all source loaders.

Data came from four benchmarks chosen for different shift types: PACS (depiction and style), VLCS (acquisition and dataset source), OfficeHome (depiction plus object context), and TerraIncognita (camera location and background). Each environment was held out once and the other three served as sources, with deterministic 80/20 splits, four seeds (0–3), a timm ResNet-50 backbone with a 2,048-dimensional feature, source batch size 32, Adam at a network learning rate of 5 × 10^-5 with zero weight decay, 5,000 updates, and evaluation every 200 updates plus update 4999. Learned scales used a separate Adam optimizer at 10^-4, and the HyperDG route clipped gradients to norm 5.

Checkpoints were chosen using only the three source-validation environments, never the held-out target. Statistics were computed by averaging seeds within a target, averaging targets within a dataset, and treating the four dataset means as the clusters. The primary procedure was an exact two-sided sign-flip test over all 2^4 = 16 target sign assignments, with Holm correction across the two primary contrasts; a t_3 interval and a four-cluster bootstrap were secondary working-model summaries. Practical equivalence used TOST with a revision-stage smallest effect of practical interest of ±1 percentage point, with ±0.5 and ±2 point sensitivities.

Why This Matters

Impact on research. The paper argues that "curvature learning" can denote substantially different computations, and that scalar Euclidean rescaling should not be evaluated or described as if it were interchangeable with manifold-valued hyperbolic learning. It provides a bounded negative result for a specific scalar training package, and explicitly states that it does not test or refute genuine hyperbolic representation learning. It also documents the methodological point that with only four dataset clusters, the smallest achievable two-sided exact p-value is .125, which constrains what claims a four-benchmark suite can support.

Real-world applications (framing from the shift types the benchmarks represent, not results reported in the paper):

• Medical imaging, where a model trained at some hospitals must work at another with different scanners or acquisition protocols.
• Wildlife monitoring with camera traps, where the shifted variable is camera location and background.
• Photo or artwork classification, where the shift is artistic style and depiction.
• Consumer or industrial image collections spanning different sources and object contexts.

Industry relevance. Teams deploying vision models across changing acquisition conditions, cameras, or customer environments face exactly the source-to-target shift this paper studies. The paper's practical message is that a small training-time feature rescaling delivered no adjusted evidence of gain over a matched ERM baseline in this protocol, while the surrounding implementation details (BatchNorm update routes, model selection, inference-time scaling) can dominate or confound an apparent effect.

Future Directions

• Replace the Euclidean head with a genuine manifold-valued one. The authors propose keeping the present splits, seeds, backbone, budget, and source-only selector while implementing a correctly specified Poincaré or Lorentz representation with a hyperbolic classifier.
• Separate manifold geometry from scalar flexibility. A fixed-versus-learned-curvature comparison inside a real hyperbolic network would test what this scalar implementation cannot.
• Repair and retest the audited mechanics. The inactive local-gradient term, the algebraically scale-cancelling norm proxy, and the train-inference scaling mismatch are all open implementation questions the authors flag rather than resolve.
• Rethink inference for small benchmark suites. The paper notes the coarseness of exact tests with four target clusters and four dataset clusters, and leaves open how to reach stronger claims within such a fixed, heterogeneous suite.

Target Audience

Researchers and practitioners working on domain generalization, robustness, and out-of-distribution evaluation; methodologists interested in controlled benchmarking and implementation auditing; and readers assessing claims about hyperbolic or curvature-based representation learning, who need to know which computations the term covers in a given paper.

Authors’ abstract

Domain-specific curvature is an appealing idea for domain generalization (DG), but curvature learning can denote substantially different computations. This paper evaluates one deliberately narrow implementation: the code calls it HyperDG, but it rescales Euclidean encoder features during training by learned positive scalars and uses an ordinary linear classifier on unscaled Euclidean features at inference. It does not implement a Lorentz or Poincaré representation, geodesic prediction, or a Sharpness-Aware Minimization inner loop. We audit the executed objective and evaluate six conditions on PACS, VLCS, Office Home, and TerraIncognita. The completed design contains 384 runs: six conditions, four held-out targets, and four run seeds per dataset. Checkpoints are selected using only the three source-validation environments. Inference first averages paired seed differences within each held-out target; targets, rather than 16 seed-by-target observations, are the per-dataset clusters. The full implementation's equal-dataset macro point estimate is 1.61 percentage points below native DomainBed ERM (95% clustered CI [-3.17, -0.04]). All four dataset point estimates favor ERM, but an exact four-dataset sign-flip test gives p = .125 and Holm-adjusted p = .250; we therefore claim no adjusted evidence of a difference. The cleaner domain-specificity contrast, separate learned source scales without their variance penalty versus one shared learned source scale, is -0.13 pp (95% CI [-0.53, 0.27]). Its 90% CI lies within a revision-stage $\pm 1$ pp smallest effect of practical interest (TOST p = .003), supporting equivalence only for the source-selected suite macro, not every dataset, target, or final-step endpoint. The study thus supplies a bounded negative result for this scalar training package; it does not test or refute genuine hyperbolic representation learning.