FDSML: Formula-Driven Supervised Metric Learning with Parameter-Aware Triplet Loss

AI summary

Overview

Research area: Computer vision pre-training without real images — specifically Formula-Driven Supervised Learning (FDSL) and deep metric learning, with image retrieval, classification, and corruption robustness as evaluation targets.

Technical level: Intermediate. The paper is readable with a working knowledge of contrastive/triplet loss and standard pre-training/fine-tuning pipelines; the fractal-generation background (Iterated Function Systems) is explained from scratch.

Scope: The paper reformulates formula-driven synthetic pre-training from discrete classification into parameter-aware metric learning and evaluates the resulting representations on classification, retrieval, and corruption-robustness benchmarks across ResNet-50, ViT-Tiny, and ViT-Base.

What This Paper Is About

Formula-Driven Supervised Learning generates synthetic images from mathematical formulas and uses the generation parameters as labels, so models can be pre-trained without any real photographs. The authors argue that treating this as ordinary classification throws away the continuity of the underlying parameter space, and that representations learned this way can be surprisingly weak for tasks that depend on embedding geometry, such as image retrieval. Their goal is to keep the synthetic, annotation-free setup but change the learning signal from "which category is this?" to "which of these two samples is more similar to this one?".

Key Contributions

1. FDSML (Formula-Driven Supervised Metric Learning): a pre-training framework that reformulates FDSL from discrete category recognition into relational comparison, so the model learns relative similarity orderings rather than only decision boundaries.

2. PA-Triplet (Parameter-Aware Triplet) Loss: a triplet objective where an anchor and two candidates are sampled from the continuous IFS parameter space, and the positive/negative assignment is decided by which candidate is closer in a six-dimensional parameter descriptor (the expectation of the weighted affine parameters). Discrete class labels are used only to restrict the candidate pool, not to assign positives.

3. FineFractalDB: a refined fractal dataset with 1,000 categories and 1,000 synthesized images per class (1 million samples total), built with far more parameter perturbations per class (1,000, versus the 25 perturbations used to generate 1,000 instances per class in FractalDB-1k), strict contraction constraints on the affine maps, and rejection sampling that discards samples whose occupied pixel ratio is below 0.05%.

4. Four candidate-filtering strategies (all, each, same, diff) that use class labels as a coarse structural prior to control triplet difficulty, together with an empirical validation of which regimes give reliable parameter-derived orderings.

Main Findings

• Retrieval improves substantially over classification-based FDSL. On ResNet-50, Recall@1 improves from 65.6% to 74.0% on Stanford Online Products and from 56.2% to 78.4% on Stanford Cars when moving from FDSL pre-training to FDSML.

• Classification-based FDSL can hurt retrieval. The FractalDB-1k FDSL model falls below training from scratch on SOP Recall@1 (65.6 vs. 70.4) and Recall@10 (81.3 vs. 83.1), which the authors attribute to classification objectives enforcing categorical separability without modeling relative similarity.

• Downstream classification is competitive. With ResNet-50, FDSML (same) reaches 91.47% on CIFAR-10 and 72.65% on ImageNet-1k, versus 90.50% and 70.43% for FractalDB-1k FDSL; ImageNet-1k supervised pre-training (96.96% CIFAR-10, 80.33% Places-30) remains the strongest baseline overall.

• Gains extend to transformers. With ViT-Tiny, FDSML (each) reaches 88.90% on ImageNet-100 versus 84.40% for FractalDB-1k FDSL and 98.00% on CIFAR-10, slightly above ImageNet-1k supervised pre-training (97.92%). With ViT-Base the gap narrows, but FDSML (each) still gives the highest CIFAR-10 and ImageNet-100 accuracy in that table.

• The descriptor's ordinal signal is regime-dependent. Agreement between the parameter distance ordering and the Chamfer-distance ordering is 89.2% for each, 80.6% for same, but below chance for all (47.2%) and diff (46.8%). Under each, a label-only rule reaches 89.4% and the descriptor picks the same positive in 99.6% of triplets; under same, no label rule exists (50.0% by definition), yet the descriptor reaches 80.6% — a signal labels cannot provide even in principle.

• Dataset and objective are complementary (ablation, ViT-Tiny). With FractalDB-1k fixed, swapping cross-entropy for PA-Triplet improves CIFAR-10 (96.65 → 97.77 for each), CIFAR-100 (81.18 → 81.72), and ImageNet-100 (84.40 → 87.93). Under FDSL, FineFractalDB provides no gain over FractalDB-1k and degrades CIFAR-100 (81.18 → 77.52), because cross-entropy assigns all instances in a class the same label and cannot exploit the extra intra-class variation.

• Robustness gains are architecture-dependent. On ImageNet-C with ResNet-50, FDSML beats FDSL in all five corruption groups, with FDSML (same) best in three (Noise 83.62, Weather 68.81, Digital 57.46, Geometric 59.13 vs. FDSL's 86.16, 71.94, 60.36, 61.15). For ViT-Tiny the trade-off is mixed: a 6.3% relative improvement in the Weather category (47.64 vs. 50.84) but degradation in Noise (58.69 for each vs. 55.31) and Digital (43.41 vs. 41.94).

• Qualitative differences appear in both architectures. Grad-CAM on ResNet-50 shows FDSML producing more spatially concentrated activations on object-relevant regions than FDSL, closer to an ImageNet-pre-trained model; layer-wise CLS-to-patch attention in ViT-Tiny shows FDSL concentrating on prominent boundaries while FDSML also attends to textured and interior regions.

Methodology in Plain English

The starting point is a fractal generator: a small set of affine transformations, each with a probability, that together trace out a shape when points are repeatedly pushed through randomly chosen transformations. Because the numbers defining those transformations fully determine the resulting image, you always know exactly how any two synthetic images relate in parameter space.

The authors build their dataset by generating many images per category, each with its own perturbation of the underlying parameters, and they enforce that every transformation actually contracts (so shapes stay well-formed rather than collapsing) and drop any rendering that covers too few pixels.

Then, instead of training a classifier, they train a metric learner. For each anchor image they draw two candidates and compute a compact six-number summary of each image's generating parameters, formed by weighting each transformation's affine coefficients by its selection probability. Whichever candidate has the smaller Euclidean distance to the anchor in that summary becomes the positive; the other becomes the negative. The standard triplet loss then pushes the anchor's embedding toward the positive and away from the negative.

Crucially, the parameter distance is used only ordinally — it decides which of two candidates is closer, not how much closer — so the model is not asked to match a perceptual scale. Class labels enter only as a filter on where candidates are drawn from: unrestricted (all), one same-class plus one different-class (each), both same-class (same), or both from other classes (diff). The authors check the sanity of this scheme by comparing, over many sampled triplets, whether the candidate ranked closer by the parameter summary is also closer by a symmetric Chamfer distance between rendered foreground pixels; they also verify that Chamfer distance responds monotonically to controlled sweeps of a single affine coefficient and that the parameter distance tracks it (Spearman ρ ≥ 0.94 within each sweep direction, ρ = 0.831 when pooled, against a rendering noise floor of 0.53 px).

Pre-training uses ResNet-50 (final layer replaced by a 512-dimensional linear embedding) and ViT-Tiny/ViT-Base with a 2-layer MLP projection head, then fine-tunes on classification and retrieval benchmarks, with retrieval fine-tuned using the Proxy-Anchor objective.

Why This Matters

The work challenges an assumption baked into most synthetic pre-training: that generating labels from formulas means the task should be classification. If continuous generative parameters carry richer structure than discrete categories, then how the supervision is shaped matters as much as where the data comes from. The retrieval results make the practical stake concrete — a synthetic pre-training recipe can look fine on classification accuracy while actively underperforming training from scratch on embedding-based tasks.

Real-world applications:

• Image retrieval and visual search in domains where collecting labeled real images is expensive or legally constrained — product catalogs, archival collections, e-commerce visual search.
• Privacy-sensitive pre-training, where a model must be bootstrapped without ingesting personal photographs.
• Fine-grained recognition and re-identification, which depend on subtle within-category distinctions — the regime where the same filtering strategy was most informative.
• Robustness-oriented deployment, where models must tolerate blur, weather, and other corruptions rather than only clean benchmark images.

Industry relevance: the approach is fully synthetic and infinitely scalable with no annotation pipeline, which is attractive for organizations that cannot license or annotate large image corpora. The paper also documents a caveat worth heeding: the parameter-distance signal is only trustworthy within a class, and the robustness benefit does not automatically transfer from CNNs to transformers.

Future Directions

• Extending PA-Triplet beyond fractals. The authors explicitly limit validation to IFS-based fractals and note that other FDSL generators such as VisualAtom or TileDB would each require their own similarity definition over generation parameters plus dedicated empirical validation.

• Scaling behavior. All results are confined to the one-million-image scale of FineFractalDB; whether parameter-aware supervision keeps paying off at larger scales is stated as an open question.

• Characterizing the cost. The paper notes that FDSML incurs extra overhead for descriptor computation and triplet construction relative to cross-entropy FDSL, and that this overhead has not been analyzed in detail.

• Understanding the architecture-dependent robustness trade-off. ViT-Tiny improved on weather corruptions but degraded on noise and digital corruptions; the authors suggest this relationship to high-frequency noise requires dedicated frequency analysis.

Target Audience

Researchers and practitioners working on synthetic or annotation-free pre-training, deep metric learning, and image retrieval who want to know whether formula-driven data can produce embeddings with meaningful geometry rather than just good classifiers. It is also useful for engineers evaluating cheap pre-training alternatives to ImageNet, since it reports honest failure cases (retrieval below scratch for classification-based FDSL) alongside the gains, and for anyone interested in how supervision shape — ordinal versus categorical — interacts with dataset design.

Authors’ abstract

Formula-Driven Supervised Learning (FDSL) enables vision pre-training without real images, but existing methods cast pre-training as classification on discretized labels, discarding continuity in the generative process. We propose Formula-Driven Supervised Metric Learning (FDSML), which derives relational supervision from image-generation parameters beyond discrete class labels. FDSML combines FineFractalDB, our refined fractal dataset with higher intra-class variation and more stable structures, with a Parameter-Aware Triplet (PA-Triplet) Loss that assigns positives and negatives according to distances in the underlying Iterated Function System parameter space. This objective trains embeddings to respect parameter-derived relative similarity orderings. Experiments show that FDSML improves transferability to image retrieval while remaining competitive on downstream classification and improving robustness under several corruption settings. On ResNet-50, Recall@1 improves from 65.6% to 74.0% on Stanford Online Products and from 56.2% to 78.4% on Stanford Cars over FDSL pre-training. These results suggest that parameter-derived relational supervision provides a stronger synthetic pre-training signal than label-based classification for transfer beyond recognition.