Learning Steadily: Accumulating Relative Point Margin Scores for Face Image Quality Assessment

AI summary

Overview

Research area: Computer vision and biometrics — specifically Face Image Quality Assessment (FIQA) for automated face recognition (FR), with a focus on the training dynamics of FR-integrated quality estimation.

Technical level: Advanced. The paper combines deep metric-learning losses (ArcFace-style angular margin), a theoretical treatment of cumulative averaging (variance, mean squared error, and ranking convergence), and standardized biometric benchmarking with Error-versus-Discard Characteristic (EDC) curves at fixed false match rates.

Scope: The paper introduces CARPM-FIQA, a temporal stabilization strategy that accumulates a relative point margin quality signal across an entire FR training trajectory and predicts those accumulated scores with a regression head for unseen face images. Published in arXiv:2609.31662v1 [cs.CV] 15 Sep 2026 and accepted for publication in IEEE Transactions on Biometrics, Behavior, and Identity Science (DOI: 10.1109/TBIOM.2026.3734769).

What This Paper Is About

FIQA answers a practical question for biometric systems: is this captured face image good enough for automated identity matching, for example in passport scans or live captures? The strongest current methods estimate quality from inside the face recognition network itself, but because the network's feature space keeps changing during training, a sample that looks high-quality at one epoch can look medium-quality a few epochs later — quality prediction becomes a moving target. The paper's goal is to make that quality signal temporally stable by averaging the relative point margin score, the ratio between intra-class compactness and inter-class separation, over the full training run instead of trusting any single epoch.

Key Contributions

1. A quality stabilization approach. CARPM-FIQA (Cumulative Average of Relative Point Margin for FIQA) accumulates relative point margin values across training epochs, producing temporally stable quality estimates that capture long-term discriminative properties of face images rather than a snapshot of one epoch.

2. A theoretical framework for cumulative averaging. The authors provide formal proofs and empirical validation showing that cumulative averaging reduces variance in quality estimates, improves mean squared error, and enhances ranking stability, with convergence guarantees as training progresses.

3. A regression model for generalization. A quality regression head is trained to predict the accumulated quality scores for unseen face images, extending the temporal stability patterns learned during training to new samples without requiring access to their training history.

4. Large-scale empirical validation. Evaluations against twelve FIQA methods (plus three IQA approaches) on eight benchmarks with four FR models at two FMR thresholds (1e-3 and 1e-4), where CARPM-FIQA places 4th (CARPM-FIQA (L)) and 6th (CARPM-FIQA (S)) of 17 compared methods by pAUC-EDC and AUC-EDC averaged across FR models and, after per-benchmark normalization, across benchmarks.

Main Findings

• Cumulative averaging separates quality groups better. On the SynFIQA dataset (5.5M images, 11 quality groups of 0.5M images each: Q0–Q9 degraded from lowest to highest quality, plus a Ref group of reference images), the cumulative method achieves a higher average Cohen's d between adjacent groups, 0.3112 versus 0.2514 for the non-cumulative method, on the small protocol (ArcFace/ResNet50/CASIA-WebFace).

• Cumulative targets beat non-cumulative targets on most benchmarks. In Table I (ArcFace loss, ResNet50 architecture, CASIA-WebFace training data), cumulative achieves the lowest Avg. norm. value for every FR model and on average, on both AUC-EDC and pAUC-EDC. Averaged across the six FR models, cumulative reaches an Avg. norm. of 1.045 (AUC-EDC) and 1.018 (pAUC-EDC), versus 1.121 and 1.101 for non-cumulative, and 1.413 and 1.160 for a frozen backbone with non-cumulative targets.

• The same pattern holds under a different training loss. With CurricularFace loss instead of ArcFace (Table II), cumulative again attains the lowest Avg. norm., 1.028 (AUC-EDC) and 1.017 (pAUC-EDC), versus 1.129 and 1.056 for non-cumulative.

• Two benchmarks consistently favor non-cumulative. AgeDB-30 and CALFW are the consistent exceptions where the non-cumulative method performs better; the paper reports this explicitly for the Table I comparison.

• XQLFW is the one benchmark where the frozen backbone wins. Averaged across FR models on XQLFW, the ordering is frozen best, followed by cumulative, then non-cumulative — even though frozen is generally the weakest variant on all other benchmarks.

• Non-cumulative targets degrade after a sweet spot. The epoch-wise analysis (Fig. 3) shows cumulative averaging improves estimates consistently, while non-cumulative performance degrades after epoch 20 in AUC-EDC; non-cumulative target distributions drift rightward and compress over epochs, reducing sample differentiation, whereas cumulative distributions stay more stable and well spread.

• The accumulated target is approximated early in training. Table III reports that a growing first-k-window reaches a Spearman rank correlation of ρ ≥ 0.95 with the full-trajectory target Q_T at k = 10 of T = 34 epochs (ArcFace, cumulative, small protocol) and at k = 8 of T = 18 (large protocol), and ρ ≥ 0.99 at k = 24 and k = 14 respectively. At a matched relative window length of roughly 24 percent of T (k = 8 of 34 small, k = 4 of 18 large), first-k windows correlate more with the full target (0.939 for ArcFace cumulative small) than last-k windows (0.845).

• Competitive standing among 17 methods. Across the eight benchmarks (LFW, AgeDB-30, CFP-FP, CALFW, Adience, CPLFW, XQLFW, IJB-C) and four FR models, CARPM-FIQA stays within a few percent of the best method's normalized average for every FR model.

Methodology in Plain English

The method reuses an established per-epoch quality signal — the relative point margin — and changes when and how it is read off the network.

The per-epoch signal. At epoch n, each face image is compared with the class centers stored in the classification layer. The similarity to its own identity's center is the intra-class term, and the highest similarity to any competing identity is the inter-class term. Their ratio, q_n(x_i) = cos(θ_yi) / max over y ≠ y_i of cos(θ_y), measures how well separated that sample is from competing identities at that moment. This is the quantity CR-FIQA used, formalized as Closest Class Similarity (CCS) over Nearest Non-Class Center Similarity (NNCCS), and related to the silhouette score and to the Relative Point Margin (RPM) concept shown to satisfy Kleinberg's clustering axioms.

The cumulative change. Instead of using q_n(x_i) directly, the algorithm keeps a running average. Starting from Q(x_i)_0 = 0, each epoch the score is updated as Q(x_i)n = (q_n(x_i) + (n-1) · Q(x_i){n-1}) / n. Because the feature space is a moving target, this running average acts as a filter: it damps the epoch-to-epoch fluctuations while still reflecting the long-run discriminative structure of the sample.

Training the network. A face recognition backbone and a quality regression head are initialized together. Each mini-batch, embeddings are extracted, the regression head predicts an estimated quality Q̂(x_i)_n, and the accumulated target Q(x_i)_n is computed. The total objective is the ArcFace loss plus a weighted quality regression loss, L = L_Arc + λ L_Q, where the quality loss is the mean per-sample loss ℓ between the accumulated target and the predicted score, and λ is a hyperparameter.

Generalizing to new images. After training, the regression head can predict accumulated quality scores for unseen face images directly, without needing to observe their per-epoch training history.

How it was tested. A controlled study on SynFIQA with labeled quality groups isolates whether averaging helps, and ablation studies across training configurations check consistency. The main evaluations use the AUC-EDC and pAUC-EDC metrics (lower is better) at FMR thresholds of 1e-3 and 1e-4, with per-benchmark normalization so that the best method in a comparison column equals 1.0 for the Avg. norm. column.

Why This Matters

Research impact. The paper offers a general lesson beyond FIQA: when a training objective's target values drift because the underlying representation keeps changing, aggregating those targets over the training trajectory can be a principled fix. The authors frame this as evidence that temporal aggregation strategies can stabilize training objectives in deep learning systems where targets inherently fluctuate, and they pair the empirical result with variance-reduction, MSE, and ranking-convergence guarantees. It also targets a methodological weakness in the FIQA literature: existing FR-integrated methods rely on single-epoch measurements and often select checkpoints based on small benchmark performance while reporting on larger datasets, which introduces selection bias.

Real-world applications:

• Border control and e-passport verification, where a captured face image must be judged usable before it enters an identity-matching pipeline.
• Live capture on consumer devices, where a phone or kiosk needs a fast, stable signal on whether a selfie or video frame is good enough for recognition.
• Large-scale enrolment, where unstable quality ranking makes it hard to decide which images to keep in a gallery.
• Watchlist and access-control systems, where quality thresholds interact directly with the operating false match rate — the paper evaluates at FMR 1e-3 and 1e-4.

Industry relevance. Biometric vendors and integrators care about two things the paper addresses: stability (a quality score that does not change definition as the model trains) and cost. Because CARPM-FIQA is FR-integrated, quality comes from the same forward pass as recognition, avoiding the multiple inference passes or backpropagation required by pre-trained FR analysis approaches such as SER-FIQ and GraFIQs, and the heavy generative models used by DifFIQA.

Future Directions

• Shrinking the accumulation requirement. Table III shows a first-k-window reaches ρ ≥ 0.95 with the full-trajectory target at k = 10 of 34 epochs (small protocol) and k = 8 of 18 (large protocol), and first-k windows correlate better with the full target than last-k windows. How short the accumulation window can be pushed without losing the gains is left open.

• Understanding the exceptions. Non-cumulative performs better on AgeDB-30 and CALFW, and the frozen backbone is best on average on XQLFW. What property of these benchmarks inverts the ordering is not resolved by the reported results.

• Extending temporal aggregation beyond FIQA. The paper's broader claim invites testing cumulative averaging on other deep learning objectives whose targets drift with the representation, rather than only on face quality.

• Clarifying the small versus large configuration trade-off. The abstract reports different overall ranks for CARPM-FIQA (L) (4th) and CARPM-FIQA (S) (6th), and Table IV shows the two variants trading places across FR models — for example, Avg. norm. 1.084 (L) versus 1.103 (S) under ArcFace, and 1.136 (L) versus 1.132 (S) under MagFace. A systematic account of when the larger variant pays off is not provided in the content available.

Target Audience

Biometrics and face recognition researchers, particularly those working on FIQA, FR-integrated quality estimation, and evaluation with EDC-style metrics at fixed FMR thresholds. It also suits machine learning researchers interested in training dynamics and in stabilizing objectives whose targets move during optimization, plus graduate students and industry engineers building recognition pipelines who need to understand why quality scores fluctuate and what a cumulative alternative costs. Readers should be comfortable with angular-margin softmax losses (ArcFace, CosFace, CurricularFace), class-center-based similarity, and ranking metrics such as Spearman correlation and pAUC.

Authors’ abstract

Face Image Quality Assessment determines the suitability of captured face images for automated face recognition (FR), a critical capability for reliable biometric systems. Existing state-of-the-art FR-integrated FIQA methods suffer from temporal instability: as the feature space evolves during training, single-epoch quality estimates fluctuate, creating a moving target that undermines reliable quality prediction. We introduce CARPM-FIQA, a stabilization strategy for FR-integrated FIQA that accumulates relative point margin measurements, the ratio between intra-class compactness and inter-class separation, across the entire training trajectory rather than relying on single-epoch estimates. This cumulative averaging approach provides theoretically grounded advantages: reduced variance in quality estimates, improved mean squared error, and enhanced ranking stability with convergence guarantees as training progresses. Through controlled experiments on the SynFIQA dataset with labeled quality groups, we demonstrate that cumulative averaging achieves superior discriminative ability, and ablation studies across different training configurations confirm consistent improvements. Evaluated against twelve FIQA methods on eight challenging benchmarks with four FR models at two FMR thresholds, CARPM-FIQA places 4th (CARPM-FIQA(L)) and 6th (CARPM-FIQA(S)) of 17 compared methods by pAUC-EDC and AUC-EDC averaged across FR models and, after per-benchmark normalization, across benchmarks, staying within a few percent of the best method's normalized average for every FR model, providing a principled solution to training instability while maintaining the performance benefits of FR integration. More broadly, our work demonstrates that temporal aggregation strategies can stabilize training objectives in deep learning systems where target values inherently fluctuate due to evolving feature representations.