MT-CC: Multi-Group Temperature Scaling for Asymmetric Calibration Behavior in Class-Incremental Learning

AI summary

Overview

Research area: Confidence calibration for deep neural networks operating in class-incremental learning (CIL), a subfield at the intersection of continual learning and uncertainty quantification.

Technical level: Advanced. The paper assumes familiarity with temperature scaling, expected calibration error, catastrophic forgetting, replay buffers, and standard CIL benchmarks.

Scope: The paper proposes MT-CC, a post-hoc calibration framework that replaces a single global temperature with multiple learned temperatures assigned by a lightweight grouping network, and evaluates it on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet against TS, ETS, IRM, PTS, AdaTS, and T-CIL.

What This Paper Is About

Neural networks trained on a stream of tasks tend to become overconfident at rates that differ from task to task: confidence and accuracy drift apart at different speeds depending on how old or new a class is, a phenomenon the authors call asymmetric calibration behavior. Because existing calibration methods in continual learning rely on one global temperature, they cannot simultaneously correct the overconfidence of heavily forgotten old classes and the milder miscalibration of freshly learned ones. The goal is a calibration scheme that assigns different temperatures to different samples based on signals that specifically reflect the incremental-learning setting, without sacrificing classification accuracy.

Key Contributions

1. Diagnosis of asymmetric miscalibration in CIL. The authors show empirically that the confidence–accuracy gap varies across tasks and widens systematically with the logit margin under continual updates, but stays roughly flat in a static model trained jointly on all classes — establishing that a single temperature is structurally insufficient.

2. Multi-group temperature scaling (MT-CC). A lightweight grouping network maps each sample to one of M groups using a CIL-aware feature vector combining the full softmax vector, the logit margin, and the probability mass assigned to old versus new classes; each group carries its own temperature.

3. Groupwise Difference-between-Confidence-and-Accuracy (GDCA) loss. An auxiliary regularizer that penalizes the gap between average calibrated confidence and empirical accuracy within each group, weighted by the square root of group size, to keep the coupled routing-and-temperature optimization stable.

4. A two-stage per-task protocol. Each task splits the validation set into a grouping-training subset and a separate calibration subset, trains routing and temperatures jointly with a differentiable relaxation of hard routing, then refines one temperature per group and carries those values forward as initialization for the next task.

Main Findings

• Single-temperature methods are inadequate in CIL. On CIFAR-10, T-CIL reaches ECE 15.34 ± 1.34, while the stronger hybrid T-CIL + ETS reaches 13.94 ± 1.35; MT-CC reaches 10.18 ± 1.48. On CIFAR-100 the corresponding numbers are 6.03 ± 0.42 for T-CIL, 5.63 ± 0.52 for T-CIL + ETS, and 4.49 ± 0.53 for MT-CC.
• Tiny-ImageNet improvement is substantial. MT-CC achieves ECE 15.82 ± 0.29 versus 18.96 ± 0.35 for T-CIL, with AECE 15.80 ± 0.32 versus 18.99 ± 0.40 and SCE 0.52 ± 0.01 versus 0.82 ± 0.01.
• Accuracy is preserved by construction. All compared methods are post-hoc, so they share the base model accuracies: 65.24% on CIFAR-10, 56.47% on CIFAR-100, and 29.42% on Tiny-ImageNet.
• Static-setting calibrators transfer poorly. PTS, AdaTS, ETS, and IRM, applied directly to CIL models, are outperformed — for example PTS reaches ECE 25.24 ± 2.01 on CIFAR-10 and 36.39 ± 1.08 on Tiny-ImageNet, worse than plain TS on the latter (27.28 ± 0.58).
• ImageNet gains hold across buffer sizes. With an ER backbone, MT-CC achieves ECE 12.59 ± 0.56 (5K), 7.25 ± 0.56 (10K), and 8.44 ± 0.69 (20K), compared with T-CIL's 13.45 ± 0.15, 7.74 ± 0.23, and 10.57 ± 0.22. ER base accuracies are 32.86%, 41.23%, and 49.52% respectively; the iCaRL base accuracies are 44.08%, 49.29%, and 54.24%, but the iCaRL calibration rows are truncated in the supplied content and are not reported here.
• Overconfidence scales with logit margin only in CIL. Using five equal-frequency logit-margin bins on CIFAR-100, the confidence–accuracy gap rises with margin in the continual model but not in the static joint-training model, and the NLL-optimal temperature rises with the margin under CIL while staying nearly constant in the static case (Appendix B.10).
• Older tasks are more miscalibrated than new ones. After all 10 CIFAR-100 tasks, earlier tasks show larger confidence–accuracy gaps than newly introduced tasks.
• Group assignment quality matters. A naive confidence-based grouping produces imbalanced groups, whereas MT-CC's grouping yields more effective allocation and lower ECE (Figure 4, after the 10th CIFAR-100 task).
• Two groups suffice. Ablations (Appendix B.1) indicate M = 2 delivers reliable calibration, and the old/new class-mass features give consistent gains over using only [δ, p], while random class-mass summaries give no comparable benefit (Appendix B.12).
• Augmented validation helps but is not required. MT-CC without augmentation still beats T-CIL on ECE (21.25 ± 1.42 vs 15.34 ± 1.34 on CIFAR-10 is not the case — the unaugmented variant is weaker than T-CIL on CIFAR-10 and CIFAR-100 but competitive on Tiny-ImageNet at 26.00 ± 0.92 vs 18.96 ± 0.35, which is also weaker); the augmented protocol is used for the headline results.

Methodology in Plain English

Standard temperature scaling divides every logit by one learned number T before the softmax, softening or sharpening all predictions uniformly. MT-CC instead keeps a small set of temperatures and learns a tiny router — a single fully connected layer with a softmax output — that decides which temperature a given sample should get.

The router looks at four things about a sample: the full softmax probability vector, the gap between the top two logits (the "margin"), and the total probability mass sitting on previously learned classes versus classes from the current task. That last pair is the CIL-specific trick: it tells the router how much a prediction is leaning toward old versus new classes, which is exactly the axis along which miscalibration differs in continual learning.

Training happens once per task. The validation set is halved: one half trains the router and temperatures together using a soft, differentiable version of "pick a group and scale by its temperature," so gradients flow through the routing decision. The other half is used afterward to fit a final temperature for each group separately by minimizing negative log-likelihood, starting from the values learned in the joint stage. Those temperatures are saved and used as the starting point for the next task.

A second loss term, GDCA, looks at each group after routing and pushes the group's average calibrated confidence toward its actual accuracy, with groups weighted by the square root of their size. The total objective is the routing loss plus a weighted GDCA term; the weight is 1.0 for ResNet experiments and 0.1 for ViT experiments. The router uses learning rate 0.01, Adam, batch size 16, and 50 epochs, with initial temperatures of [1.5, 1.5] and M = 2 groups.

Why This Matters

Impact on research. Almost all CIL work optimizes accuracy, treating calibration as an afterthought. This paper reframes calibration in continual learning as a heterogeneity problem rather than a single-parameter fitting problem, and its diagnostic (margin-dependent overconfidence appears under CIL but not in static joint training) is a reusable finding for anyone studying confidence under distribution shift. It also shows that static-setting adaptive calibrators like PTS and AdaTS degrade badly when transplanted into CIL, which is a useful negative result.

Real-world applications:

• On-device personalization. Phones and wearables that incrementally learn new user classes or behaviors need trustworthy confidence, because downstream systems act on high-confidence predictions and defer on low-confidence ones.
• Autonomous perception stacks. Vehicles and robots that receive periodic model updates must not silently become overconfident about classes learned many updates ago.
• Medical imaging with rolling deployments. A diagnostic model that adds new finding categories over time must report calibrated probabilities that clinicians can threshold safely.
• Industrial inspection and anomaly detection. Defect detectors retrained on new product lines need reliability estimates that stay meaningful for both legacy and newly added defect types.

Industry relevance. MT-CC is post-hoc: it bolts onto an already-trained continual learner without touching the backbone and without changing a single prediction. That is exactly the profile enterprises want, since recalibration can be added to a deployed model via a small router plus a handful of temperatures, and the calibration cost is a validation-set pass rather than a retraining run.

Future Directions

• Adaptive group counts. The paper fixes M = 2 and shows this is sufficient, but leaves open whether the number of groups should grow with the number of tasks or be selected per task from the observed confidence spread.
• Reducing dependence on augmented validation. The headline results use the T-CIL augmented validation protocol; the unaugmented variant is weaker, so how far MT-CC can be pushed with raw validation data alone (or with synthetic replay) remains an open question.
• Generalization beyond ER and iCaRL. Main experiments use ER as the base learner with additional comparisons to other CIL methods; testing whether the routing features transfer to regularization-based or parameter-isolation approaches would strengthen the claim of framework-level generality.
• Interaction with train-time calibration. MT-CC is purely post-hoc; whether combining groupwise routing with train-time calibration objectives yields further gains, or whether the two correct the same errors redundantly, is not addressed.

Note: the supplied content does not contain an explicit future-work section, so these are open questions implied by the paper's scope and ablations.

Target Audience

Researchers and graduate students working on continual learning, uncertainty quantification, or model reliability; practitioners deploying incrementally updated models who need calibrated confidence without retraining; and engineers who have a trained CIL model in production and want a drop-in, accuracy-preserving calibration layer. Readers should already be comfortable with softmax temperature scaling and the standard ECE/AECE/SCE metrics.

Authors’ abstract

As deep neural networks are required to continuously adapt to evolving data, class-incremental learning (CIL) has become an active research area. However, most existing works primarily focus on improving accuracy while overlooking confidence calibration, which measures the trustworthiness of the model's predictions. Although temperature scaling (TS) is effective for static calibration, it has rarely been explored in continual settings where data distributions evolve over time. In this paper, we introduce Multi-Group Temperature Scaling for Continual Calibration (MT-CC), a novel post-hoc calibration framework that addresses the asymmetric calibration behavior in CIL, where the rate of confidence changes does not align with the rate of accuracy degradation across tasks, leading to different levels of miscalibration. Our key contribution is to design multiple temperature parameters through a lightweight grouping network that separates samples by confidence statistics and task-aware signals in CIL. Furthermore, a Groupwise Difference-between-Confidence-and-Accuracy (GDCA) regularization is incorporated to promote intra-group consistency. Experiments demonstrate that MT-CC significantly improves calibration performance while maintaining accuracy, offering a principled framework for reliable continual learning.