Fly-CL: A Fly-Inspired Framework for Enhancing Efficient Decorrelation and Reduced Training Time in Pre-trained Model-based Continual Representation Learning

AI summary

Overview

Research area: Continual learning (CL) with frozen pre-trained models, specifically representation-based class-incremental learning, plus biologically inspired algorithm design drawn from the fly olfactory circuit.

Technical level: Advanced. The paper assumes familiarity with continual learning, class prototypes, cosine-similarity classifiers, ridge regression, generalized cross-validation, and linear algebra (SVD, Cholesky factorization, random projection theory).

Scope: The paper proposes Fly-CL, a bio-inspired framework that decorrelates class prototypes for pre-trained-model-based continual representation learning while cutting training time relative to eight baselines across five datasets and two backbone families.

What This Paper Is About

Continual learning methods that keep a pre-trained model nearly frozen work by matching new inputs against class prototypes (averaged features of each class), but those prototypes are often highly correlated with one another, which blurs decision boundaries — a problem the authors call multicollinearity. Existing fixes for this are accurate but computationally expensive, which blocks use in real-time, low-latency settings. Fly-CL addresses both issues at once by mimicking the signal-processing stages of the fly olfactory circuit, adding random sparse expansion and top-k sparsification before prototype matching, and replacing costly hyperparameter search with a streaming ridge classifier.

Key Contributions

1. The authors propose an efficient, biologically plausible decorrelation framework that substantially reduces computational cost compared with current state-of-the-art CL methods while achieving comparable or improved accuracy.
2. They validate the method's decorrelation effectiveness and robustness through extensive experiments across different data setups and model architectures, supported by both theoretical and empirical analysis (Theorems 4.1 and 4.2, plus prototype correlation heatmaps).
3. They provide a theoretical argument that the under-studied KC to MBON pathway in the fly olfactory circuit also supports decorrelation, by showing its consistency with ridge classification under the Hebbian learning rule (Section 6).
4. They show the framework can be adapted to online continual learning by updating streaming statistics per batch, and they quantify where the time savings come from component by component.

Main Findings

• Multicollinearity is the stated bottleneck: Inter-prototype correlations are described as significantly greater than zero, which the authors say reduces the effective angular separation between classes in the embedding space and makes cosine similarity less sensitive to inter-class distinctions.

• Large speedups with comparable or better accuracy (ViT-B/16, Table 1): On CIFAR-100, Fly-CL reduces post-extraction training time by 91% with a marginal accuracy drop of 0.32% compared with state-of-the-art methods. On CUB-200-2011 and VTAB, it achieves 83% and 67% reductions in post-extraction training time versus the most efficient baseline while improving overall accuracy by 1.17% and 2.38%, respectively.

• Consistent gains on ResNet-50 (Table 2): Fly-CL improves overall accuracy by 1.89%, 1.53%, and 1.20% on CIFAR-100, CUB-200-2011, and VTAB, while reducing post-extraction training time by 93%, 74%, and 38% versus the most efficient baselines.

• Absolute numbers reported for Fly-CL (ViT-B/16, Table 1): CIFAR-100 average training time per task 19.07s, post-extraction time 5.38s, overall accuracy 93.89%; CUB-200-2011 4.43s, 0.35s, 93.84%; VTAB 2.48s, 0.34s, 96.54%.

• Absolute numbers reported for Fly-CL (ResNet-50, Table 2): CIFAR-100 14.28s, 5.25s, 84.61%; CUB-200-2011 3.90s, 0.44s, 80.25%; VTAB 2.53s, 0.34s, 94.00%.

• Online continual learning remains competitive (Table 3, ViT-B/16): Fly-CL in online mode reports CIFAR-100 25.46s, 12.57s, 92.96%; CUB-200-2011 6.44s, 2.33s, 92.59%; VTAB 3.17s, 1.09s, 96.38%, outperforming RanPAC and F-OAL in online mode on training time.

• Robustness difference from a baseline: F-OAL shows significant performance degradation on CNN backbones, which the authors attribute to error accumulation in its iterative update mechanism; Fly-CL does not exhibit this issue.

• Feature extraction dominates remaining time: The gap between average training time per task and post-extraction training time in Tables 1 and 2 indicates that feature extraction becomes the dominant time consumer in Fly-CL. The authors note model quantization (e.g., INT8) could reduce feature extraction time by around 4x without significant accuracy degradation, and that frameworks like TVM can be used for hardware-specific deployment.

• Component-level time savings (Table 4, CUB-200-2011, NVIDIA GeForce RTX 3090): Random projection drops from 0.22s to 0.08s actual runtime per task; ridge selection from 7.34s to 0.14s; prototype calculation from 0.20s to 0.10s; similarity comparison from 0.21s to 0.08s.

• Every component contributes: Ablations removing the projection layer, the streaming ridge classification, or data normalization — or all of them — cause noticeable accuracy degradation on the tested datasets.

• Hyperparameter defaults and sensitivity: Accuracy improves with projection dimension m and saturates beyond m = 10,000; accuracy increases monotonically with weight sparsity p and does not drop significantly unless p is excessively small; there is a broad plateau for the activation sparsity k. Defaults are m = 10,000, p = 300, k = 3,000.

• Theoretical guarantees: Theorem 4.1 states that, for sufficiently large p and d, any submatrix of the sparse random projection matrix is invertible with probability at least 1 − o(1). Theorem 4.2 bounds the performance degradation from top-k sparsification and shows the bound decays polynomially when k = Ω(m^α) with 0 < α < 1.

• Hyperparameter search cost reduction: Vanilla regularization selection via grid search and cross-validation is given as O(l m³) for l candidates; the GCV-based approach is dominated by an SVD at O(n_t² m). Prototype solving via Cholesky factorization reduces complexity from O(2/3 m³) to O(1/3 m³).

Methodology in Plain English

Fly-CL keeps a pre-trained encoder frozen and processes its output features through a pipeline that mirrors known fly olfactory processing:

1. Feature extraction and normalization. Images pass through a frozen pre-trained backbone (ViT-B/16 or ResNet-50 in the experiments), producing a d-dimensional embedding, which is then normalized according to the backbone-dataset combination.

2. Sparse random expansion (PN to KC analog). The embedding is multiplied by a fixed, very sparse random matrix W with m rows and d columns, where m is much larger than d and each row has exactly p non-zero entries drawn from a standard normal distribution. This lifts features into a much higher-dimensional space.

3. Top-k filtering (APL lateral inhibition analog). Only the k largest-magnitude components survive; the rest are set to zero. This implements a winner-take-all style of inhibition.

4. Streaming ridge classification (KC to MBON analog). Rather than storing all features, the method maintains two running statistics: a Gram matrix G capturing feature self-correlation and a matrix S accumulating class-specific weights. The classifier matrix C is solved as a regularized least-squares problem, C_t = (G_t + λI_m)⁻¹ S_t, with Cholesky factorization used for speed.

5. Adaptive regularization. Instead of expensive grid search over λ, the authors use a generalized cross-validation criterion computed from the singular value decomposition of the current task's feature matrix, choosing the λ that minimizes the GCV value.

6. Prediction. Inference is a similarity match: the highest-scoring class column of C for the sparsified high-dimensional feature wins. The top-k sparsity also reduces the cost of this similarity comparison.

The claim of "progressive decorrelation" is illustrated by Pearson correlation heatmaps of 10 randomly selected class prototypes after feature extraction, after random projection, and after similarity matching — correlations are shown dropping at each stage.

Why This Matters

Impact on research. The paper attacks a specific, named failure mode (multicollinearity between class prototypes) in the increasingly popular representation-based branch of pre-trained-model continual learning, and it does so with both a complexity analysis and an explicit biological mapping. It also makes a claim about a circuit stage — the KC to MBON pathway — that the authors say has received far less attention than the PN to KC stage in prior modeling work, and for which physiological evidence remains inconclusive. That framing invites neuroscience-inspired follow-up work.

Real-world applications (as framed in the paper):

• Smart camera systems that gain new object-recognition capabilities without retraining the entire model.
• Edge computing and other resource-constrained deployments, which the paper cites as a reason representation-based methods are practical.
• Real-time and low-latency applications where the time cost of advanced decorrelation methods is prohibitive.
• Hardware-specific deployment using tools such as TVM and quantization schemes such as INT8 to shrink the remaining feature-extraction bottleneck.

Industry relevance. The reported reductions in post-extraction training time (from roughly 38% to 93% across settings) target the operational cost of continuously updating deployed models. The method is described as compatible with a wide range of pre-trained backbones and with online batch-wise updates, which matters for systems that receive data streams rather than fixed task batches. Storage is also avoided: unlike mixture-based approaches, Fly-CL does not require storing previous models.

Future Directions

• Physiological grounding of the KC to MBON pathway. The authors note that evidence for this pathway's role in decorrelation remains inconclusive; the Section 6 discussion arguing consistency with ridge classification under Hebbian learning is the starting point, and the provided content is truncated there.
• Beyond the reported benchmarks. Results on datasets with severe domain shifts are referenced as being in Table 6, but those numbers are not included in the provided content, and ImageNet-R and ImageNet-A are listed as datasets without reported results here.
• Practical acceleration of feature extraction. Since feature extraction dominates runtime in Fly-CL, the paper raises quantization (INT8, roughly 4x) and TVM-based deployment as untested paths toward larger end-to-end speedups.
• Broader hyperparameter and architecture sweeps. Defaults of m = 10,000, p = 300, and k = 3,000 were set from experiments centered on the tested setups; the paper's own sensitivity plots show plateaus and saturation behavior that could be explored further across other backbones and label-space sizes.

Target Audience

Researchers and graduate students working on continual learning, class-incremental learning, and pre-trained-model adaptation; computational neuroscientists interested in whether fly olfactory circuit motifs (expansion coding, lateral inhibition, Hebbian plasticity) translate into useful machine learning algorithms; and applied engineers building on-device or low-latency systems who need to add new classes without retraining or storing prior models.

Authors’ abstract

Using a nearly-frozen pretrained model, the continual representation learning paradigm reframes parameter updates as a similarity-matching problem to mitigate catastrophic forgetting. However, directly leveraging pretrained features for downstream tasks often suffers from multicollinearity in the similarity-matching stage, and more advanced methods can be computationally prohibitive for real-time, low-latency applications. Inspired by the fly olfactory circuit, we propose Fly-CL, a bio-inspired framework compatible with a wide range of pretrained backbones. Fly-CL substantially reduces training time while achieving performance comparable to or exceeding that of current state-of-the-art methods. We theoretically show how Fly-CL progressively resolves multicollinearity, enabling more effective similarity matching with low time complexity. Extensive simulation experiments across diverse network architectures and data regimes validate Fly-CL's effectiveness in addressing this challenge through a biologically inspired design. Code is available at https://github.com/gfyddha/Fly-CL.