FlyPrompt: Brain-Inspired Random-Expanded Routing with Temporal-Ensemble Experts for General Continual Learning

AI summary

Overview

Research area: General Continual Learning (GCL) applied to parameter-efficient tuning (PET) of pretrained vision transformers, combined with a brain-inspired (NeuroAI) design borrowed from the fruit fly olfactory memory system.

Technical level: Advanced. The paper assumes familiarity with continual learning settings (GCL, CIL, TIL, DIL), prompt/adaptor-based PET, ridge regression, exponential moving averages, and generalization-bound style analysis.

Scope: This paper proposes FlyPrompt, a two-component framework — a closed-form random-expansion router and a temporal ensemble of per-task output heads — that improves expert assignment and expert quality for single-pass, boundary-free continual learning.

What This Paper Is About

General Continual Learning asks a model to learn from a non-stationary, single-pass data stream in which task boundaries are unclear and label spaces may overlap. Existing continual PET methods add prompt experts to a pretrained backbone, but they usually assume clear task cues and multiple training epochs, which do not hold in GCL. The paper targets two unresolved problems: how to route inputs to the right expert without task labels or iterative training, and how to keep each expert's representations and decision boundaries strong under sparse, imbalanced supervision.

Key Contributions

1. A problem decomposition for GCL. The authors reframe GCL as two interacting subproblems — expert routing (which expert gets each input) and expert competence improvement (how good each expert is) — and argue this is a better abstraction than the classic task-identity-prediction / within-task-prediction split, because in GCL the class-to-expert mapping is one-to-many.

2. Random Expanded Analytic Router (REAR). A routing module that applies a fixed random projection with a nonlinearity (ReLU) to expand backbone features from dimension d to M (with M > d, set to M = 10^4), accumulates a Gram matrix G and a prototype matrix Q online, and computes the router in closed form as (G + λI)^{-1} Q once at evaluation time. The design is inspired by projection neurons connecting sparsely and randomly to Kenyon cells in the fruit fly mushroom body, an expansion the paper describes as nearly 40-fold, followed by global inhibition for sparsity. Unlike RanPAC and ACIL, the random expansion is used only for routing, while prompts and heads remain trainable.

3. Task-wise Experts with Temporal Ensemble (TE²). Each expert receives an average-prompt warm start (the new prompt initialized as the mean of previously learned prompts), a non-parametric logit mask that suppresses unseen labels, and a bank of n EMA output heads with distinct decay rates. At inference, the online head plus the n EMA heads are each passed through softmax and combined by element-wise maximum, then masked. This mirrors the fruit fly's γ, α′/β′, and α/β Kenyon cell subtypes that operate on different timescales.

4. Theory plus broad empirical validation. Theorem 1 bounds the population excess risk of the ridge router as approximately sqrt(log(N)/M) + (sqrt(N)λ)^{-1} + λ; Theorem 2 bounds the EMA head parameter error as approximately ζ²/L + (L·P_t)², where L = 1/(1−α). Empirically, FlyPrompt is evaluated on three GCL benchmarks against prompt-based CL methods, GCL state-of-the-art methods, and prominent offline CL methods.

Main Findings

• Two bottlenecks confirmed by preliminary analysis. On the Sup-21K backbone across CIFAR-100, ImageNet-R, and CUB-200, routers that explicitly predict expert identity (DualPrompt, MVP, MISA) show limited expert-selection accuracy, and even an oracle router that always selects a correct expert leaves previous methods with inferior final average accuracy (A_last). The authors read this as evidence that routing and per-expert competence are separate, both-broken problems.

• Experts do specialize. Centered kernel alignment (CKA) analysis of MVP's expert representations on the three datasets shows experts occupy distinct feature subspaces, which is why accurate assignment matters.

• Leading results under the Sup-21K backbone. FlyPrompt reaches A_auc / A_last of 83.24 / 86.76 on CIFAR-100, 56.58 / 55.27 on ImageNet-R, and 70.64 / 73.40 on CUB-200. For comparison, the strongest listed baselines in the same table are MISA at 80.35 on CIFAR-100 A_auc and CODA-P at 80.91 A_last, CODA-P at 51.87 / 48.09 on ImageNet-R, and CODA-P at 66.01 / 62.90 on CUB-200. The abstract reports gains of up to 11.23%, 12.43%, and 7.62% over state-of-the-art baselines on CIFAR-100, ImageNet-R, and CUB-200 respectively.

• Advantage holds across six pretrained backbones. With Sup-21K/1K, FlyPrompt reports 78.48 / 80.39 (CIFAR-100), 62.01 / 56.55 (ImageNet-R), 54.42 / 55.50 (CUB-200); with iBOT-21K, 75.58 / 79.36, 57.75 / 54.39, 28.86 / 36.79; with iBOT-1K, 70.14 / 74.84, 61.50 / 57.18, 38.75 / 45.00; with DINO-1K, 65.92 / 72.66, 57.29 / 54.72, 37.38 / 44.66; with MoCo v3-1K, 64.12 / 71.51, 52.32 / 49.06, 27.92 / 33.32. In the MoCo v3-1K CUB-200 column, FlyPrompt's A_auc of 27.92 is marked as second-best rather than best — MVP's 28.48 is bolded there — so the method does not top every single cell.

• It also beats offline CL methods. Against S-Prompt++, HiDe-Prompt, HiDe-LoRA, HiDe-Adapter, NoRGa, and SD-LoRA under Sup-21K, FlyPrompt still reports the best numbers on all three benchmarks (for example 83.24 / 86.76 versus S-Prompt++ at 80.21 / 83.48 and HiDe-LoRA at 80.07 / 82.00 on CIFAR-100).

• Two EMA heads suffice in practice. The theory allows a geometric bank of EMA windows, but the paper reports that two EMA heads with windows of 10 and 100 (α = 0.9 and 0.99) are sufficient.

• Several design choices are ablated in the appendix. The logit mask is evaluated in Tab. 15, the analytic router against the analytic classifier (RanPAC) in Tab. 13, REAR against alternative routing strategies in Tab. 17, and different (r_D, r_B) settings plus online CL in Appendix F.1.

Methodology in Plain English

The authors start by inspecting how existing prompt-based continual learners fail, measuring both how often their routers pick a usable expert and how well they would do if routing were perfect. Finding both weaknesses, they split the problem in two and borrow solutions from the fruit fly.

For routing, instead of training a network to choose experts, they project each image's backbone features through a fixed random matrix into a much larger space and pass it through a nonlinearity. While data streams in, they keep running sums of feature correlations and per-task feature totals. Only at evaluation time do they solve for the router in one closed-form step (a regularized least-squares solution), which removes any need for gradient updates or multiple passes. A theoretical bound shows that making the expansion wider and the number of samples larger shrinks the routing error.

For expert quality, each new task's prompt starts from the average of all previous prompts rather than a random initialization. Each expert keeps several copies of its classification head that lag behind the live head by different exponential moving average rates. Predictions from the live head and the lagging heads are converted to probabilities with softmax, and the highest probability per class is taken. Slow heads preserve stable long-term information while the live head adapts quickly — the algorithmic analogue of the fly's short-, intermediate-, and long-term memory compartments. A second theoretical bound shows the multi-window bank always contains a head near the optimal bias-variance trade-off as the data distribution drifts.

Everything is evaluated on the Si-Blurry GCL protocol with a disjoint class ratio r_D = 50%, blurry sample ratio r_B = 10%, and five sessions, reporting average anytime accuracy (A_auc, measured every 1000 batches) and final accuracy (A_last), averaged over five runs with standard deviation.

Why This Matters

Impact on research. The paper argues that expert condition quality — not just routing accuracy — limits prompt-based continual learning, and backs this with an oracle-router experiment in which perfect routing still leaves baselines behind. It also shows that a forward-only, closed-form router can outperform trained similarity- or contrastive-based routers, which is a challenge to the standard recipe of co-training routers with the data stream. The two theorems give an explicit recipe for why random expansion helps: error shrinks roughly with log(N)/M, so wider expansions and more samples buy robustness without iterative refinement. As a NeuroAI contribution, it maps three specific fly circuit features (sparse random expansion, multi-timescale Kenyon cell plasticity, modular compartmentalization) onto concrete algorithmic components.

Real-world applications. The paper itself names autonomous agents and personal assistants as motivating settings where systems must learn from dynamic environments without clear task definitions. Other natural fits implied by the setting are:

• On-device assistants that must adapt to a user's changing preferences from a single stream of interactions, without storing or replaying past data.
• Robotics or embodied agents operating in environments whose categories shift over time.
• Deployment scenarios built on frozen pretrained backbones, where only small prompt parameters and lightweight statistics can be updated.
• Long-lived vision systems that must absorb new classes and re-encounter old ones without a clean retraining window.

Industry relevance. The method's main practical argument is cost: the router is solved once, in closed form, from accumulated statistics, so it avoids the repeated gradient passes that continual PET normally requires. The only persistent state is a Gram matrix and a prototype matrix, and with two EMA heads per expert the memory overhead stays modest, which matters for edge and streaming deployments. That FlyPrompt also outperforms several offline CL methods under the same protocol suggests the approach is competitive even when an industry setting can afford multiple epochs.

Future Directions

• Closing the remaining per-cell gap. FlyPrompt is second-best on CUB-200 A_auc with the MoCo v3-1K backbone (27.92 vs MVP's 28.48), so understanding why the routing-plus-ensemble advantage does not transfer uniformly across backbone pretraining regimes is an open question.

• Extending the router to one-to-many class-expert mappings. The paper notes that the same class can appear in multiple tasks, making the class-to-expert correspondence inherently one-to-many; how REAR should express and exploit that ambiguity rather than committing to a single argmax expert is not resolved.

• Sizing the EMA bank automatically. Theory permits a geometric bank of decay rates, but practice settles on two heads with windows 10 and 100. A principled rule for choosing the bank size and windows from observed drift P_t would connect Theorem 2 more tightly to deployment.

• Testing beyond Si-Blurry and image classification. The default configuration is r_D = 50%, r_B = 10% over five sessions, with other (r_D, r_B) settings relegated to Appendix F.1. Broader stress tests of blurriness, longer session counts, and non-vision modalities would show whether the fruit-fly-inspired decomposition generalizes.

Target Audience

Researchers and graduate students working on continual learning, parameter-efficient tuning, and test-time or streaming adaptation of pretrained models; practitioners deploying models that must adapt to shifting data without replay buffers or task labels; and NeuroAI researchers interested in how specific biological circuits (mushroom body expansion and compartmentalized memory) translate into concrete algorithmic mechanisms with provable properties. Readers without a background in continual learning benchmarks and ridge regression will find the theory sections demanding, though the two-component architecture is described accessibly.

Authors’ abstract

General continual learning (GCL) challenges intelligent systems to learn from single-pass, non-stationary data streams without clear task boundaries. While recent advances in continual parameter-efficient tuning (PET) of pretrained models show promise, they typically rely on multiple training epochs and explicit task cues, limiting their effectiveness in GCL scenarios. Moreover, existing methods often lack targeted design and fail to address two fundamental challenges in continual PET: how to allocate expert parameters to evolving data distributions, and how to improve their representational capacity under limited supervision. Inspired by the fruit fly's hierarchical memory system characterized by sparse expansion and modular ensembles, we propose FlyPrompt, a brain-inspired framework that decomposes GCL into two subproblems: expert routing and expert competence improvement. FlyPrompt introduces a randomly expanded analytic router for instance-level expert activation and a temporal ensemble of output heads to dynamically adapt decision boundaries over time. Extensive theoretical and empirical evaluations demonstrate FlyPrompt's superior performance, achieving up to 11.23%, 12.43%, and 7.62% gains over state-of-the-art baselines on CIFAR-100, ImageNet-R, and CUB-200, respectively. Our source code is available at https://github.com/AnAppleCore/FlyGCL.