Do Vision CIL Methods Transfer to Protein? A Diagnostic Benchmark of Frozen Protein Encoders

AI summary

Overview

• Research area: Class-incremental learning (CIL) applied to protein function annotation using frozen pretrained protein encoders.
• Technical level: Advanced. The paper assumes familiarity with continual-learning baselines (EWC, iCaRL, DER++, RanPAC, FeCAM, EASE), Mahalanobis/covariance-based classification, and protein language/structure models.
• Scope: A controlled diagnostic benchmark of nine CIL methods across five frozen protein encoders and two enzyme datasets at different scales, testing whether vision-derived CIL recipes transfer to protein representations.

What This Paper Is About

Protein function annotation faces a label space that keeps growing, so a deployed model must absorb newly discovered enzyme classes without retraining on the full historical database. Class-incremental learning is the natural framework for this, but existing CIL methods were developed and validated largely on vision benchmarks, and their behavior over frozen protein representations has not been systematically examined. This paper asks whether those methods transfer, and if not, why.

Key Contributions

1. First systematic benchmark of CIL on frozen protein representations. The study spans nine CIL methods, five frozen encoders, two enzyme datasets, and four phase-construction protocols, with Naive fine-tuning and Joint training as lower and offline reference points.
2. A negative transfer result with a diagnostic explanation. No existing method is reliably strong across encoders; the dominant variation is encoder-dependent rather than phase-dependent, and method rankings shift substantially when only the representation changes.
3. Three traced causes of instability. The authors identify weak gradient-based stability signals when only the classifier head is trainable, substantial heterogeneity in embedding geometry, and covariance capacity requirements that change jointly with dataset scale and encoder geometry.
4. A design criterion rather than another baseline. The paper argues for encoder-aware protein CIL, in which statistical capacity adapts to representation geometry instead of being fixed in advance, and it treats the representation as a central component of continual-learning design rather than a passive feature extractor.

Main Findings

• No uniform winner across encoders. In the main benchmark, the strongest method on one encoder is often mediocre on another. RanPAC reaches 74.3% average incremental accuracy on Enzyme Reaction with ESM2-650M but only 21.8% with GearNet-Edge; on EC Number it reaches 82.2% with ESM2-650M and 26.1% with GearNet-Edge.
• Regularization gives inconsistent gains. EWC improves only unevenly over the Naive lower bound (for example, 21.8% versus 13.7% on Enzyme Reaction with ESM2-650M, but 6.9% versus 7.4% with GearNet-Edge), consistent with the limited gradient signal available when the encoder is frozen.
• Replay retains plasticity but still forgets. Replay-based methods keep high new-class plasticity yet incur substantial forgetting, particularly on EC Number, where DER++ reaches 64.5% average accuracy and 57.9% plasticity on ESM2-650M but still forgets 39.7%.
• The two prototype methods fail in opposite directions. FeTrIL retains high plasticity (98.8% on EC Number with ESM2-650M) yet forgets severely (67.9%). FeCAM preserves old classes effectively (5.8% forgetting on Enzyme Reaction with ESM2-650M) but its plasticity on EC Number falls to 58.7%, near the bottom of the table.
• Phase construction is not the driver. Across Random, Finer-EC, and Mixed protocols, the overall pattern stays broadly stable. On Enzyme Reaction, RanPAC and FeCAM achieve 60–63% accuracy across all protocols; on EC Number, RanPAC remains near 69% while FeCAM reaches 44–50%. Finer-EC gives modestly higher accuracy for some methods, for example FeCAM rising from 44.7% to 49.5% on EC Number, without altering the broader pattern.
• Encoder-driven variance persists under every protocol. RanPAC varies by approximately ±20–28% across encoders, indicating that encoder-dependent variability is not specific to a particular phase schedule.
• Weak gradient stability signals. Inter-phase conflict cosines are near zero (from −0.047 to −0.010) and Fisher information is highly concentrated in the top 1% of head parameters (0.78 to 0.98), leaving little conflicting signal for regularization to preserve.
• Embedding geometry varies sharply. Within-class covariance condition numbers span roughly five orders of magnitude across encoders (from 8.6×10⁴ for ProtBert on Enzyme Reaction to 7.1×10⁹ for GearNet-Edge), intrinsic rank r95 ranges from 60 to 217, and class separability ranges from 0.165 to 1.179.
• Covariance capacity is dataset- and encoder-dependent. On Enzyme Reaction, full shared covariance is best for all five encoders, outperforming per-class covariance by 6.3–22.4%. On EC Number, per-class covariance is best for four of five encoders, improving over full shared covariance by 3.4–6.5%. GearNet-Edge is the consistent exception: full shared covariance improves accuracy from 37.1% to 47.2% on Enzyme Reaction and from 58.5% to 72.4% on EC Number.
• Errors under one shared covariance are structured. On EC Number, confusions concentrate on transferases and hydrolases, a large and functionally distinct pair of EC families. With the covariance estimator held fixed, severity varies sharply by encoder: ESM2-150M misclassifies 461 hydrolase instances as transferases (for example 3.1.4.35 → 2.4.1.229), whereas the same failure is much weaker for SaProt (84 instances, 2.3.2.27 → 3.2.2.6).

Methodology in Plain English

The authors build a single controlled pipeline. Each protein is passed once through a frozen encoder and the resulting embedding is cached and reused for every method, so the backbone is never updated and continual-learning behavior is isolated from backbone adaptation. Five encoders are used: ESM2-650M (dimension 1280), ESM2-150M (640), ProtBert (1024), GearNet-Edge (3072), and SaProt (1280), covering sequence-only language models, a structure-based graph network, and a sequence–structure fusion model.

Classes arrive in seven sequential phases with disjoint class sets, and after each phase every method is scored on all classes seen so far, producing a lower-triangular accuracy matrix. From that matrix the paper reports average accuracy, forgetting, and plasticity, averaged over three random seeds. The incremental stream is built under four protocols that differ in how closely they follow the Enzyme Commission hierarchy: EC-first (one phase per top-level EC category), Random (shuffled but with EC-first phase sizes), Finer-EC (finer EC prefixes greedily packed into phases), and Mixed (classes from different top-level groups combined per phase). Each seed uses one shuffled phase order shared by all methods.

The nine CIL methods are grouped by strategy: EWC for regularization; Replay, iCaRL, DER++, and RER for replay; FeCAM and FeTrIL for prototype and covariance statistics; RanPAC and EASE for projection and subspaces. Crucially, each method uses one fixed hyperparameter configuration across every encoder–dataset pair, with no setting-specific tuning, which directly tests the plug-and-play transfer setting.

Diagnostics then probe why transfer fails. Gradient-side statistics measure inter-phase conflict cosine and Fisher concentration on the MLP head. Geometry-side statistics measure embedding dimension, intrinsic rank, effective rank, within-class covariance condition number, and class separability. A separate controlled experiment sweeps covariance granularity — three covariance-free controls (Euclidean nearest-class-mean, cosine nearest-class-mean, ridge classifier) against four Mahalanobis variants (shared diagonal, full shared, grouped with K = 8, and per-class) — on 256-dimensional PCA projections. Training uses a two-layer MLP with hidden dimension 1024, Adam at a learning rate of 10⁻³ for 100 epochs per phase, buffers of 2,000 samples for replay methods, EWC λ = 2000, FeCAM ridge λ = 10⁻⁴, RanPAC with D = 12800 and λ = 1.0, and a rank-256 adapter for EASE; all experiments ran on a single NVIDIA RTX 4090 GPU.

Why This Matters

The paper reframes a routine model-selection question as a representation question. Its practical message is that picking a stronger vision-derived CIL baseline is not the main lever for frozen protein encoders; the encoder's geometry is. That gives benchmark builders and method designers a concrete criterion — adapt statistical capacity to the representation — instead of a leaderboard entry.

Real-world applications:

• Enzyme function annotation pipelines that must add newly discovered EC classes without retraining on the full historical database, motivated by Swiss-Prot growing from roughly 4,700 EC numbers in 2018 to over 5,000 in 2024.
• Biocatalysis and metabolic engineering, where correctly assigning EC numbers to candidate enzymes affects pathway design and enzyme selection.
• Functional database curation and protein database maintenance, where incremental annotation models could absorb new labels while retaining historical coverage.
• Structure-aware annotation for proteins lacking reliable sequence-only signal, since the benchmark includes a structure graph network and a sequence–structure fusion model alongside sequence language models.

Industry relevance: pharmaceutical and biotechnology research groups, agricultural and industrial enzyme companies, and bioinformatics platform vendors all rely on function annotation; knowing that a CIL recipe validated on vision data may collapse when the encoder changes (for example RanPAC dropping to 21.8% and 26.1% on GearNet-Edge) informs how such systems should be validated before deployment.

Future Directions

• Encoder-aware covariance capacity. Rather than choosing between one shared covariance and per-class covariance, methods could interpolate: classes with compatible geometry share statistical strength while geometrically distinct regions retain separate covariance structure, with representation geometry deciding the granularity.
• Extending the principle beyond covariance. The authors suggest the same adaptivity may apply to projection dimension, distance metrics, regularization strength, and memory allocation.
• Partially or fully adapted encoders. The study deliberately freezes encoders; adapted encoders may exhibit different stability and geometric properties and remain an important open direction.
• Beyond enzyme function classification. Whether the same failure modes extend to other protein tasks under the EC hierarchy is stated as open, as is the use of the EC hierarchy itself as a biological prior for deciding when the hierarchy should define statistical grouping.
• Separating tuning gains from representation effects. Future work can compare the fixed-hyperparameter plug-and-play protocol used here against encoder-specific tuning to quantify how much of the gap is recoverable through adaptation.

Target Audience

Continual-learning researchers testing methods on non-vision modalities; protein representation learning researchers who use pretrained encoders for downstream function prediction; practitioners in computational biology and bioinformatics who deploy annotation models over growing label spaces; and benchmark designers interested in diagnostic evaluation rather than leaderboard rankings. Readers without a background in CIL or covariance-based classification will find the results tables and diagnostic section demanding, since the paper presumes familiarity with metrics such as average incremental accuracy, forgetting, plasticity, and Mahalanobis prototypes.

Authors’ abstract

Protein function annotation operates over an expanding label space, making class-incremental learning (CIL) a natural framework for incorporating newly discovered functions without full retraining. Yet existing CIL methods have been developed largely in vision and have not been systematically evaluated over frozen protein representations. We conduct a controlled benchmark of nine representative CIL methods across five sequence-, structure-, and sequence--structure-based protein encoders and two enzyme function datasets at different scales, while keeping the encoders frozen and evaluation settings consistent across representations. We find that no method transfers reliably across encoders, and that these failures persist across alternative phase constructions. Diagnostic analyses reveal that this instability arises from weak gradient-based stability signals, substantial heterogeneity in embedding geometry, and covariance assumptions whose appropriate capacity changes with both dataset scale and encoder. These findings suggest that the representation itself should be treated as a central component of continual-learning design rather than a passive feature extractor. More generally, reliable protein CIL will require encoder-aware methods that adapt their statistical assumptions and capacity to the geometry of the representation on which they operate. Code is available at https://github.com/zygao930/ProteinCIL.