Transporting Task Vectors across Different Architectures without Training

AI summary

Overview

• Research area: Machine learning — transfer learning, task arithmetic, and parameter-space model merging, applied to Transformer vision and language models.
• Technical level: Advanced. The paper assumes familiarity with task vectors, activation spaces, SVD, and orthogonal Procrustes analysis.
• Scope: A single paper introducing Theseus, a training-free method that transports task-specific fine-tuning updates (task vectors) from one pre-trained model to another whose layers have different widths and/or depths, by matching the functional effect of the update on intermediate activations rather than the parameters themselves.

What This Paper Is About

Fine-tuning a large pre-trained model produces a task-specific parameter update (a task vector), and prior work has shown such updates can be transferred between models only when those models share the same architecture. When models differ in width — different hidden dimensions in attention projections and MLP layers — direct parameter alignment, padding, or projection breaks down because the source and target representation spaces differ in dimension, basis, and geometry. This paper reframes the transfer as a functional matching problem on observed activations and solves it in closed form via orthogonal Procrustes alignment, so the update can be moved to a differently-sized model with forward passes only.

Key Contributions

1. A functional formulation of task-vector transport. Instead of matching parameters, the update is characterized by the bilinear form it induces between input and output activations of a layer, and transport is defined as finding a target-layer update whose induced bilinear form matches the source's (Eq. 3 and Eq. 4).
2. A Procrustes-based alignment with a closed-form transport rule. Two orthogonal alignment maps (for input and output activations) are obtained from SVD of cross-covariance matrices, yielding the rule τ_B = T_out τ_A T_inᵀ, which is training-free and norm-preserving.
3. Empirical validation across widths, architectures, and modalities. The method is evaluated on vision models of mismatched widths, on identically-architected ViT-B/16 models with different pre-training distributions, on an encoder–decoder language setting (T5-3B → T5-Large), and in a combined depth-and-width setting (ViT-B/16 → ViT-L/14).
4. A practical, memory-bounded implementation. Activation statistics are accumulated incrementally as cross-covariances (C_in, C_out), so memory is bounded by the models' internal feature dimensions rather than the number of calibration tokens M.

Main Findings

• Forward width transfer improves over zero-shot without gradients. Transporting a task update from a CLIP ViT-B/16 pre-trained on LAION-2B (A) to the wider ViT-B/16+ pre-trained on LAION-400M (B) on the 8-Vision benchmark raises average accuracy from the 58.76 zero-shot baseline to 64.59 (+5.83) with 1 alignment batch, and to 69.48 (+10.72) with 50 alignment batches. Higher alignment budgets help monotonically in this setting.
• Supervised fine-tuning remains an upper bound. Fully fine-tuned model B averages 90.39 (+31.63) and fully fine-tuned model A averages 92.96 (+34.20) on the same benchmark, so Theseus recovers a substantial but partial fraction of the achievable gain.
• Naive and inversion-based baselines fail or stagnate. Zero-padding the source update gives 58.50 (-0.26), a random update gives 59.00 (+0.24), and the pseudo-inverse baseline (τ_pinv) degrades badly at 1 batch (42.38, -16.38) and 2 batches (52.25, -6.51), which the paper attributes to rank-deficient activation matrices. The Tikhonov-regularized variant reaches 57.68 (-1.08), 59.60 (+0.84), and 60.29 (+1.53) at 1, 2, and 5 batches. A randomized-alignment control stays near zero-shot (59.13, +0.37 at 1 batch).
• Language transfer works through a frozen encoder. Transporting a T5-3B encoder update to T5-Large under a linear-probing protocol improves average accuracy over five GLUE natural language inference tasks from the 57.87 zero-shot target to 79.91 (+22.04) with 20 batches, 84.08 (+7.30) with 50 batches, and 85.84 (+3.93) with 100 batches. Fine-tuned T5-3B averages 91.26 and fine-tuned T5-Large averages 87.92.
• The method also competes under identical architectures. With ViT-B/16 models differing only in pre-training (A: DataComp-XL → B: LAION-2B), Theseus exceeds TransFusion (61.46, +1.14) at every shot level, reaching 71.40 (+11.09) at 1-shot and 75.47 (+15.16) at 20-shot. It stays close to or above the gradient-based GradFix, which reaches 71.16 (+10.84) at 1-shot and 73.68 (+13.37) at 20-shot. Combining the two gives 74.36 (+14.05) at 20-shot.
• Depth-and-width transport is possible with a layer-interpolation heuristic. Transferring from ViT-B/16 to ViT-L/14 (both pre-trained on DataComp-XL) with 10 alignment batches raises the 73.63 zero-shot average to 80.24 (+6.61). Fine-tuned model A reaches 93.23 (+19.60) and fine-tuned model B 94.59 (+20.96).
• Norm preservation is guaranteed. Because T_in and T_out have orthonormal rows, ‖τ_B‖_F = ‖τ_A‖_F, so the transported update keeps the magnitude and relative geometry of the original, and orthogonality removes the scaling ambiguity that destabilizes unconstrained least squares.
• Generative tasks transfer too. Transporting a summarization update from Flan-T5 to T5-v1.1 yields a 36.5% relative improvement in ROUGE-L over the zero-shot target model.
• Direction and width relationships are flexible. The formulation is symmetric, so transport can be run either way; when widths differ, transport acts as a norm-preserving embedding into a higher-dimensional aligned subspace or a projection onto the maximally aligned shared subspace when the target is narrower. Reverse transport (B → A, with the ViT-B/16 target pre-trained on the stronger DataComp-XL) is reported to still improve performance, though detailed numbers are left to the appendices.

Methodology in Plain English

The setup is two Transformer-style base models, A (source) and B (target), that may differ in width, depth, or pre-training data. Model A is fine-tuned on a downstream task, producing a per-layer weight change τ_A. The goal is to construct a corresponding change τ_B for model B with no further training.

The key move is to stop thinking about weights and start thinking about effects. Applying τ_A to a layer changes that layer's outputs by ΔH_out,A = H_in,A τ_Aᵀ. Rather than trying to match that change token by token, the paper summarizes it as a matrix G_A = ΔH_out,A H_out,Aᵀ, which captures how the update's induced change at one token aligns with the original output representation at another. This matrix defines a bilinear form over token representations — the "functional signature" of the update — and transport becomes the search for a τ_B whose bilinear form is as close as possible to A's.

That objective is underdetermined when the two models have different widths, and naive least-squares solutions can amplify poorly conditioned, nearly-zero directions. The fix is to constrain the alignment maps to be orthogonal, which preserves norms and angles and removes arbitrary rescaling. To do this, the method collects activations from a small calibration set for both models, interpolates sequence lengths when they differ (bilinear interpolation when L^A < L^B), and computes cross-covariance matrices C_in = H_in,Aᵀ H_in,B and C_out = H_out,Aᵀ H_out,B. SVD of these covariances solves two orthogonal Procrustes problems in closed form: T_in = U_in V_inᵀ and T_out = U_out V_outᵀ. The transported update is then simply τ_B = T_out τ_A T_inᵀ, added to the target model as θ_B + α τ_B, where α is a scalar chosen by a linear search on a validation set.

The method is applied to all Transformer linear submodules, including attention projections and MLP layers, while positional and patch embeddings are excluded; only forward passes and frozen models are used. Activation statistics are accumulated incrementally across batches so memory stays bounded by feature dimensions rather than token count. Calibration is measured in two protocols: a ℬ-shot protocol with ℬ ∈ {1, 2, 5, 10, 20} batches of batch size 32, and a standard 𝒦-shot protocol with 𝒦 ∈ {1, 2, 5, 10, 20} examples per class. For models of different depth, a simple interpolation heuristic induces a coarse layer correspondence before transport.

Why This Matters

Impact on research. The paper argues that task identity should be defined by functional behavior rather than parameter values — the framing alludes to the Ship of Theseus thought experiment. This shifts the axis of the model-merging and rebasin literature, which has largely assumed architectural equivalence and matching parameter dimensionality, toward functional matching that survives mismatched widths. It also offers an alternative to gradient-based transport such as GradFix and to knowledge distillation and teacher–student approaches, which relearn the task rather than transporting an existing update. The reported results suggest closed-form alignment can match or exceed iterative optimization on identical architectures while being substantially more stable than pseudo-inverse approaches in low-shot regimes.

Real-world applications (as implied by the evaluated settings):

• Consolidating task knowledge across a fleet of model checkpoints of different sizes and pre-training data, without retraining each variant.
• Rapidly specializing small or widely-deployed models using updates learned on larger, more capable siblings.
• Low-shot specialization when only a handful of calibration batches (as few as one) are available for the target model.
• Extending an already strong model with task-specific behavior in high-accuracy regimes, as shown by the reverse vision transfer into a DataComp-XL-pre-trained target and by text summarization transfer (36.5% relative ROUGE-L gain).

Industry relevance. Serving multiple model sizes is common in deployment, and relearning a task for every size is expensive. A training-free, forward-pass-only transport rule that preserves the update's Frobenius norm offers a cheap path to reuse task-specific work across model variants, including when the target model already has a strong zero-shot baseline. The paper reports the method runs without backpropagation and that alignment cost is decoupled from calibration-set size through incremental covariance accumulation, which matters for production-scale calibration data.

Future Directions

1. Principled handling of depth mismatch. Depth differences are currently addressed with a "simple interpolation-based heuristic" for layer correspondence; the paper explicitly leaves detailed exploration of depth to future work, and the reported depth-and-width gains (+6.61 on average for ViT-B/16 → ViT-L/14) are smaller than the width-only gains.
2. Closing the gap to supervised fine-tuning. Large margins remain between transported updates and fully fine-tuned models in both vision (e.g., 69.48 vs 90.39 and 92.96 on the 8-Vision benchmark at 50 batches) and language (85.84 vs 87.92 and 91.26 on average GLUE accuracy at 100 batches).
3. Scaling calibration and alignment further. Results improve monotonically with the number of alignment batches in both the vision and language experiments, raising the question of whether even richer activation statistics or better calibration-data selection could narrow the remaining gap.
4. Extending beyond classification and the current benchmarks. Summarization is the only non-classification task reported; broader generative, multimodal, and multi-layer or cross-depth transport settings are open questions. The appendices also reference an empirical ablation against feature translation methods (Appendix G) and comparisons of sequence alignment strategies (Section E.1) that point to further questions about which alignment choices matter most.

Target Audience

Readers who will benefit most are researchers and practitioners working on model merging, task arithmetic, and parameter-space transfer, especially those who need to move task-specific updates between models of different sizes rather than identical ones. It is also relevant to engineers maintaining families of pre-trained models who want a training-free way to reuse fine-tuning work, and to readers interested in representation alignment via Procrustes or CCA as a tool for functional comparison of neural networks. The paper is written at an advanced level: it assumes comfort with matrix decompositions, activation-space geometry, and standard transfer-learning and fine-tuning protocols, and readers unfamiliar with task vectors or rebasin should read the cited background work (Ilharco et al., 2023; Ainsworth et al., 2023; Rinaldi et al., 2025) first.

Authors’ abstract

Adapting large pre-trained models to downstream tasks often produces task-specific parameter updates that are expensive to relearn for every model variant. While recent work has shown that such updates can be transferred between models with identical architectures, transferring them across models of different widths remains unexplored. In this work, we introduce Theseus, a training-free method for transporting task updates across heterogeneous-width models. Rather than matching parameters, we characterize a task update by the functional effect it induces on intermediate representations. We formalize task-vector transport as a functional matching problem on observed activations and show that, after aligning representation spaces via orthogonal Procrustes analysis, it admits a stable closed-form solution that preserves the geometry of the update. We evaluate Theseus on vision and language models across different widths, showing consistent improvements over baselines without additional training or backpropagation. Our results show that task updates can be meaningfully transferred across architectures when task identity is defined functionally rather than parametrically. Code is available at https://github.com/apanariello4/merge-and-rebase.