Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts

AI summary

Overview

• Research area: Efficient large language model pre-training; specifically model growth / checkpoint recycling for Mixture-of-Experts (MoE) architectures.
• Technical level: Advanced. The paper assumes familiarity with Transformer and MoE architectures, Chinchilla scaling laws, routing and top-k gating, and optimizer-state analysis.
• Scope: A framework for expanding already-converged MoE checkpoints along two orthogonal axes (depth and width) so that prior training compute ("sunk cost") is reused rather than discarded.

What This Paper Is About

Pre-training LLMs is enormously expensive, yet the smaller checkpoints and intermediate artifacts produced along the way are usually thrown away once training finishes. This paper asks how to "recycle" a fully converged MoE checkpoint by growing it into a larger model, so that the compute already spent on it is not wasted. The authors propose two growth operations, layer copying for depth and noisy expert duplication for width, and study when and how to apply them.

Key Contributions

1. An orthogonal growth framework for converged MoE models. Two growth dimensions are proposed and shown to be mutually independent: depth growth via interpositional layer copying, and width growth via expert duplication with small Gaussian noise.
2. Evidence that interposition beats stacking for converged models. The paper identifies a layer-wise weight-norm signature of convergence and shows that stacking disrupts it while interposition preserves it, with a boundary condition at approximately 1× the Chinchilla-optimal FLOPs.
3. A systematic study of growth timing and sunk cost. Across 12 checkpoints sampled between 8k and 96k steps, the authors show a strong positive correlation between prior training investment and post-growth accuracy, and compare growth against from-scratch training under both fixed additional and fixed total FLOPs budgets.
4. A large-scale validation. The framework scales a 17B MoE model to 70B parameters over a 1-trillion-token dataset, reporting a 10.6% average accuracy improvement over a from-scratch baseline given the same additional compute budget.

Main Findings

• Interposition outperforms stacking on converged models. Using a 3B MoE model (700M active parameters) grown to 6B (1B active) with growth factor k=2, the interpositional method consistently produced lower training loss and higher downstream accuracy than stacking.
• The convergence boundary is roughly 1× Chinchilla-optimal FLOPs. For the 3B model, the compute-optimal value is reported as F_c ≈ 6 · N_a · (20N_a) = 5.88 × 10^19. Once total training FLOPs exceed F_c, interposition is preferred; the point where the two methods begin to diverge emerges at approximately 1× F_c. A stable layer-wise increasing weight-norm pattern emerged around 4k steps (≈1.2 F_c).
• Stacking creates a norm discontinuity. For the converged 3B model, the norm drop from layer 19 to the stacked layer 20 is approximately 10× the average inter-layer variation, which the paper argues must then be repaired with further training budget.
• Small noise on copied experts helps. Direct expert replication (α = 0) and noise-augmented duplication produced comparable language modeling loss, but the noisy version gained approximately 1% on downstream tasks. Excessive noise was harmful, and randomly initializing new experts while retaining originals performed significantly worse. The paper notes that MoE upcycling from dense models typically requires noise of at least 50%, in contrast to the minimal scale used here (α = 0.01).
• Depth growth and width growth differ in character. Depth growth generally yielded better downstream performance than width growth. Width growth better preserved stability immediately after expansion, consistent with Function-Preserving principles.
• The two growth axes are orthogonal. Adam first-moment cosine similarity between pre-growth and each grown variant stayed near zero throughout training (all |cos| < 0.04). Cumulative weight-update cosine similarity between depth-grown and width-grown models decreased monotonically, with MoE-specific parameters diverging most (overall 0.630 at +2k, 0.490 at +8k, 0.431 at +16k). Final performance was invariant to whether width was applied before depth.
• More sunk cost means better final models (up to a point). With a fixed additional budget of 3 × 10^20 FLOPs, average accuracy rose from 36.29 when growing from the 0k checkpoint (a from-scratch 6B baseline) to 47.88 from the 88k checkpoint. Gains diminished once the base model entered the learning rate annealing stage beyond 72k steps.
• Growth is competitive under a fixed total budget. Under matched total FLOPs, grown models were comparable to or slightly superior to scratch training, with earlier checkpoints (and thus larger post-growth budgets) performing best. The suggested heuristic is to allocate additional FLOPs at least on the same order of magnitude as the sunk cost.
• Large-scale results. The 70B model reached an average accuracy of 64.17, which is 2.21 points above the 35B checkpoint and 5.62 points above the initial 17B model. Controlled for total training FLOPs, the 70B model still beat the 17B baseline by 2.96 points (a 4.0% relative gain), and exceeded the from-scratch baseline by 6.18 points (a 10.6% relative gain) under the same extra FLOPs budget.
• Consistent norm pattern across open-source MoEs. The increasing layer-wise weight-norm trend was also observed in Deepseek-v2-Lite-16B-A2.4B, Qwen1.5-MoE-14.3B-A2.7B-Chat, Mixtral-8x7B, Hunyuan-A13B-Instruct, Dots-LLM1-142B-A14B, and GroveMoE-Inst-33B-A3.2B.

Methodology in Plain English

The researchers start from an already fully trained MoE model and enlarge it along two independent directions.

For depth, they compare two ways of duplicating layers. The common "stacking" approach repeats the entire block of layers in sequence (layer 1 through n, then layer 1 through n again). Their "interposition" approach instead duplicates each layer in place (layer 1 twice, layer 2 twice, and so on). They measure the average Frobenius norm of expert weight matrices across layers and find that well-converged models show a characteristic pattern: small and variable early, increasing through the middle, slightly decreasing near the end. Stacking breaks this pattern; interposition keeps it intact. They trace the boundary where interposition starts to win back to the Chinchilla-optimal compute point.

For width, they double both the total number of experts (E → 2E) and the number of activated experts (k → 2k), so that tokens can actually reach the new capacity. New experts and their router weights are copies of existing ones plus Gaussian noise drawn from N(0, (α σ_orig)^2) with α = 0.01. Exact copies would receive identical gradients and never diverge; the small perturbation breaks that symmetry without destroying what the experts already learned. The paper also tests routing variants that expand capacity without adding compute, and add compute without adding capacity.

To test growth timing, they train a 3B MoE model to full convergence with warmup, constant, and annealing phases, save checkpoints from 8k to 96k steps, and grow each one under controlled budgets. To verify orthogonality, they compare Adam optimizer first moments and cumulative weight updates on the parameters shared between depth-grown and width-grown models. The scaling study grows a 17B model to 70B by first doubling layers from 28 to 56 (yielding a 35B model) and then doubling experts from 96 to 192 (yielding 70B), over a total of 1T tokens.

Why This Matters

Impact on research. The paper reframes intermediate training checkpoints as an asset rather than waste, and extends model growth research from early-training BERT- and ViT-era settings to fully converged, large-scale MoE models. It also provides a mechanistic account, the layer-wise weight-norm profile, for why some growth strategies work and others do not, and offers empirical orthogonality evidence that two growth axes can be composed freely.

Real-world applications:

• Recycling discarded or superseded checkpoints from hyperparameter sweeps and preliminary runs into larger production models.
• Compute-budget-constrained model scaling for organizations that cannot afford full from-scratch training of a frontier model.
• Reducing the environmental footprint of model development by reusing existing computation instead of repeating it.
• Planning training curricula, such as deciding which checkpoint to fork and how much continued-training budget to allocate.

Industry relevance. The growth strategy was validated at 17B-to-70B scale with a 1T-token dataset, and the paper reports gains under both fixed-additional-FLOPs and fixed-total-FLOPs accounting, which are the budgets practitioners actually manage. The finding that checkpoint choice and post-growth learning rate scheduling matter materially gives teams concrete operational levers.

Future Directions

• Determining the optimal learning rate schedule for continued training after growth, since the reported diminishing gains from late annealing-phase checkpoints were attributed to using a single constant learning rate (3 × 10^-4) for all runs.
• Extending the growth-factor analysis further. The paper tested k = 2 and k = 4 (expanding a 20-layer 3B MoE into an 80-layer architecture), leaving open how very large expansion factors behave.
• Investigating additional routing configurations during width growth beyond the variants tested: 6B-A700M (top-4/128) and 3B-A900M (top-8/64) against the 6B-A900M (top-8/128) baseline.
• Applying orthogonal growth to architectures and scales beyond the MoE models studied, and quantifying how far the positive sunk-cost correlation holds past the 88k-step checkpoint where gains began to flatten.

Target Audience

This paper is most useful for machine learning researchers working on training efficiency, scaling laws, and MoE architectures; infrastructure and pre-training engineers planning large-scale training runs or managing checkpoint assets; and graduate students with a background in Transformer training who want to understand model growth as an alternative to from-scratch scaling. Readers without familiarity with Chinchilla scaling, MoE routing, or residual-network structure will find the methodology sections demanding.

Authors’ abstract

As the computational demands for pre-training Large Language Models (LLMs) continue to surge, the need for efficient training paradigms becomes critical. Despite the vast resources already invested in existing pre-trained checkpoints, these assets often remain under-leveraged due to architectural limitations. We introduce an "orthogonal growth" strategy designed to "recycle" these checkpoints by strategically expanding their parameters prior to continued training. Our method focuses on optimizing converged Mixture-of-Experts (MoE) models through two dimensions: interpositional layer copying for increased depth and noisy expert duplication for expanded width. Through extensive scaling laws analysis, we demonstrate a strong positive correlation between the "sunk cost" (prior investment) and the final model accuracy. Empirical results on models up to 70B parameters and 1T tokens show that our recycling approach yields a 10.6% accuracy improvement compared to training from scratch under identical extra compute budgets. This work provides a cost-effective blueprint for sustainable large-scale LLM development.