The Pulse
Multiverse Computing Prunes Half of Llama 3.3 With 77% MMLU
Multiverse Computing presents a physics-based method for removing transformer blocks from large language models. Its approach scores 76.9% on MMLU after deleting 40 of Llama 3.3 70B Instruct’s 80 blocks, beating a leading baseline by 22.9 p
AI.info Team ·
Multiverse Computing published a new account of its block-pruning method on September 21, describing a way to remove half the transformer blocks from Meta’s Llama 3.3 70B Instruct while retaining a 76.9% score on the MMLU benchmark. The result comes from a paper revised on September 11 and authored by David Jansen, Roman Rausch, Ali Hashemi, David Montero and Román Orús.
The research treats model compression as a physics problem rather than a simple ranking exercise. Instead of scoring each transformer block independently and deleting the lowest-ranked ones, the method models how removal decisions interact. At 50% depth compression, its selected configuration beats the paper’s block-influence baseline by 22.9 percentage points on MMLU.
The paper, “LLM Compression by Block Removal with Constrained Binary Optimization,” appears alongside a company explanation on Hugging Face. The authors also provide implementation code through a public GitHub repository.
Why deleting blocks is harder than ranking them
Removing complete transformer blocks reduces a model’s depth, which can lower memory use and inference cost without requiring changes to the remaining weight matrices. The decision becomes difficult when several blocks are removed together: a block that appears expendable on its own may become important when another block disappears.
Most block-removal techniques use local measures such as magnitude, sensitivity or block influence. Others restrict the search to one consecutive run of layers. Multiverse Computing argues that both approaches discard information about interactions between blocks, particularly when compression becomes aggressive.
The researchers assign each block a binary variable: zero means the block stays, while one means it is removed. They then introduce a second-order approximation of the model’s loss and calculate an approximate Hessian matrix. The diagonal terms represent individual effects, while the off-diagonal terms represent pairwise interactions between removal decisions.
An Ising glass for model compression
The resulting constrained binary optimization problem searches for a fixed number of blocks whose removal produces the lowest estimated energy. The authors map that problem to an Ising glass, a physical model in which many binary variables interact across a system.
That energy calculation is much cheaper than running a full benchmark for every possible pruning pattern. The team computes the Hessian with forward and backward passes over a calibration set, then reuses it for different compression targets. For the experiments, the calibration data included 2,048 samples randomly drawn from OpenHermes-2.5.
Exact enumeration remains possible for some models but grows quickly. The paper says that checking the roughly 29 billion configurations created by removing eight of Llama 3.3 70B Instruct’s 80 blocks took about two days on a single GPU. For larger searches, the authors use heuristic approaches, including an open-source tabu solver that can produce low-energy candidates in seconds.
Llama’s 80 blocks reveal the gap
Llama 3.3 70B Instruct has 80 transformer blocks and scored 82.2% on MMLU before compression in the reported evaluation. With 32 blocks removed, the constrained binary optimization method scored 76.6%, compared with 59.3% for the block-influence baseline.
At the deepest test, the method removed 40 of the 80 blocks and scored 76.9% on MMLU. The block-influence method fell to 54.0%. The constrained method also led the baseline on HellaSwag, Winogrande, ARC Challenge and BBH at that 50% depth setting, although its GSM8K score was zero in the reported run.
The advantage is not universal. At lighter compression levels, the method generally performs in line with competing techniques. The paper reports similar results across Llama 3.1 8B Instruct and Qwen3 14B, with some configurations tested after one epoch of knowledge-distillation retraining.
The best answer may not be the lowest-energy one
One of the paper’s more unusual findings is that the lowest-energy configuration is not always the strongest model. The researchers examine several low-energy solutions, including excited states above the ground state, and find that some perform better after retraining.
For Llama 3.1 8B Instruct, the lowest-energy candidates typically removed blocks toward the end of the model. A higher-energy candidate that removed an earlier block produced better results across several benchmarks after light retraining. The finding challenges the assumption that useful pruning must remove one uninterrupted section of late or middle layers.
That broader candidate pool also changes how practitioners might evaluate compression. Rather than treating optimization as a search for one definitive pruning pattern, the method produces a set of configurations that can be tested against different accuracy, speed and retraining requirements.
Testing the method on Nemotron’s mixed architecture
Multiverse Computing also applies the method to NVIDIA’s Nemotron-3-Nano-30B-A3B-FP8, a 52-block hybrid model that combines Mamba2 layers, attention layers and mixture-of-experts layers. Its blocks are arranged in a non-uniform structure, making simple layer-ranking strategies less suitable.
Without retraining, selected configurations that removed two or three mixture-of-experts layers beat the block-influence baseline on AIME25 or GPQA, depending on the configuration. In one broader evaluation, a configuration removing two MoE layers scored 85.0% on AIME25 and 69.5% on GPQA, compared with 65.0% and 70.1% for the baseline configuration.
The paper does not show that every block can be removed safely. Its own results show sharp variation among candidate configurations, especially on mathematical reasoning tests. The contribution is a way to expose and search those differences before spending the larger budget required for retraining and full evaluation.