The Pulse
Amazon's HOPE accounts for pairwise interactions when pruning MoE experts
Amazon describes HOPE, a method that uses pairwise expert interactions to prune mixture-of-experts language models. The method was evaluated on three models; at 50% pruning, it achieved an average rank of 1.58 and matched the unpruned Qwen3
AI.info Team · Written with AI · fact-checked ·
HOPE accounts for pairwise expert interactions
On November 9, 2026, Amazon presented HOPE, a method for pruning experts in mixture-of-experts language models. Instead of ranking experts separately, HOPE accounts for pairwise interactions between experts that are consistently co-activated. The post says independent scoring can miss those cooperative relationships, particularly when pruning removes up to 50% of experts.
Why pruning experts matters
The post gives Qwen3.5-122B as an example: each of its 48 mixture-of-experts layers has 256 experts, and expert MLPs account for 95% of its approximately 250 GB parameter memory footprint. For any given token, only 8 experts in each layer are used. HOPE estimates pairwise interactions from a calibration dataset, then selects a set of experts to prune by minimizing a quadratic objective.
HOPE’s average rank improves to 1.58 at 50% pruning
Amazon evaluated HOPE on Qwen3.5-122B, Qwen3.5-35B and GLM-4.5-Air, using Evol-CodeAlpaca and SWE-Bench Verified trajectories for calibration. The evaluation compared it with four first-order methods across pruning rates from 10% to 50%, using the Tulu3 suite and SWE-Bench Pro. Across all conditions, HOPE achieved an average rank of 2.07 out of 5 methods, with top-1 finishes in 39% of conditions and top-2 finishes in 67%.
At 50% pruning, HOPE’s average rank improved to 1.58, compared with 2.42 for the next-best method, REAP. On SWE-Bench Pro, the post reports a score of 42.3 for Qwen3.5-35B-A3B at 50% pruning, matching the unpruned model’s 41.8; REAP scored 36.2. The post says HOPE’s advantage is most pronounced at higher pruning rates, while first-order methods such as REAP and MAN are sometimes competitive at 10% to 20%.
HOPE has been tested on Qwen3, Qwen3-Next, Qwen3.5 and GLM-4.5
HOPE’s package has three steps: calibration, solving the quadratic program, and pruning the model checkpoint. The calibration step runs a single forward pass over the calibration data, and the post says solving takes a few seconds per layer. The resulting pruned model is a standard HuggingFace checkpoint.
HOPE captures pairwise interactions, but even higher-order interactions, including triplets and beyond, could provide additional signal, although they are combinatorially difficult to estimate.