# Amazon's HOPE accounts for pairwise interactions when pruning MoE experts

Published 2026-11-09 · AI.info Team · AI.info

> Amazon describes HOPE, a method that uses pairwise expert interactions to prune mixture-of-experts language models. The method was evaluated on three models; at 50% pruning, it achieved an average rank of 1.58 and matched the unpruned Qwen3.5-35B-A3B on SWE-Bench Pro.

## HOPE accounts for pairwise expert interactions

On November 9, 2026, [Amazon](https://ai.info/companies/amazon) presented HOPE, a method for pruning experts in mixture-of-experts language models. Instead of ranking experts separately, HOPE accounts for pairwise interactions between experts that are consistently co-activated. The post says independent scoring can miss those cooperative relationships, particularly when pruning removes up to 50% of experts.

Source [huggingface.co](https://huggingface.co/blog/amazon/higher-order-pruning-experts)

## Expert MLPs account for 95% of Qwen3.5-122B’s approximately 250 GB parameter memory footprint

The post gives Qwen3.5-122B as an example: each of its 48 mixture-of-experts layers has 256 experts, and expert MLPs account for 95% of its approximately 250 GB parameter memory footprint. For any given token, only 8 experts in each layer are used. HOPE estimates pairwise interactions from a calibration dataset, then selects a set of experts to prune by minimizing a quadratic objective.

## HOPE’s average rank improves to 1.58 at 50% pruning

Amazon evaluated HOPE on Qwen3.5-122B, Qwen3.5-35B and GLM-4.5-Air, using Evol-CodeAlpaca and SWE-Bench Verified trajectories for calibration. The evaluation compared it with four first-order methods across pruning rates from 10% to 50%, using the Tulu3 suite and SWE-Bench Pro. Across all conditions, HOPE achieved an average rank of 2.07 out of 5 methods, with top-1 finishes in 39% of conditions and top-2 finishes in 67%.

At 50% pruning, HOPE’s average rank improved to 1.58, compared with 2.42 for the next-best method, REAP. On SWE-Bench Pro, the post reports a score of 42.3 for Qwen3.5-35B-A3B at 50% pruning, matching the unpruned model’s 41.8; REAP scored 36.2. The post says HOPE’s advantage is most pronounced at higher pruning rates, while first-order methods such as REAP and MAN are sometimes competitive at 10% to 20%.

## HOPE has been tested on Qwen3, Qwen3-Next, Qwen3.5 and GLM-4.5

HOPE’s package has three steps: calibration, solving the quadratic program, and pruning the model checkpoint. The calibration step runs a single forward pass over the calibration data, and the post says solving takes a few seconds per layer. The resulting pruned model is a standard [HuggingFace](https://ai.info/companies/huggingface) checkpoint.

HOPE captures pairwise interactions, but even higher-order interactions, including triplets and beyond, could provide additional signal, although they are combinatorially difficult to estimate.

## Sources

1. Original source · Blog post [Hugging Face](https://huggingface.co/blog/amazon/higher-order-pruning-experts) huggingface.co

## Related stories

- [Edge0 Serves a 35B MoE at 20 Tokens a Second in 3 GiB](https://ai.info/news/edge0-serves-35b-moe-at-20-tokens-per-second-in-3-gib)
- [Amazon releases tools to check AI-generated mathematical proofs](https://ai.info/news/amazon-ai-tools-verifiable-mathematical-proofs)
- [Amazon says Strands Decider replaced Sonnet for ticket labeling](https://ai.info/news/amazon-strands-decider-replaces-sonnet-ticket-labeling)
- [Amazon says autonomous AI agents wrote its AWS Emulator](https://ai.info/news/amazon-ai-agents-built-aws-emulator)
- [Amazon’s AI Buildout Drives $42.4B Trailing Free-Cash Outflow](https://ai.info/news/amazon-ai-spending-trailing-free-cash-outflow-42-4-billion)

## Explore

[More articles](https://ai.info/news)

---

Canonical: https://ai.info/news/amazon-hope-pairwise-moe-expert-pruning
