Skip to content

Result/2026/6 min

Expert routing in Qwen3-Coder-Next is close to uniform

Dibyaprakash Pradhan · Experimental result, published here

Abstract

Mixture-of-Experts checkpoints are widely assumed to collapse onto a small set of favoured experts, which would make pruning a matter of removing dead weight. Profiling every MoE layer of Qwen3-Coder-Next does not support that assumption: routing is only mildly concentrated, cold experts are rare, and their distribution across layers is uneven. This changes what pruning buys and how it should be measured.

Published
2026-04-21
Venue
Experimental result, published here
Keywords
Mixture of Experts · expert routing · routing collapse · Qwen3-Coder-Next · model compression · checkpoint pruning · expert coverage
Cite
BibTeX below

Setup

All 48 MoE layers of Qwen3-Coder-Next (FP8) were profiled in router-only mode: the router weight tensors are loaded from the safetensors checkpoint and probed with random unit-norm hidden states, without materialising the experts themselves. 1,024 samples produced 491,520 router selections across 512 experts per layer.

ModelLayersExpertsTop-kSamplesCoverageElapsed
Qwen3-Coder-Next FP848512 → 12810 → 41,02499.34%4.98 s
OLMoE-1B-7B1664 → 328 → 4512100%0.38 s

Configuration and cost of both profiling runs. Both executed on CPU.

Result: routing is only mildly concentrated

The routing distribution has a Gini coefficient of 0.1006, where 0 is perfectly uniform. The busiest expert in the model was selected 1,453 times; the quietest 488. A factor of three between extremes across 512 experts is not the winner-take-all behaviour the collapse narrative predicts.

Heatmap of expert selection counts across 48 layers and 512 experts
Fig. 1 — Expert selection counts, 48 layers × 512 experts. The absence of strong vertical banding is the finding: utilisation is spread rather than concentrated.Source: MoE-Watcher-Modifier, stats/plots/qwen3_report512_heatmap.png
BucketObservedUniformRatio
Top 1%1.42%1%1.42×
Top 5%7.02%5%1.40×
Top 10%13.22%10%1.32×
Top 20%25.04%20%1.25×
Top 50%57.12%50%1.14×

Routing mass by expert percentile, against what uniform routing would give. The largest deviation anywhere is 1.42×.

Cold experts are rare but unevenly distributed

Mean cold experts per layer: 3.396 of 512, roughly 0.66%. But 45 of 48 layers contained at least one cold expert, and a single layer contained 27. The scarcity is real; its distribution is not uniform across depth.

This asymmetry is the argument for per-layer keep-lists over one global ranking. A model-wide opinion about which experts matter will be wrong for the layers that disagree with it, and here most layers disagree slightly.

Illustration of the expert pruning plan for Qwen3-Coder-Next
Fig. 2 — Per-layer keep-list construction for the 512 → 128 plan, with top-k reduced from 10 to 4.Source: MoE-Watcher-Modifier, stats/plots/qwen3_report512_illustration.png

Interpretation

Because routing is close to uniform, pruning is not the removal of dead capacity — it is a priced trade. Cutting 512 experts to 128 retained 99.34% of observed routing mass, which means 0.66% of it is now unserved and those tokens route elsewhere. Coverage is the number that makes the trade legible, and it should be reported alongside any compression ratio.

Reproduction

pip install torch safetensors rich

python3 moe_monitor.py router-only \
  --model-dir Qwen/Qwen3-Coder-Next-FP8 \
  --output ./stats/report-512.json \
  --keep-experts 128 \
  --new-topk 4 \
  --samples 1024 \
  --strategy per-layer

The report JSON backing every number above is committed in the repository under stats/, alongside the plan file and the generated figures.

References

  1. [1]Pradhan, D. MoE-Watcher-Modifier: model-agnostic Mixture-of-Experts analysis and pruning toolkit.
  2. [2]Shazeer, N. et al. Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. ICLR 2017.
  3. [3]Fedus, W., Zoph, B., Shazeer, N. Switch Transformers: scaling to trillion parameter models with simple and efficient sparsity.
  4. [4]Muennighoff, N. et al. OLMoE: Open Mixture-of-Experts language models.

Cite this

@misc{pradhan2026routing,
  title  = {Expert routing in Qwen3-Coder-Next is close to uniform},
  author = {Pradhan, Dibyaprakash},
  year   = {2026},
  note   = {Measured with MoE-Watcher-Modifier},
  url    = {https://diby.app/research/expert-routing-qwen3}
}
← All research