Skip to content

Result/2026/7 min

Random-probe expert rankings degrade generation at 3% pruning

Dibyaprakash Pradhan · Negative result, published here

Abstract

Router-only profiling is cheap enough to be nearly free, which makes it tempting to prune on its ranking alone. This experiment tests how far that ranking can be trusted by making the smallest meaningful cut — two experts per layer, top-k unchanged — and comparing generations against the unmodified checkpoint. The cut is not free. The failure mode is informative about why, and points at what the ranking would have to be built from instead.

Published
2026-04-21
Venue
Negative result, published here
Keywords
expert pruning quality · MoE degradation · random probe routing statistics · OLMoE · negative result · routing collapse repetition
Cite
BibTeX below

Motivation

Router-only profiling loads a few megabytes of router weights and probes them with random unit-norm hidden states. It runs on CPU in under a second for a 16-layer model. Because it is so cheap, the obvious temptation is to prune directly on its ranking.

The question this experiment asks is how much that ranking is worth. Rather than testing an aggressive prune — where degradation would be unsurprising and uninformative — it tests the smallest cut that is still a cut.

Setup

OLMoE-1B-7B, 16 layers, 64 experts per layer, top-k 8. Router-only statistics over 1,024 samples produced 131,072 router selections, with 100% expert coverage — every expert was selected at least once. The plan drops the two lowest-ranked experts in each layer and leaves top-k at 8.

PropertyOriginalPruned
Experts per layer6462
Top-k88
Total tensors3,2193,123
Checkpoint size13.0 GB13.44 GB
Reduction—3.1% of experts

The intervention is deliberately minimal: a 3.1% reduction in expert count with the routing width unchanged.

Both checkpoints were then run greedily — do_sample=False, 30 new tokens — over the same five prompts, on the same hardware, in the same session.

Result

PromptOutcome
The capital of France isIdentical output
def fibonacci(n):Different branch structure, still valid
Mixture of Experts models work byDifferent wording, still correct
The largest planet in the solar system isCorrect, slightly repetitive
To make a cup of tea, you need toCollapses into a repetition loop

Greedy generation, original versus 62-expert prune. Four of five prompts survive; one fails outright.

The tea prompt is the informative one. The original drifts in the ordinary way a 1B model drifts — boiling water, a kettle, a stove. The pruned model emits "The water is boiling." and then emits it again, and again, for the remainder of the budget.

Two experts out of sixty-four, with routing width unchanged, is enough to turn a coherent continuation into a loop.

Why random probes fail here

The planner drops the two experts scoring lowest against random hidden states. Random hidden states are not the hidden states your prompts produce. An expert that looks marginal under isotropic noise may sit directly on the path of a common real input pattern — and the routing distribution is close enough to uniform that the ranking has very little margin to be wrong with.

This connects to the earlier measurement on Qwen3-Coder-Next, where the routing Gini was 0.1006 and the largest deviation from uniform anywhere in the model was 1.42×. When experts are nearly equally used, the difference between rank 62 and rank 64 is close to noise, and a ranking derived from the wrong input distribution will order them close to arbitrarily.

What follows from this

  • Router-only rankings are a structural baseline, not a production keep-list. Use them to decide whether a model is worth pruning, not which experts to remove.
  • Coverage is not sufficient as a safety metric. This run reported 100% coverage and still degraded — an expert being selected at least once says nothing about how much depends on it.
  • The keep-list should be derived from the workload. Collecting real routing statistics from live traffic is what the daemon proxy exists for; the two dropped experts would then genuinely be the least-used ones on that workload.
  • Any prune deserves a side-by-side generation check before it is trusted, and the check should include prompts long enough to expose looping.

The broader point is that cheap measurement and good measurement are different things, and the gap between them is exactly where a compression pipeline quietly loses quality. Publishing the failure is more useful than publishing the compression ratio.

Reproduction

python3 moe_monitor.py router-only \
  --model-dir allenai/OLMoE-1B-7B-0924 \
  --output ./stats/olmoe-report-62.json \
  --keep-experts 62 \
  --new-topk 8 \
  --samples 1024 \
  --strategy per-layer

python3 moe_prune.py prune-checkpoint \
  --model-dir allenai/OLMoE-1B-7B-0924 \
  --plan ./stats/olmoe-report-62-plan.json \
  --output-dir ./OLMoE-1B-7B-pruned-62 \
  --copy-support-files

The report and plan JSON for both this run and the 64 → 32 run are committed in the repository under stats/, together with the full session log.

References

  1. [1]Pradhan, D. MoE-Watcher-Modifier: model-agnostic Mixture-of-Experts analysis and pruning toolkit.
  2. [2]Pradhan, D. Expert routing in Qwen3-Coder-Next is close to uniform.
  3. [3]Muennighoff, N. et al. OLMoE: Open Mixture-of-Experts language models.
  4. [4]Lu, X. et al. Not all experts are equal: efficient expert pruning and skipping for mixture-of-experts large language models.

Cite this

@misc{pradhan2026randomprobe,
  title  = {Random-probe expert rankings degrade generation at 3% pruning},
  author = {Pradhan, Dibyaprakash},
  year   = {2026},
  note   = {Negative result. Measured with MoE-Watcher-Modifier},
  url    = {https://diby.app/research/random-probe-rankings-degrade-generation}
}
← All research