Pruning a Mixture-of-Experts checkpoint without a GPU

The wall

The original goal was narrow: run Qwen3-Coder-Next-FP8 — 80 GB, 512 experts across 48 layers — on a single NVIDIA L4 with 22 GB of VRAM. Pruning experts is the obvious lever, and the obvious place to pull it is the serving framework.

That does not work. In vLLM 0.19.0, fused_moe/layer.py allocates all 512 expert tensors on the GPU inside __init__, before any CPU offload path engages. The allocation happens during construction, so there is no flag to set and no hook to intercept — the model has to fit before you are given the opportunity to make it smaller.

You cannot use the inference stack to shrink a model that the inference stack cannot load. The constraint is structural, not configurational.

The interesting question is the general one this failure implies: can a tool prune any MoE checkpoint, for any runtime, on any hardware — so that nobody has to repeat this?

Design: treat it as a file-format problem

The rewriter never calls from_pretrained. It opens the safetensors shards directly and processes them one at a time, copying through the tensors named in the keep-list and dropping the rest. Four properties follow immediately.

• No GPU required at any point in the rewrite.
• Peak memory is bounded by the largest single shard, not by the size of the model — an 80 B checkpoint rewrites on a laptop.
• No dependence on framework versions, because no framework is loaded.
• The source checkpoint is never modified; the rewrite writes a new directory.

The rewriter drops expert tensors absent from the keep-list, renames retained experts to compact IDs from 0 to target−1, slices each router weight matrix to match, updates config.json with the new expert count and top-k, and writes a pruning manifest recording the source and the plan that produced the output.

Measured rewrites

Validated end to end on OLMoE-1B-7B — 16 layers, 64 experts per layer, top-k 8, three shards, 13.0 GB. Hardware was an NVIDIA L4 box, but the rewrite itself ran on CPU.

A dry-run subcommand validates a plan against the checkpoint structure without writing anything; on this model it completed in 0.1 s. That matters more than it sounds, because a pruning plan that disagrees with the checkpoint layout should fail in a tenth of a second rather than after a 16-second write.

Two bugs the end-to-end test surfaced

Both were only findable by actually loading a pruned checkpoint, which is an argument for end-to-end tests over unit tests when the contract you depend on belongs to somebody else.

Schema drift between documentation and checkpoint

The OLMoE schema expected router tensors at model.layers.{n}.mlp.router.weight, following the model card and the transformers source. The published checkpoint uses model.layers.{n}.mlp.gate.weight. Inspection reported router_keys_found: 0 against expert_keys_found: 3072 — a shape that unambiguously indicates a naming mismatch rather than a missing component, which is why the inspect subcommand reports both counts separately.

Missing format metadata

The pruned shards loaded fine as safetensors and crashed the transformers loader with an invalid-metadata error. The rewriter had been writing its own provenance keys into the shard metadata and, in doing so, dropped the format: pt entry that transformers requires.

metadata = {
    "format": "pt",          # required by the transformers loader
    "moe_pruner": "true",
    "source_shard": str(src.root / shard_name),
}

What this does and does not establish

It establishes that the mechanical part of expert pruning — deciding what to keep, rewriting the checkpoint, producing something a standard loader accepts — is cheap, portable and independent of the serving stack. Seven seconds on CPU for a 13 GB model.

It establishes nothing about quality. The 64 → 32 output loads and generates, and what it generates is incoherent: prompts collapse into token repetition. Removing half the experts without any recovery training breaks routing, exactly as expected. Making the mechanics free does not make the decision free, and the decision is the hard part.

That question — how much quality a given prune actually costs, and whether random-probe rankings are good enough to choose by — is taken up separately.