Profiling a MoE model without loading it

The obvious way to find out which experts a Mixture-of-Experts model uses is to run the model. Load the checkpoint, install hooks on every gate, push prompts through, record what the router picks. It works, it is accurate, and it requires enough memory to hold a model you may be trying to shrink precisely because it does not fit.

There is a cheaper path, and it comes from a structural observation: the routing decision does not depend on the experts. It depends on the router — one weight matrix per MoE layer, mapping a hidden state to a score per expert. Everything else in the checkpoint is what happens after the decision.

Reading only the routers

Safetensors checkpoints are indexed, so you can pull individual tensors without materialising the rest of the file. Load only the router weight matrices — a few megabytes across the whole model — probe each one with random unit-norm hidden states, and record which experts come out on top.

Five seconds for a 48-layer sweep changes what the tool is for. At that cost, profiling stops being an experiment you schedule and becomes something you run while deciding whether to bother.

What random probes can and cannot tell you

The honest limitation: random hidden states are not your tokens. What this measures is the router's structural preference — the shape of the decision boundary in isolation — rather than the distribution your workload induces.

• Good for: a first pass, structural comparison between layers, spotting cold experts, deciding whether a model is worth pruning at all.
• Not sufficient for: a production keep-list on a domain-specific workload.
• Complementary to: full-model mode with your real prompts, and the traffic daemon.

Closing the loop with live traffic

The strongest signal is the one your users generate. The daemon runs as a transparent HTTP proxy in front of any OpenAI-compatible server — ollama, vLLM, llama.cpp, LM Studio — and accumulates routing statistics from real requests in a background thread.

It works using the same observation. The daemon loads the router weights and the embedding table, extracts prompt text from each forwarded request, tokenises it with the checkpoint's own tokenizer, and pushes the token embeddings through the routers itself. The backend serves the request untouched; nothing is added to the response path.

python3 moe_monitor.py daemon \
  --model-dir /path/to/checkpoint \
  --backend http://localhost:11434 \
  --listen-port 8080 \
  --keep-experts 128 \
  --new-topk 4 \
  --report-every 50

The iterative version of this is the one I would actually recommend. Profile cheaply, prune once, and the resulting checkpoint may finally fit in memory — at which point you can afford full-model mode on it and get a far better ranking for the second pass.