Training an AI model is slow — back-propagation explained
Training is slow because every step runs the network forwards, then runs a second pass backwards to compute how every parameter should change, and the backward pass needs the forward pass's intermediate values kept in memory the whole time.
- 2026-03-20
- Programmatic DIB
- Training · Back-propagation · Deep learning
What the backward pass costs
Back-propagation is the chain rule applied efficiently: compute the loss, then walk backwards through the network working out how much each parameter contributed to it. The backward pass costs roughly twice the forward pass, so a training step is around three times the work of inference on the same batch.
That factor is the cheap part of the explanation. The expensive part is memory.
Why memory is the real constraint
The backward pass needs the intermediate activations the forward pass produced. They must be held from the moment they are computed until the gradient reaches them, which means activation memory scales with both batch size and depth — and it frequently, not the parameters, is what decides the largest batch you can run.
Gradient checkpointing is the standard answer: discard most activations and recompute them during the backward pass, trading extra compute for a large reduction in memory. Choosing to do more arithmetic in order to use less memory is a recurring theme in this field.
- A training step is roughly three times the work of inference on the same batch.
- Activation memory, not parameter count, often caps batch size.
- Gradient checkpointing trades recomputation for memory.