Gradient descent explained
Gradient descent is the whole of learning reduced to one instruction: measure which direction increases the error, and take a small step the other way. Everything else in an optimiser is a refinement of how large that step should be.
- 2026-03-23
- Programmatic DIB
- Optimisation · Gradient descent · Deep learning
The idea
The gradient points in the direction of steepest increase of the loss. Step the opposite way and the loss goes down. Repeat a few million times and the parameters end up somewhere useful.
The step size — the learning rate — is the parameter that decides whether this works. Too large and the updates overshoot and diverge; too small and training is correct but takes forever. Most practical difficulty in training is some version of this tension.
Why it is stochastic in practice
Computing the gradient over the entire dataset for a single step is prohibitive, so it is estimated from a mini-batch instead. The estimate is noisy, and the noise turns out to be useful — it helps the optimiser escape poor regions rather than settling into the first flat spot it finds.
Modern optimisers build on this by keeping running statistics of past gradients to scale each parameter's step individually, which is why they converge in far fewer steps than plain gradient descent.
- Step against the gradient; repeat. That is the entire learning rule.
- The learning rate is the parameter that decides whether training works at all.
- Mini-batch noise is a feature, not just a compromise.