Adam
Momentum and RMSProp in one optimiser: a running mean of gradients divided by a running mean of their magnitudes, both bias-corrected for their cold start. It is the default for transformer training, and the two extra states are why it costs three times the memory of plain SGD.
- Bias-corrected first moment — the mean gradient.
- Bias-corrected second moment — the mean squared gradient.
- A small constant for numerical stability.
Cosine learning-rate schedule
Decays the learning rate along a cosine curve from its peak to its floor. It holds a high rate early, then eases off smoothly rather than dropping in steps, and it is the schedule nearly every large language model is trained under.
- The current step.
- The total number of steps in the schedule.
- The peak learning rate, reached after warmup.
Gradient descent
Step downhill: subtract a scaled gradient from the parameters and repeat. The entire difficulty is in the step size — too large and it diverges, too small and it never arrives.
- The parameters being optimised.
- The learning rate, the step size.
- The gradient of the loss with respect to the parameters.
L1 regularization
Penalises the absolute size of the weights instead of the square. The constant gradient near zero drives weights exactly to zero rather than merely close, which is what makes it a feature selector.
- The regularisation strength.
- The absolute value of one weight.
L2 regularization
Adds the squared size of the weights to the loss, so the optimiser pays for complexity. It shrinks every weight toward zero without quite reaching it, which spreads influence across features rather than concentrating it.
- The regularisation strength.
- The sum of squared weights.
Lagrange multipliers
Turns a constrained problem into an unconstrained one by pricing the constraint. At the optimum the gradients of objective and constraint line up, and the multiplier is how much the objective would improve if the constraint relaxed.
- The objective being optimised.
- The constraint, written so it equals zero when satisfied.
- The multiplier — the shadow price of the constraint.
Momentum
Keeps a running average of past gradients so the step has inertia. Consistent directions build up speed while oscillating ones cancel out, which is what gets you along a narrow valley instead of bouncing across it.
- The velocity, an exponentially weighted average of gradients.
- The decay rate, usually around 0.9.
Newton's method
Uses curvature as well as slope, jumping straight to the minimum of a local quadratic fit. It converges in dramatically fewer steps and is almost never used at scale, because inverting the Hessian is hopeless for billions of parameters.
- The Hessian of second derivatives.
- Its inverse, which rescales the step per direction.
RMSProp
Gives every parameter its own learning rate by dividing by the recent size of its gradient. Parameters with consistently large gradients get smaller steps, which keeps badly scaled problems from stalling.
- A running average of squared gradients.
- Its decay rate.
- A small constant that prevents division by zero.
Stochastic gradient descent
The same step, but with the gradient estimated from a small batch instead of the full dataset. The noise this introduces is not purely a cost — it helps escape sharp minima, and it is the only reason training at scale is affordable.
- A minibatch sampled from the training set.
- The loss averaged over that batch only.