Cheat sheet

Optimization

10 formulas · Back to the formula library

  1. Adam

    θt+1=θt−ηm^tv^t+ϵ\theta_{t+1} = \theta_t - \eta \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}

    Momentum and RMSProp in one optimiser: a running mean of gradients divided by a running mean of their magnitudes, both bias-corrected for their cold start. It is the default for transformer training, and the two extra states are why it costs three times the memory of plain SGD.

    Symbols

    m^t\hat{m}_t
    Bias-corrected first moment — the mean gradient.
    v^t\hat{v}_t
    Bias-corrected second moment — the mean squared gradient.
    ϵ\epsilon
    A small constant for numerical stability.
  2. Cosine learning-rate schedule

    ηt=ηmin⁡+12(ηmax⁡−ηmin⁡)(1+cos⁡tπT)\eta_t = \eta_{\min} + \tfrac{1}{2}(\eta_{\max} - \eta_{\min})\left(1 + \cos\frac{t\pi}{T}\right)

    Decays the learning rate along a cosine curve from its peak to its floor. It holds a high rate early, then eases off smoothly rather than dropping in steps, and it is the schedule nearly every large language model is trained under.

    Symbols

    tt
    The current step.
    TT
    The total number of steps in the schedule.
    ηmax⁡\eta_{\max}
    The peak learning rate, reached after warmup.
  3. Gradient descent

    θt+1=θt−η∇θL(θt)\theta_{t+1} = \theta_t - \eta \nabla_{\theta} L(\theta_t)

    Step downhill: subtract a scaled gradient from the parameters and repeat. The entire difficulty is in the step size — too large and it diverges, too small and it never arrives.

    Symbols

    θ\theta
    The parameters being optimised.
    η\eta
    The learning rate, the step size.
    ∇θL\nabla_{\theta} L
    The gradient of the loss with respect to the parameters.
  4. L1 regularization

    Ltotal=L+λ∑i∣θi∣L_{\text{total}} = L + \lambda \sum_{i} |\theta_i|

    Penalises the absolute size of the weights instead of the square. The constant gradient near zero drives weights exactly to zero rather than merely close, which is what makes it a feature selector.

    Symbols

    λ\lambda
    The regularisation strength.
    ∣θi∣|\theta_i|
    The absolute value of one weight.
  5. L2 regularization

    Ltotal=L+λ∥θ∥22L_{\text{total}} = L + \lambda \|\theta\|_2^2

    Adds the squared size of the weights to the loss, so the optimiser pays for complexity. It shrinks every weight toward zero without quite reaching it, which spreads influence across features rather than concentrating it.

    Symbols

    λ\lambda
    The regularisation strength.
    ∥θ∥22\|\theta\|_2^2
    The sum of squared weights.
  6. Lagrange multipliers

    L(x,λ)=f(x)−λ g(x)\mathcal{L}(x, \lambda) = f(x) - \lambda\, g(x)

    Turns a constrained problem into an unconstrained one by pricing the constraint. At the optimum the gradients of objective and constraint line up, and the multiplier is how much the objective would improve if the constraint relaxed.

    Symbols

    ff
    The objective being optimised.
    gg
    The constraint, written so it equals zero when satisfied.
    λ\lambda
    The multiplier — the shadow price of the constraint.
  7. Momentum

    vt+1=βvt+∇θL,θt+1=θt−ηvt+1v_{t+1} = \beta v_t + \nabla_{\theta} L, \qquad \theta_{t+1} = \theta_t - \eta v_{t+1}

    Keeps a running average of past gradients so the step has inertia. Consistent directions build up speed while oscillating ones cancel out, which is what gets you along a narrow valley instead of bouncing across it.

    Symbols

    vv
    The velocity, an exponentially weighted average of gradients.
    β\beta
    The decay rate, usually around 0.9.
  8. Newton's method

    θt+1=θt−H−1∇θL\theta_{t+1} = \theta_t - H^{-1} \nabla_{\theta} L

    Uses curvature as well as slope, jumping straight to the minimum of a local quadratic fit. It converges in dramatically fewer steps and is almost never used at scale, because inverting the Hessian is hopeless for billions of parameters.

    Symbols

    HH
    The Hessian of second derivatives.
    H−1H^{-1}
    Its inverse, which rescales the step per direction.
  9. RMSProp

    st+1=ρst+(1−ρ)(∇θL)2,θt+1=θt−ηst+1+ϵ∇θLs_{t+1} = \rho s_t + (1-\rho)(\nabla_\theta L)^2, \qquad \theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{s_{t+1}} + \epsilon} \nabla_\theta L

    Gives every parameter its own learning rate by dividing by the recent size of its gradient. Parameters with consistently large gradients get smaller steps, which keeps badly scaled problems from stalling.

    Symbols

    ss
    A running average of squared gradients.
    ρ\rho
    Its decay rate.
    ϵ\epsilon
    A small constant that prevents division by zero.
  10. Stochastic gradient descent

    θt+1=θt−η∇θLB(θt)\theta_{t+1} = \theta_t - \eta \nabla_{\theta} L_{\mathcal{B}}(\theta_t)

    The same step, but with the gradient estimated from a small batch instead of the full dataset. The noise this introduces is not purely a cost — it helps escape sharp minima, and it is the only reason training at scale is affordable.

    Symbols

    B\mathcal{B}
    A minibatch sampled from the training set.
    LBL_{\mathcal{B}}
    The loss averaged over that batch only.
Esc
↑↓ navigate↵ open? shortcuts

Keyboard shortcuts

Press Esc to close

Anywhere

⌘K
Search and commands
/
Search
?
This sheet
Esc
Close

Reading a lesson

J
Next lesson
K
Previous lesson
B
Bookmark