Cheat sheet

Deep learning

13 formulas · Back to the formula library

  1. Adam

    θt+1=θt−ηm^tv^t+ϵ\theta_{t+1} = \theta_t - \eta \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}

    Momentum and RMSProp in one optimiser: a running mean of gradients divided by a running mean of their magnitudes, both bias-corrected for their cold start. It is the default for transformer training, and the two extra states are why it costs three times the memory of plain SGD.

    Symbols

    m^t\hat{m}_t
    Bias-corrected first moment — the mean gradient.
    v^t\hat{v}_t
    Bias-corrected second moment — the mean squared gradient.
    ϵ\epsilon
    A small constant for numerical stability.
  2. Batch normalization

    BN(xi)=γxi−μBσB2+ϵ+β\mathrm{BN}(x_i) = \gamma \frac{x_i - \mu_{\mathcal{B}}}{\sqrt{\sigma_{\mathcal{B}}^2 + \epsilon}} + \beta

    Standardises each feature across the batch rather than within an example. It stabilises and speeds up convolutional training, at the cost of a train/test mismatch and a dependence on batch size that transformers refuse to accept.

    Symbols

    μB\mu_{\mathcal{B}}
    The feature's mean across the current batch.
    σB2\sigma_{\mathcal{B}}^2
    Its variance across the batch.
    ϵ\epsilon
    A small constant for numerical stability.
  3. Cross-entropy

    H(P,Q)=−∑xP(x)log⁡Q(x)H(P, Q) = -\sum_{x} P(x) \log Q(x)

    The average number of bits needed to encode data from P using a code optimised for Q. It splits into the entropy of P plus the divergence from P to Q, which is why minimising it over a model is the same as minimising that divergence.

    Symbols

    PP
    The true distribution, usually a one-hot label in practice.
    QQ
    The predicted distribution the model outputs.
  4. Discrete convolution

    (f∗g)[n]=∑mf[m] g[n−m](f * g)[n] = \sum_{m} f[m]\, g[n-m]

    Slides one signal over another, multiplying and summing at every offset. Reusing the same small kernel everywhere is what gives convolutional networks translation equivariance and their very small parameter count.

    Symbols

    ff
    The input signal, such as an image row.
    gg
    The kernel being slid across it.
    nn
    The output position.
  5. Dropout

    h~=m⊙h1−p,mi∼Bernoulli(1−p)\tilde{h} = \frac{m \odot h}{1 - p}, \qquad m_i \sim \mathrm{Bernoulli}(1-p)

    Randomly zeroes activations during training so no single unit can be relied on, then scales the survivors so the expected output is unchanged. At inference nothing is dropped, and the scaling is what makes the two regimes agree.

    Symbols

    mm
    A random binary mask, redrawn every step.
    pp
    The probability a unit is dropped.
    ⊙\odot
    Element-wise multiplication.
  6. GELU

    GELU(x)=x⋅Φ(x)\mathrm{GELU}(x) = x \cdot \Phi(x)

    Weights each input by the probability that a standard normal falls below it, giving a smooth curve that lets small negatives through instead of clamping them. It is the activation inside most transformer feed-forward blocks.

    Symbols

    xx
    The pre-activation input.
    Φ\Phi
    The standard normal cumulative distribution function.
  7. Layer normalization

    LN(x)=γ⊙x−μσ2+ϵ+β\mathrm{LN}(\mathbf{x}) = \gamma \odot \frac{\mathbf{x} - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta

    Standardises each example across its own features, then lets the model rescale and shift the result. Because it never looks across the batch, it behaves identically at batch size one — which is why transformers use it and not batch norm.

    Symbols

    μ\mu
    The mean over the feature dimension of this example.
    σ2\sigma^2
    The variance over the same dimension.
    γ\gamma
    A learned per-feature scale.
    β\beta
    A learned per-feature shift.
  8. ReLU

    ReLU(x)=max⁡(0,x)\mathrm{ReLU}(x) = \max(0, x)

    Pass positives through, clamp negatives to zero. Its gradient is one or nothing, so it neither saturates nor vanishes, and that single property is most of why very deep networks became trainable.

    Symbols

    xx
    The pre-activation input.
    max⁡\max
    Selects the larger of the two arguments.
  9. Scaled dot-product attention

    Attention(Q,K,V)=softmax ⁣(QK⊤dk)V\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right) V

    Every token proposes a query, every token advertises a key, and the match between them decides whose value gets read. Dividing by the square root of the key dimension keeps the dot products from growing with width and flattening the softmax gradient.

    Symbols

    QQ
    The queries — what each position is looking for.
    KK
    The keys — what each position offers.
    VV
    The values — what actually gets mixed and returned.
    dkd_k
    The key dimension, whose square root rescales the scores.
  10. Sigmoid

    σ(x)=11+e−x\sigma(x) = \frac{1}{1 + e^{-x}}

    Squashes any real number into the range zero to one, so the output can be read as a probability. Its gradient vanishes at both ends, which is exactly why deep networks moved to ReLU for hidden layers and kept sigmoid for the output.

    Symbols

    xx
    The input, an unbounded real number.
    e−xe^{-x}
    The exponential that produces the S-shape.
  11. Sinusoidal positional encoding

    PE(pos,2i)=sin⁡ ⁣(pos100002i/d)PE_{(pos, 2i)} = \sin\!\left(\frac{pos}{10000^{2i/d}}\right)

    Attention is order-blind, so position has to be added to the input explicitly. Using sinusoids at geometrically spaced frequencies means relative offsets show up as fixed rotations, and positions longer than any seen in training still produce valid codes.

    Symbols

    pospos
    The index of the token in the sequence.
    ii
    The index of the dimension within the encoding.
    dd
    The model's embedding dimension.
  12. Softmax

    σ(z)i=ezi∑j=1Kezj\sigma(\mathbf{z})_i = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}

    Turns a vector of unbounded scores into a probability distribution: exponentiate so everything is positive, then divide by the total so the parts sum to one. Larger gaps between scores produce a sharper distribution.

    Symbols

    z\mathbf{z}
    The vector of raw scores, usually called logits.
    KK
    The number of classes.
    ee
    The exponential, which makes every score positive and ordering-preserving.
  13. Temperature sampling

    pi=exp⁡(zi/T)∑jexp⁡(zj/T)p_i = \frac{\exp(z_i / T)}{\sum_{j} \exp(z_j / T)}

    Divides the logits before the softmax, which rescales how peaked the distribution is. Below one it sharpens toward the top choice; above one it flattens; at zero it becomes plain greedy decoding.

    Symbols

    TT
    The temperature.
    ziz_i
    The logit for token i.
    pip_i
    The resulting sampling probability.
Esc
↑↓ navigate↵ open? shortcuts

Keyboard shortcuts

Press Esc to close

Anywhere

⌘K
Search and commands
/
Search
?
This sheet
Esc
Close

Reading a lesson

J
Next lesson
K
Previous lesson
B
Bookmark