Adam
Momentum and RMSProp in one optimiser: a running mean of gradients divided by a running mean of their magnitudes, both bias-corrected for their cold start. It is the default for transformer training, and the two extra states are why it costs three times the memory of plain SGD.
- Bias-corrected first moment — the mean gradient.
- Bias-corrected second moment — the mean squared gradient.
- A small constant for numerical stability.
Batch normalization
Standardises each feature across the batch rather than within an example. It stabilises and speeds up convolutional training, at the cost of a train/test mismatch and a dependence on batch size that transformers refuse to accept.
- The feature's mean across the current batch.
- Its variance across the batch.
- A small constant for numerical stability.
Cross-entropy
The average number of bits needed to encode data from P using a code optimised for Q. It splits into the entropy of P plus the divergence from P to Q, which is why minimising it over a model is the same as minimising that divergence.
- The true distribution, usually a one-hot label in practice.
- The predicted distribution the model outputs.
Discrete convolution
Slides one signal over another, multiplying and summing at every offset. Reusing the same small kernel everywhere is what gives convolutional networks translation equivariance and their very small parameter count.
- The input signal, such as an image row.
- The kernel being slid across it.
- The output position.
Dropout
Randomly zeroes activations during training so no single unit can be relied on, then scales the survivors so the expected output is unchanged. At inference nothing is dropped, and the scaling is what makes the two regimes agree.
- A random binary mask, redrawn every step.
- The probability a unit is dropped.
- Element-wise multiplication.
GELU
Weights each input by the probability that a standard normal falls below it, giving a smooth curve that lets small negatives through instead of clamping them. It is the activation inside most transformer feed-forward blocks.
- The pre-activation input.
- The standard normal cumulative distribution function.
Layer normalization
Standardises each example across its own features, then lets the model rescale and shift the result. Because it never looks across the batch, it behaves identically at batch size one — which is why transformers use it and not batch norm.
- The mean over the feature dimension of this example.
- The variance over the same dimension.
- A learned per-feature scale.
- A learned per-feature shift.
ReLU
Pass positives through, clamp negatives to zero. Its gradient is one or nothing, so it neither saturates nor vanishes, and that single property is most of why very deep networks became trainable.
- The pre-activation input.
- Selects the larger of the two arguments.
Scaled dot-product attention
Every token proposes a query, every token advertises a key, and the match between them decides whose value gets read. Dividing by the square root of the key dimension keeps the dot products from growing with width and flattening the softmax gradient.
- The queries — what each position is looking for.
- The keys — what each position offers.
- The values — what actually gets mixed and returned.
- The key dimension, whose square root rescales the scores.
Sigmoid
Squashes any real number into the range zero to one, so the output can be read as a probability. Its gradient vanishes at both ends, which is exactly why deep networks moved to ReLU for hidden layers and kept sigmoid for the output.
- The input, an unbounded real number.
- The exponential that produces the S-shape.
Sinusoidal positional encoding
Attention is order-blind, so position has to be added to the input explicitly. Using sinusoids at geometrically spaced frequencies means relative offsets show up as fixed rotations, and positions longer than any seen in training still produce valid codes.
- The index of the token in the sequence.
- The index of the dimension within the encoding.
- The model's embedding dimension.
Softmax
Turns a vector of unbounded scores into a probability distribution: exponentiate so everything is positive, then divide by the total so the parts sum to one. Larger gaps between scores produce a sharper distribution.
- The vector of raw scores, usually called logits.
- The number of classes.
- The exponential, which makes every score positive and ordering-preserving.
Temperature sampling
Divides the logits before the softmax, which rescales how peaked the distribution is. Below one it sharpens toward the top choice; above one it flattens; at zero it becomes plain greedy decoding.
- The temperature.
- The logit for token i.
- The resulting sampling probability.