Arithmetic intensity
How much arithmetic you do per byte you move. Below the hardware's compute-to- bandwidth ratio you are memory-bound and the tensor cores sit idle, which is the normal state of single-stream token generation.
- Intensity, in FLOPs per byte.
- Floating-point operations performed.
- Traffic between memory and the compute units.
Chinchilla scaling law
Loss falls as a power law in both parameters and training tokens, with an irreducible floor. Fitting it showed the models of the day were badly undertrained — roughly twenty tokens per parameter is the compute-optimal balance.
- The number of parameters.
- The number of training tokens.
- The irreducible loss, the entropy of the data itself.
- The exponent governing returns to scale.
KV-cache size
Every generated token must keep its keys and values so later tokens can attend to them. The cache grows linearly with sequence length and batch size, and at long contexts it, not the weights, is what fills the GPU.
- The number of layers.
- The number of key/value heads.
- The dimension of each head.
- The sequence length so far.
- The batch size.
- Bytes per value — two for fp16.
Little's law
The average number of requests in a system equals arrival rate times time spent inside. For a serving stack it fixes the relationship between throughput, latency and how many requests must be in flight — you cannot improve all three at once.
- Requests resident in the system on average.
- Arrival rate, in requests per second.
- Average time a request spends in the system.
Memory-bound token latency
When generating one token at a time, every weight has to be read from memory and barely any arithmetic is done with it. The floor on time per token is therefore set by bandwidth, not by FLOPs — which is the whole argument for batching.
- Bytes of model weights read per token.
- Bytes of KV cache read per token.
- Achievable memory bandwidth, in bytes per second.
Model memory footprint
Parameters times bytes per parameter. At fp16 a 7-billion-parameter model needs about 14 GB before any activations or cache; at 4-bit it needs under 4. Training needs several times more, because the optimiser keeps its own copies.
- The number of parameters.
- Bytes per parameter — 4 at fp32, 2 at fp16, 0.5 at int4.
Perplexity
The exponential of average cross-entropy, read as an effective number of choices. A perplexity of 20 means the model is about as uncertain as if it were picking uniformly among 20 tokens at every step.
- The number of tokens scored.
- The probability the model gave the true next token.
- Converts average log-loss back into a count of choices.
Speculative decoding speedup
A small draft model proposes k tokens and the large model verifies them all in one pass. The expected yield depends almost entirely on the acceptance rate, so a draft that agrees often beats a draft that is merely fast.
- The probability the target model accepts a drafted token.
- How many tokens the draft model proposes per step.
Temperature sampling
Divides the logits before the softmax, which rescales how peaked the distribution is. Below one it sharpens toward the top choice; above one it flattens; at zero it becomes plain greedy decoding.
- The temperature.
- The logit for token i.
- The resulting sampling probability.
Top-p (nucleus) sampling
Keeps the smallest set of tokens whose probabilities already add up to p, then samples within it. The cutoff moves with the model's confidence, which is what makes it behave better than a fixed top-k.
- The full vocabulary.
- The nucleus — the kept set.
- The cumulative probability mass to retain, often 0.9.
Transformer forward FLOPs
A forward pass costs roughly two floating-point operations per parameter per token — one multiply and one add. Training adds the backward pass for about three times more, giving the familiar six-N-D rule.
- The number of model parameters.
- The number of tokens processed.
- One multiply and one add per parameter per token.