Cheat sheet

Inference

11 formulas · Back to the formula library

  1. Arithmetic intensity

    I=FLOPsbytes movedI = \frac{\text{FLOPs}}{\text{bytes moved}}

    How much arithmetic you do per byte you move. Below the hardware's compute-to- bandwidth ratio you are memory-bound and the tensor cores sit idle, which is the normal state of single-stream token generation.

    Symbols

    II
    Intensity, in FLOPs per byte.
    FLOPsFLOPs
    Floating-point operations performed.
    bytesmovedbytes moved
    Traffic between memory and the compute units.
  2. Chinchilla scaling law

    L(N,D)=E+ANα+BDβL(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}

    Loss falls as a power law in both parameters and training tokens, with an irreducible floor. Fitting it showed the models of the day were badly undertrained — roughly twenty tokens per parameter is the compute-optimal balance.

    Symbols

    NN
    The number of parameters.
    DD
    The number of training tokens.
    EE
    The irreducible loss, the entropy of the data itself.
    α\alpha
    The exponent governing returns to scale.
  3. KV-cache size

    MKV=2⋅L⋅H⋅dh⋅S⋅B⋅bM_{\text{KV}} = 2 \cdot L \cdot H \cdot d_h \cdot S \cdot B \cdot b

    Every generated token must keep its keys and values so later tokens can attend to them. The cache grows linearly with sequence length and batch size, and at long contexts it, not the weights, is what fills the GPU.

    Symbols

    LL
    The number of layers.
    HH
    The number of key/value heads.
    dhd_h
    The dimension of each head.
    SS
    The sequence length so far.
    BB
    The batch size.
    bb
    Bytes per value — two for fp16.
  4. Little's law

    L=λWL = \lambda W

    The average number of requests in a system equals arrival rate times time spent inside. For a serving stack it fixes the relationship between throughput, latency and how many requests must be in flight — you cannot improve all three at once.

    Symbols

    LL
    Requests resident in the system on average.
    λ\lambda
    Arrival rate, in requests per second.
    WW
    Average time a request spends in the system.
  5. Memory-bound token latency

    ttoken≈Mweights+MKVBWt_{\text{token}} \approx \frac{M_{\text{weights}} + M_{\text{KV}}}{\mathrm{BW}}

    When generating one token at a time, every weight has to be read from memory and barely any arithmetic is done with it. The floor on time per token is therefore set by bandwidth, not by FLOPs — which is the whole argument for batching.

    Symbols

    MweightsM_{\text{weights}}
    Bytes of model weights read per token.
    MKVM_{\text{KV}}
    Bytes of KV cache read per token.
    BW\mathrm{BW}
    Achievable memory bandwidth, in bytes per second.
  6. Model memory footprint

    Mweights=N⋅bM_{\text{weights}} = N \cdot b

    Parameters times bytes per parameter. At fp16 a 7-billion-parameter model needs about 14 GB before any activations or cache; at 4-bit it needs under 4. Training needs several times more, because the optimiser keeps its own copies.

    Symbols

    NN
    The number of parameters.
    bb
    Bytes per parameter — 4 at fp32, 2 at fp16, 0.5 at int4.
  7. Perplexity

    PPL=exp⁡ ⁣(−1N∑i=1Nlog⁡p(xi∣x<i))\mathrm{PPL} = \exp\!\left(-\frac{1}{N} \sum_{i=1}^{N} \log p(x_i \mid x_{<i})\right)

    The exponential of average cross-entropy, read as an effective number of choices. A perplexity of 20 means the model is about as uncertain as if it were picking uniformly among 20 tokens at every step.

    Symbols

    NN
    The number of tokens scored.
    p(xi∣x<i)p(x_i \mid x_{<i})
    The probability the model gave the true next token.
    exp⁡\exp
    Converts average log-loss back into a count of choices.
  8. Speculative decoding speedup

    E[tokens per step]=1−αk+11−α\mathbb{E}[\text{tokens per step}] = \frac{1 - \alpha^{k+1}}{1 - \alpha}

    A small draft model proposes k tokens and the large model verifies them all in one pass. The expected yield depends almost entirely on the acceptance rate, so a draft that agrees often beats a draft that is merely fast.

    Symbols

    α\alpha
    The probability the target model accepts a drafted token.
    kk
    How many tokens the draft model proposes per step.
  9. Temperature sampling

    pi=exp⁡(zi/T)∑jexp⁡(zj/T)p_i = \frac{\exp(z_i / T)}{\sum_{j} \exp(z_j / T)}

    Divides the logits before the softmax, which rescales how peaked the distribution is. Below one it sharpens toward the top choice; above one it flattens; at zero it becomes plain greedy decoding.

    Symbols

    TT
    The temperature.
    ziz_i
    The logit for token i.
    pip_i
    The resulting sampling probability.
  10. Top-p (nucleus) sampling

    Vp=min⁡{V′⊆V:∑x∈V′p(x)≥p}\mathcal{V}_p = \min \left\{ \mathcal{V}' \subseteq \mathcal{V} : \sum_{x \in \mathcal{V}'} p(x) \geq p \right\}

    Keeps the smallest set of tokens whose probabilities already add up to p, then samples within it. The cutoff moves with the model's confidence, which is what makes it behave better than a fixed top-k.

    Symbols

    V\mathcal{V}
    The full vocabulary.
    Vp\mathcal{V}_p
    The nucleus — the kept set.
    pp
    The cumulative probability mass to retain, often 0.9.
  11. Transformer forward FLOPs

    Cfwd≈2NDC_{\text{fwd}} \approx 2ND

    A forward pass costs roughly two floating-point operations per parameter per token — one multiply and one add. Training adds the backward pass for about three times more, giving the familiar six-N-D rule.

    Symbols

    NN
    The number of model parameters.
    DD
    The number of tokens processed.
    22
    One multiply and one add per parameter per token.
Esc
↑↓ navigate↵ open? shortcuts

Keyboard shortcuts

Press Esc to close

Anywhere

⌘K
Search and commands
/
Search
?
This sheet
Esc
Close

Reading a lesson

J
Next lesson
K
Previous lesson
B
Bookmark