Cheat sheet

Probability

13 formulas · Back to the formula library

  1. Bayes' theorem

    P(A∣B)=P(B∣A) P(A)P(B)P(A \mid B) = \frac{P(B \mid A)\, P(A)}{P(B)}

    Flips a conditional probability around: it converts what you would expect to see given a hypothesis into how much the thing you saw should move your belief in it. Every probabilistic update in machine learning is a version of this.

    Symbols

    P(A∣B)P(A \mid B)
    The posterior — belief in A after seeing B.
    P(B∣A)P(B \mid A)
    The likelihood — how expected B was under A.
    P(A)P(A)
    The prior — belief in A beforehand.
    P(B)P(B)
    The evidence, which normalises the result.
  2. Bernoulli distribution

    p(x)=θx(1−θ)1−x,x∈{0,1}p(x) = \theta^{x} (1-\theta)^{1-x}, \quad x \in \{0, 1\}

    A single yes-or-no trial with probability theta of yes. Its negative log is binary cross-entropy, which is the loss behind every binary classifier you have trained.

    Symbols

    θ\theta
    The probability of the outcome being one.
    xx
    The outcome, either zero or one.
  3. Binomial distribution

    P(k)=(nk)θk(1−θ)n−kP(k) = \binom{n}{k} \theta^{k} (1-\theta)^{n-k}

    Counts successes across n independent yes-or-no trials. The binomial coefficient accounts for every order the successes could have arrived in, which is where the combinatorics enters.

    Symbols

    nn
    The number of trials.
    kk
    The number of successes.
    (nk)\binom{n}{k}
    The number of ways to choose k from n.
  4. Central limit theorem

    Xˉn−μσ/n→dN(0,1)\frac{\bar{X}_n - \mu}{\sigma / \sqrt{n}} \xrightarrow{d} \mathcal{N}(0, 1)

    Averages of independent samples become normally distributed as the sample grows, no matter what the original distribution looked like. The square root of n is why halving your error costs four times the data.

    Symbols

    Xˉn\bar{X}_n
    The sample mean over n draws.
    μ\mu
    The true mean.
    σ\sigma
    The true standard deviation.
    N(0,1)\mathcal{N}(0, 1)
    The standard normal distribution.
  5. Conditional probability

    P(A∣B)=P(A∩B)P(B)P(A \mid B) = \frac{P(A \cap B)}{P(B)}

    Narrows the world to the cases where B happened, then asks how often A happens inside it. Independence is exactly the case where this narrowing changes nothing.

    Symbols

    A∩BA \cap B
    The event that both A and B occur.
    P(B)P(B)
    The probability of the condition, which must not be zero.
  6. Covariance

    Cov⁡(X,Y)=E ⁣[(X−E[X])(Y−E[Y])]\operatorname{Cov}(X, Y) = \mathbb{E}\!\left[(X - \mathbb{E}[X])(Y - \mathbb{E}[Y])\right]

    Measures whether two variables rise and fall together. It is positive when they move in step, negative when they move oppositely, and zero when knowing one tells you nothing linear about the other.

    Symbols

    XX
    The first random variable.
    YY
    The second random variable.
  7. Expected value

    E[X]=∑xx p(x)\mathbb{E}[X] = \sum_{x} x\, p(x)

    The long-run average of a random variable, each outcome weighted by how often it happens. Nearly every loss function in machine learning is an expectation you are approximating with a batch.

    Symbols

    XX
    The random variable.
    p(x)p(x)
    The probability of outcome x.
    E\mathbb{E}
    The expectation operator.
  8. Kullback–Leibler divergence

    DKL(P∥Q)=∑xP(x)log⁡P(x)Q(x)D_{\mathrm{KL}}(P \parallel Q) = \sum_{x} P(x) \log \frac{P(x)}{Q(x)}

    The extra bits you pay, on average, when you encode samples drawn from P using a code built for Q. It is zero only when the two distributions agree everywhere, and it is not symmetric — swapping P and Q gives a different number.

    Symbols

    PP
    The true distribution — the one the data actually comes from.
    QQ
    The approximating distribution — the model, or the code you built.
  9. Law of total probability

    P(A)=∑iP(A∣Bi) P(Bi)P(A) = \sum_{i} P(A \mid B_i)\, P(B_i)

    Splits a messy probability into cases you can actually compute, then weights each by how likely that case was. It is the denominator in Bayes' theorem whenever the evidence is not handed to you directly.

    Symbols

    BiB_i
    A set of cases that are exclusive and cover everything.
    P(A∣Bi)P(A \mid B_i)
    The probability of A within case i.
  10. Normal distribution

    p(x)=1σ2πexp⁡ ⁣(−(x−μ)22σ2)p(x) = \frac{1}{\sigma\sqrt{2\pi}} \exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)

    The bell curve: fully described by a mean and a spread, and the shape that sums of many independent effects converge to. Its log is a plain squared error, which is why Gaussian assumptions and L2 losses keep turning up together.

    Symbols

    μ\mu
    The mean, where the curve is centred.
    σ\sigma
    The standard deviation, how wide it is.
    exp⁡\exp
    The exponential function.
  11. Shannon entropy

    H(X)=−∑xp(x)log⁡p(x)H(X) = -\sum_{x} p(x) \log p(x)

    The average surprise in a random variable, and the floor on how few bits any lossless code can spend on it. A fair coin costs one bit; a coin that always lands heads costs nothing to describe.

    Symbols

    XX
    The random variable being measured.
    pp
    Its probability mass function.
    log⁡\log
    Base two for bits, base e for nats.
  12. Softmax

    σ(z)i=ezi∑j=1Kezj\sigma(\mathbf{z})_i = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}

    Turns a vector of unbounded scores into a probability distribution: exponentiate so everything is positive, then divide by the total so the parts sum to one. Larger gaps between scores produce a sharper distribution.

    Symbols

    z\mathbf{z}
    The vector of raw scores, usually called logits.
    KK
    The number of classes.
    ee
    The exponential, which makes every score positive and ordering-preserving.
  13. Variance

    Var⁡(X)=E ⁣[(X−E[X])2]\operatorname{Var}(X) = \mathbb{E}\!\left[(X - \mathbb{E}[X])^2\right]

    The average squared distance from the mean — how spread out a random variable is. Squaring is what makes it differentiable and additive over independent variables, and also what makes it sensitive to outliers.

    Symbols

    XX
    The random variable.
    E[X]\mathbb{E}[X]
    Its mean.
Esc
↑↓ navigate↵ open? shortcuts

Keyboard shortcuts

Press Esc to close

Anywhere

⌘K
Search and commands
/
Search
?
This sheet
Esc
Close

Reading a lesson

J
Next lesson
K
Previous lesson
B
Bookmark