Bayes' theorem
Flips a conditional probability around: it converts what you would expect to see given a hypothesis into how much the thing you saw should move your belief in it. Every probabilistic update in machine learning is a version of this.
- The posterior — belief in A after seeing B.
- The likelihood — how expected B was under A.
- The prior — belief in A beforehand.
- The evidence, which normalises the result.
Bernoulli distribution
A single yes-or-no trial with probability theta of yes. Its negative log is binary cross-entropy, which is the loss behind every binary classifier you have trained.
- The probability of the outcome being one.
- The outcome, either zero or one.
Binomial distribution
Counts successes across n independent yes-or-no trials. The binomial coefficient accounts for every order the successes could have arrived in, which is where the combinatorics enters.
- The number of trials.
- The number of successes.
- The number of ways to choose k from n.
Central limit theorem
Averages of independent samples become normally distributed as the sample grows, no matter what the original distribution looked like. The square root of n is why halving your error costs four times the data.
- The sample mean over n draws.
- The true mean.
- The true standard deviation.
- The standard normal distribution.
Conditional probability
Narrows the world to the cases where B happened, then asks how often A happens inside it. Independence is exactly the case where this narrowing changes nothing.
- The event that both A and B occur.
- The probability of the condition, which must not be zero.
Covariance
Measures whether two variables rise and fall together. It is positive when they move in step, negative when they move oppositely, and zero when knowing one tells you nothing linear about the other.
- The first random variable.
- The second random variable.
Expected value
The long-run average of a random variable, each outcome weighted by how often it happens. Nearly every loss function in machine learning is an expectation you are approximating with a batch.
- The random variable.
- The probability of outcome x.
- The expectation operator.
Kullback–Leibler divergence
The extra bits you pay, on average, when you encode samples drawn from P using a code built for Q. It is zero only when the two distributions agree everywhere, and it is not symmetric — swapping P and Q gives a different number.
- The true distribution — the one the data actually comes from.
- The approximating distribution — the model, or the code you built.
Law of total probability
Splits a messy probability into cases you can actually compute, then weights each by how likely that case was. It is the denominator in Bayes' theorem whenever the evidence is not handed to you directly.
- A set of cases that are exclusive and cover everything.
- The probability of A within case i.
Normal distribution
The bell curve: fully described by a mean and a spread, and the shape that sums of many independent effects converge to. Its log is a plain squared error, which is why Gaussian assumptions and L2 losses keep turning up together.
- The mean, where the curve is centred.
- The standard deviation, how wide it is.
- The exponential function.
Shannon entropy
The average surprise in a random variable, and the floor on how few bits any lossless code can spend on it. A fair coin costs one bit; a coin that always lands heads costs nothing to describe.
- The random variable being measured.
- Its probability mass function.
- Base two for bits, base e for nats.
Softmax
Turns a vector of unbounded scores into a probability distribution: exponentiate so everything is positive, then divide by the total so the parts sum to one. Larger gaps between scores produce a sharper distribution.
- The vector of raw scores, usually called logits.
- The number of classes.
- The exponential, which makes every score positive and ordering-preserving.
Variance
The average squared distance from the mean — how spread out a random variable is. Squaring is what makes it differentiable and additive over independent variables, and also what makes it sensitive to outliers.
- The random variable.
- Its mean.