Channel capacity
The highest rate at which information can cross a noisy channel with errors driven arbitrarily low. Shannon's result is that below this rate reliable communication is possible and above it is not — with no gentle slope in between.
- The capacity, in bits per channel use.
- The mutual information between input and output.
- Maximised over every way of choosing inputs.
Conditional entropy
How much uncertainty is left in Y once you already know X. It reaches zero when X determines Y outright, and stays at the full entropy of Y when X tells you nothing about it.
- The joint entropy of both variables.
- The entropy of X alone.
Cross-entropy
The average number of bits needed to encode data from P using a code optimised for Q. It splits into the entropy of P plus the divergence from P to Q, which is why minimising it over a model is the same as minimising that divergence.
- The true distribution, usually a one-hot label in practice.
- The predicted distribution the model outputs.
Jensen–Shannon divergence
A symmetric, always-finite cousin of the KL divergence, built by comparing both distributions against their average. Its square root is a true distance, which KL divergence never is.
- The mixture, equal to half of P plus half of Q.
- The first distribution.
- The second distribution.
Joint entropy
The total uncertainty in two variables considered together. It is never more than the sum of the separate entropies, and the shortfall is exactly how much the two share.
- The joint probability of both outcomes.
- The first random variable.
- The second random variable.
Kullback–Leibler divergence
The extra bits you pay, on average, when you encode samples drawn from P using a code built for Q. It is zero only when the two distributions agree everywhere, and it is not symmetric — swapping P and Q gives a different number.
- The true distribution — the one the data actually comes from.
- The approximating distribution — the model, or the code you built.
Mutual information
How many bits knowing one variable saves you on the other. Unlike correlation it catches any kind of dependence, not just linear ones, and it is symmetric in its two arguments.
- The joint distribution.
- What the joint would be if the two were independent.
Perplexity
The exponential of average cross-entropy, read as an effective number of choices. A perplexity of 20 means the model is about as uncertain as if it were picking uniformly among 20 tokens at every step.
- The number of tokens scored.
- The probability the model gave the true next token.
- Converts average log-loss back into a count of choices.
Self-information
The surprise of one particular outcome. A certain event carries none; a one-in-a- million event carries about twenty bits. Entropy is just this quantity averaged over every outcome.
- A single outcome.
- Its probability.
Shannon entropy
The average surprise in a random variable, and the floor on how few bits any lossless code can spend on it. A fair coin costs one bit; a coin that always lands heads costs nothing to describe.
- The random variable being measured.
- Its probability mass function.
- Base two for bits, base e for nats.