Cosine similarity
The dot product with length divided out, leaving only the angle between two vectors. It runs from -1 to 1 and ignores magnitude entirely, which is exactly what you want when comparing embeddings whose scale carries no meaning.
- The first vector, often a query embedding.
- The second vector, often a stored embedding.
Determinant (2 by 2)
The signed factor by which a linear map scales area. A determinant of zero means the map flattens the plane onto a line and cannot be undone, which is the compact test for whether a matrix is invertible.
- The top-left entry.
- The bottom-right entry.
- The product of the off-diagonal entries.
Dot product
Multiplies two vectors into a single number that measures how much they point the same way. It is positive when they broadly agree, zero when they are perpendicular, and negative when they oppose — which is why almost every similarity score in machine learning starts here.
- The first vector.
- The second vector, of the same length.
- The angle between them.
Eigenvalue equation
Finds the directions a matrix leaves pointing the same way, stretching them by a single number. Those directions are the natural coordinate system of the map, which is what makes PCA and spectral methods work.
- A square matrix.
- An eigenvector — a direction the matrix preserves.
- The eigenvalue, the factor that direction is scaled by.
Euclidean norm
The straight-line length of a vector, straight out of Pythagoras. It is the default meaning of "how big" for a vector, and the quantity that gradient clipping, weight decay and distance metrics are all written in terms of.
- The vector being measured.
- Its i-th component.
- The number of dimensions.
Matrix multiplication
Each entry of the product is the dot product of one row of A with one column of B. Composing two linear maps this way is the single operation that dominates both training and inference cost, which is why hardware is designed around it.
- The left matrix, with shape m by n.
- The right matrix, with shape n by p.
- The shared inner dimension being summed over.
Outer product
Takes two vectors and produces a full matrix, every element of one multiplied by every element of the other. The result always has rank one, which is the idea low-rank adapters are built on.
- A column vector of length m.
- A column vector of length n.
- Transpose, turning a column into a row.
Singular value decomposition
Factors any matrix at all into a rotation, a stretch along axes, and another rotation. Truncating the stretch to its largest few values gives the best low-rank approximation there is, which is the basis of compression and of LoRA.
- Orthogonal matrix whose columns are the output directions.
- Diagonal matrix of singular values, largest first.
- Orthogonal matrix whose columns are the input directions.
Trace
The sum of a square matrix's diagonal. It equals the sum of the eigenvalues, survives a change of basis untouched, and lets you cycle matrices inside it — the trick behind many compact derivations of gradients.
- A square matrix.
- Its i-th diagonal entry.
Vector projection
The shadow one vector casts on another: the part of a that lies along b, with the perpendicular part discarded. Least squares is this idea applied to a whole subspace at once.
- The vector being projected.
- The direction it is projected onto.