Cheat sheet

Linear algebra

10 formulas · Back to the formula library

  1. Cosine similarity

    sim⁡(a,b)=a⋅b∥a∥ ∥b∥\operatorname{sim}(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{a}\|\,\|\mathbf{b}\|}

    The dot product with length divided out, leaving only the angle between two vectors. It runs from -1 to 1 and ignores magnitude entirely, which is exactly what you want when comparing embeddings whose scale carries no meaning.

    Symbols

    a\mathbf{a}
    The first vector, often a query embedding.
    b\mathbf{b}
    The second vector, often a stored embedding.
  2. Determinant (2 by 2)

    det⁡(abcd)=ad−bc\det\begin{pmatrix} a & b \\ c & d \end{pmatrix} = ad - bc

    The signed factor by which a linear map scales area. A determinant of zero means the map flattens the plane onto a line and cannot be undone, which is the compact test for whether a matrix is invertible.

    Symbols

    aa
    The top-left entry.
    dd
    The bottom-right entry.
    bcbc
    The product of the off-diagonal entries.
  3. Dot product

    a⋅b=∑i=1naibi=∥a∥ ∥b∥cos⁡θ\mathbf{a} \cdot \mathbf{b} = \sum_{i=1}^{n} a_i b_i = \|\mathbf{a}\|\,\|\mathbf{b}\|\cos\theta

    Multiplies two vectors into a single number that measures how much they point the same way. It is positive when they broadly agree, zero when they are perpendicular, and negative when they oppose — which is why almost every similarity score in machine learning starts here.

    Symbols

    a\mathbf{a}
    The first vector.
    b\mathbf{b}
    The second vector, of the same length.
    θ\theta
    The angle between them.
  4. Eigenvalue equation

    Av=λv,v≠0A\mathbf{v} = \lambda \mathbf{v}, \quad \mathbf{v} \neq \mathbf{0}

    Finds the directions a matrix leaves pointing the same way, stretching them by a single number. Those directions are the natural coordinate system of the map, which is what makes PCA and spectral methods work.

    Symbols

    AA
    A square matrix.
    v\mathbf{v}
    An eigenvector — a direction the matrix preserves.
    λ\lambda
    The eigenvalue, the factor that direction is scaled by.
  5. Euclidean norm

    ∥x∥2=∑i=1nxi2\|\mathbf{x}\|_2 = \sqrt{\sum_{i=1}^{n} x_i^2}

    The straight-line length of a vector, straight out of Pythagoras. It is the default meaning of "how big" for a vector, and the quantity that gradient clipping, weight decay and distance metrics are all written in terms of.

    Symbols

    x\mathbf{x}
    The vector being measured.
    xix_i
    Its i-th component.
    nn
    The number of dimensions.
  6. Matrix multiplication

    (AB)ij=∑k=1nAikBkj(AB)_{ij} = \sum_{k=1}^{n} A_{ik} B_{kj}

    Each entry of the product is the dot product of one row of A with one column of B. Composing two linear maps this way is the single operation that dominates both training and inference cost, which is why hardware is designed around it.

    Symbols

    AA
    The left matrix, with shape m by n.
    BB
    The right matrix, with shape n by p.
    kk
    The shared inner dimension being summed over.
  7. Outer product

    (uv⊤)ij=uivj(\mathbf{u} \mathbf{v}^{\top})_{ij} = u_i v_j

    Takes two vectors and produces a full matrix, every element of one multiplied by every element of the other. The result always has rank one, which is the idea low-rank adapters are built on.

    Symbols

    u\mathbf{u}
    A column vector of length m.
    v\mathbf{v}
    A column vector of length n.
    ⊤\top
    Transpose, turning a column into a row.
  8. Singular value decomposition

    A=UΣV⊤A = U \Sigma V^{\top}

    Factors any matrix at all into a rotation, a stretch along axes, and another rotation. Truncating the stretch to its largest few values gives the best low-rank approximation there is, which is the basis of compression and of LoRA.

    Symbols

    UU
    Orthogonal matrix whose columns are the output directions.
    Σ\Sigma
    Diagonal matrix of singular values, largest first.
    VV
    Orthogonal matrix whose columns are the input directions.
  9. Trace

    tr⁡(A)=∑i=1nAii\operatorname{tr}(A) = \sum_{i=1}^{n} A_{ii}

    The sum of a square matrix's diagonal. It equals the sum of the eigenvalues, survives a change of basis untouched, and lets you cycle matrices inside it — the trick behind many compact derivations of gradients.

    Symbols

    AA
    A square matrix.
    AiiA_{ii}
    Its i-th diagonal entry.
  10. Vector projection

    proj⁡ba=a⋅bb⋅bb\operatorname{proj}_{\mathbf{b}}\mathbf{a} = \frac{\mathbf{a} \cdot \mathbf{b}}{\mathbf{b} \cdot \mathbf{b}} \mathbf{b}

    The shadow one vector casts on another: the part of a that lies along b, with the perpendicular part discarded. Least squares is this idea applied to a whole subspace at once.

    Symbols

    a\mathbf{a}
    The vector being projected.
    b\mathbf{b}
    The direction it is projected onto.
Esc
↑↓ navigate↵ open? shortcuts

Keyboard shortcuts

Press Esc to close

Anywhere

⌘K
Search and commands
/
Search
?
This sheet
Esc
Close

Reading a lesson

J
Next lesson
K
Previous lesson
B
Bookmark