Cheat sheet

Calculus

10 formulas · Back to the formula library

  1. Chain rule

    ddxf(g(x))=f′(g(x))⋅g′(x)\frac{d}{dx} f(g(x)) = f'(g(x)) \cdot g'(x)

    To differentiate a composition, multiply the outer rate of change by the inner one. Backpropagation is nothing more than this rule applied layer by layer from the loss backwards, reusing each partial result.

    Symbols

    ff
    The outer function.
    gg
    The inner function.
    g′(x)g'(x)
    The inner function's own derivative.
  2. Derivative

    f′(x)=lim⁡h→0f(x+h)−f(x)hf'(x) = \lim_{h \to 0} \frac{f(x+h) - f(x)}{h}

    The instantaneous rate of change: the slope of the line between two points on the curve as they slide together. Every gradient-based method is this quantity, generalised to many variables and computed in bulk.

    Symbols

    ff
    The function being differentiated.
    hh
    The shrinking gap between the two points.
    lim⁡\lim
    The value the ratio approaches as h goes to zero.
  3. Directional derivative

    Duf=∇f⋅uD_{\mathbf{u}} f = \nabla f \cdot \mathbf{u}

    How fast a function changes if you walk in one chosen direction. It is largest when that direction lines up with the gradient, which is the proof that the gradient really is the steepest way up.

    Symbols

    u\mathbf{u}
    A unit vector giving the direction of travel.
    ∇f\nabla f
    The gradient at the current point.
  4. Fundamental theorem of calculus

    ∫abf′(x) dx=f(b)−f(a)\int_a^b f'(x)\, dx = f(b) - f(a)

    Differentiation and integration undo each other: accumulating a rate of change over an interval gives you the net change. It is what makes integrals computable at all rather than merely definable.

    Symbols

    f′f'
    The rate of change being accumulated.
    aa
    The lower limit of the interval.
    bb
    The upper limit.
  5. Gradient

    ∇f=(∂f∂x1,…,∂f∂xn)\nabla f = \left( \frac{\partial f}{\partial x_1}, \dots, \frac{\partial f}{\partial x_n} \right)

    The vector of all the partial derivatives, pointing in the direction of steepest increase. Training walks against it, which is the whole of what "descent" means.

    Symbols

    ∇\nabla
    The gradient operator, read as del or nabla.
    ∂\partial
    A partial derivative — vary one input, hold the rest.
    ff
    A function of many variables returning one number.
  6. Hessian

    Hij=∂2f∂xi ∂xjH_{ij} = \frac{\partial^2 f}{\partial x_i \, \partial x_j}

    The matrix of second derivatives, describing curvature rather than slope. Its eigenvalues say whether a flat point is a minimum, a maximum or a saddle — and in high dimensions, saddles are what you actually hit.

    Symbols

    HH
    The Hessian matrix, symmetric for well-behaved functions.
    ∂2\partial^2
    A second partial derivative.
  7. Integration by parts

    ∫u dv=uv−∫v du\int u \, dv = uv - \int v \, du

    The product rule, read backwards. It trades one integral for another, and is worth the trade whenever the new one is simpler — the standard move for integrals mixing a polynomial with an exponential.

    Symbols

    uu
    The factor you choose to differentiate.
    dvdv
    The factor you choose to integrate.
  8. Jacobian

    Jij=∂fi∂xjJ_{ij} = \frac{\partial f_i}{\partial x_j}

    The full matrix of partial derivatives for a function with several outputs — one row per output, one column per input. Autodiff never builds it in full; it multiplies by it one vector at a time.

    Symbols

    fif_i
    The i-th output of the function.
    xjx_j
    The j-th input.
    JJ
    The matrix collecting every such pairing.
  9. Product rule

    (fg)′=f′g+fg′(fg)' = f'g + fg'

    The derivative of a product is not the product of the derivatives: each factor takes a turn varying while the other is held still. The two terms are exactly those two turns, added together.

    Symbols

    ff
    The first factor.
    gg
    The second factor.
  10. Taylor series

    f(x)=∑n=0∞f(n)(a)n!(x−a)nf(x) = \sum_{n=0}^{\infty} \frac{f^{(n)}(a)}{n!} (x-a)^n

    Rebuilds a function near a point from its derivatives there. Truncate after the first term and you have a linear approximation; after the second, a quadratic one — which is precisely what second-order optimisers are minimising.

    Symbols

    aa
    The point the expansion is centred on.
    f(n)f^{(n)}
    The n-th derivative of f.
    n!n!
    n factorial, which keeps the terms from exploding.
Esc
↑↓ navigate↵ open? shortcuts

Keyboard shortcuts

Press Esc to close

Anywhere

⌘K
Search and commands
/
Search
?
This sheet
Esc
Close

Reading a lesson

J
Next lesson
K
Previous lesson
B
Bookmark