Chain rule
To differentiate a composition, multiply the outer rate of change by the inner one. Backpropagation is nothing more than this rule applied layer by layer from the loss backwards, reusing each partial result.
- The outer function.
- The inner function.
- The inner function's own derivative.
Derivative
The instantaneous rate of change: the slope of the line between two points on the curve as they slide together. Every gradient-based method is this quantity, generalised to many variables and computed in bulk.
- The function being differentiated.
- The shrinking gap between the two points.
- The value the ratio approaches as h goes to zero.
Directional derivative
How fast a function changes if you walk in one chosen direction. It is largest when that direction lines up with the gradient, which is the proof that the gradient really is the steepest way up.
- A unit vector giving the direction of travel.
- The gradient at the current point.
Fundamental theorem of calculus
Differentiation and integration undo each other: accumulating a rate of change over an interval gives you the net change. It is what makes integrals computable at all rather than merely definable.
- The rate of change being accumulated.
- The lower limit of the interval.
- The upper limit.
Gradient
The vector of all the partial derivatives, pointing in the direction of steepest increase. Training walks against it, which is the whole of what "descent" means.
- The gradient operator, read as del or nabla.
- A partial derivative — vary one input, hold the rest.
- A function of many variables returning one number.
Hessian
The matrix of second derivatives, describing curvature rather than slope. Its eigenvalues say whether a flat point is a minimum, a maximum or a saddle — and in high dimensions, saddles are what you actually hit.
- The Hessian matrix, symmetric for well-behaved functions.
- A second partial derivative.
Integration by parts
The product rule, read backwards. It trades one integral for another, and is worth the trade whenever the new one is simpler — the standard move for integrals mixing a polynomial with an exponential.
- The factor you choose to differentiate.
- The factor you choose to integrate.
Jacobian
The full matrix of partial derivatives for a function with several outputs — one row per output, one column per input. Autodiff never builds it in full; it multiplies by it one vector at a time.
- The i-th output of the function.
- The j-th input.
- The matrix collecting every such pairing.
Product rule
The derivative of a product is not the product of the derivatives: each factor takes a turn varying while the other is held still. The two terms are exactly those two turns, added together.
- The first factor.
- The second factor.
Taylor series
Rebuilds a function near a point from its derivatives there. Truncate after the first term and you have a linear approximation; after the second, a quadratic one — which is precisely what second-order optimisers are minimising.
- The point the expansion is centred on.
- The n-th derivative of f.
- n factorial, which keeps the terms from exploding.