What is a neural network?
Layers, weights, biases and activations — the whole forward pass, drawn on one handwritten digit.
This lesson is part of a preview roadmap.
The engine around it is real — progress, notes and bookmarks all work — but the video has not been through the sourcing and review protocol a shipped lesson goes through. See the roadmap for what that means.
What is a neural network?
The player loads from youtube-nocookie.com when you press play. Nothing reaches YouTube before that.
Not started
What you'll learn
- A layer is a matrix multiplication, a bias, and a nonlinearity, applied in that order.
- Weights are what the network learns; the architecture is what you choose before it learns anything.
- Without the nonlinearity, any stack of layers collapses into a single layer.
The architecture, before any learning
This node is deliberately about the forward pass only. A network that has learned nothing still computes something, and understanding what it computes — a chain of matrix multiplications with a squashing function between them — makes the training story in the next nodes much shorter.
Where this sits
Everything so far has been fixed. The next node introduces the cost function and the idea of nudging weights downhill, which is where a network stops being a formula and starts being a thing that learns.
Check yourself
What would happen to a ten-layer network if you removed every activation function?
It would become mathematically identical to a one-layer network, because a composition of linear maps is itself linear.
What is the bias term for?
It shifts the threshold at which a unit starts responding, so a unit is not forced to pass through the origin.
Go deeper
The maths underneath
Notes
Notes are saved to your account.
Everything else on this page works without one — progress and bookmarks keep themselves in this browser and come with you when you sign in. Notes are the one thing worth not trusting to a browser you might clear.
Your notes could not be loaded.