The previous post explained what a neural network is. This one opens up a single layer and works through the arithmetic by hand, because once you have done it once, deep learning stops feeling mysterious.
The four steps of a neural network layer
Written as one line:
$$a = f(Wx + b)$$
Working one layer out by hand
Take a layer with 3 inputs and 2 neurons. The weight matrix is 2 × 3 — two rows because there are two neurons, three columns because each neuron has one weight per input.
🧪 Worked example: one layer, start to finish
Input x = [2, 1, 3]. Weights W = [[0.5, −1.0, 0.2], [1.5, 0.0, −0.5]]. Biases b = [1.0, −2.0]. Activation: ReLU.
Step 1 — multiply (the dot products)
Neuron 1: (0.5×2) + (−1.0×1) + (0.2×3) = 1.0 − 1.0 + 0.6 = 0.6
Neuron 2: (1.5×2) + (0.0×1) + (−0.5×3) = 3.0 + 0 − 1.5 = 1.5
Step 2 — add the bias
Neuron 1: 0.6 + 1.0 = 1.6 · Neuron 2: 1.5 − 2.0 = −0.5
Step 3 — apply ReLU, which is max(0, z)
Neuron 1: max(0, 1.6) = 1.6 · Neuron 2: max(0, −0.5) = 0
Output: a = [1.6, 0]. That vector is what the next layer receives.
Notice what happened to neuron 2: it produced a negative value, and ReLU flattened it to zero. That neuron contributes nothing to the next layer for this particular input. On a different input it might fire strongly. Different neurons switching on and off for different inputs is how a neural network layer represents varied patterns.
Why the shapes must line up
Most errors beginners hit in NumPy or PyTorch are shape errors, and they all come from this rule:
| Object | Shape | Why |
|---|---|---|
| Input x | 3 × 1 | One value per input feature |
| Weights W | 2 × 3 | Rows = neurons out, columns = inputs in |
| Wx | 2 × 1 | The inner 3s cancel |
| Bias b | 2 × 1 | One per neuron |
| Output a | 2 × 1 | Becomes the next layer’s input |
The rule is the one from matrix multiplication dimensions: the columns of W must equal the rows of x. If a framework complains about a shape mismatch, this is almost always what it means.
Doing a whole batch at once
In practice you do not push one example through at a time. You stack many rows into a matrix and process them together — this is why GPUs help so much, and why the linear algebra matters more than the calculus in day-to-day work.
With 32 examples at once, x becomes 32 × 3, and one matrix multiplication produces all 32 outputs. The arithmetic per example is unchanged; it just happens in parallel.
⚠️ Why the bias matters more than it looks
What the layer is actually learning
Each row of W is one neuron’s opinion about which inputs matter. A large positive weight means “this input strongly increases my output”; a large negative one means the opposite; near zero means “I ignore this”.
Training adjusts every one of those numbers. In the worked example the weights were given — in reality they start random and are corrected repeatedly by backpropagation, which is the next post in this sequence.
🔑 Key Takeaways
- Every neural network layer computes a = f(Wx + b) — multiply, add, activate.
- The weight matrix has one row per neuron and one column per input.
- Shape errors in code almost always mean those dimensions do not line up.
- ReLU setting a neuron to zero is normal — different neurons fire for different inputs.
- The bias frees the layer from being forced through the origin.
Further reading
PyTorch’s nn.Linear documentation is exactly the Wx + b step described here, and NumPy’s matmul reference covers the batching rules if you want to implement a layer yourself.
Where to go next
- Backpropagation explained — the next post in this sequence
- What is a neural network? — if you skipped straight here
- Matrix multiplication dimensions — the shape rule in full
- Start Here — the full learning path