Neural Network Explained: 4 Simple Ideas for Beginners

By /
⚡ TL;DR: A neural network is a stack of simple functions. Each neuron multiplies its inputs by learned weights, adds a bias, and passes the result through an activation function. Stack enough of them in layers and the network can represent almost any relationship.

If you have read about ReLU, sigmoid or softmax without ever being told what a neural network actually is, this page fixes that. It assumes you know what a machine learning model is and have met linear regression. Nothing else.

Start with one neuron

A single neuron is barely more complicated than linear regression. It takes some inputs, weights them, adds a bias, and applies one more function:

$$\text{output} = f(w_1x_1 + w_2x_2 + \dots + w_nx_n + b)$$

✅ Compare with linear regression: Linear regression computes wx + b and stops. A neuron computes the same thing, then wraps it in an activation function f. That single extra step is the entire difference — and it is what makes neural networks powerful.

The weighted sum inside is a dot product between the input vector and the weight vector. If that phrase means something to you, you already understand most of the arithmetic in deep learning.

Why a neural network needs the activation function

This is the part that is usually skipped, and it is the reason the whole field works.

Suppose you removed the activation functions and stacked three layers of pure weighted sums. Layer one computes a linear function of the input. Layer two computes a linear function of that — but a linear function of a linear function is just another linear function. So does layer three.

⚠️ Without activation functions, depth is worthless

A hundred stacked linear layers collapse mathematically into a single straight line. You would have spent enormous compute to reinvent linear regression. The activation function bends the output at each step, and it is the bending — repeated across layers — that lets a neural network describe curves, corners and complicated boundaries.

Neurons become layers, layers become a network

One neuron is not much use. Put several side by side, each with its own weights, and you have a layer. Feed one layer’s outputs into the next and you have a neural network.

Neural network diagram showing an input layer, hidden layer and output layer with every neuron connected by weighted links
Every line in this neural network is one weight the model learns. This small network already has 20 of them.
Input layer
One neuron per feature. Does no computation — it just holds your data.
Hidden layers
Where the work happens. “Hidden” only means you never observe their values directly.
Output layer
One neuron for regression; one per class for classification.

The diagram above has 3 inputs, 4 hidden neurons and 2 outputs. Count the connections: 3 × 4 = 12 into the hidden layer, plus 4 × 2 = 8 into the output layer, giving 20 weights, plus 6 biases. Every one of those 26 numbers is learned from data.

🧪 Scale check

That tiny network has 26 parameters. A network recognising handwritten digits typically has around 100,000. GPT-scale language models have hundreds of billions.

Nothing about the arithmetic changes — it is the same multiply, add, activate at every scale. There is simply a great deal more of it.

What “deep” means

A neural network with one hidden layer is called shallow. Two or more, and people say deep learning. That is the whole definition — there is no magic threshold.

Depth matters because each layer builds on the last. In an image network, early layers detect edges, middle layers combine edges into shapes like eyes and wheels, and later layers combine those into faces and cars. Each layer works with what the previous one found, which is far more efficient than trying to jump from raw pixels to “cat” in one step.

How a neural network learns

Exactly the same loop as every other model on this site:

  1. Forward pass. Push the data through the network and get predictions.
  2. Measure the error with a loss function — MSE for numbers, cross entropy for classes.
  3. Backward pass. Work out how much each of those 26 (or 26 billion) weights contributed to the error. This is backpropagation, and it is just the chain rule applied repeatedly.
  4. Update every weight a small step in the direction that reduces the error, then repeat.

The next two posts in this sequence take steps 1 and 3 apart in detail: how a single layer works, then backpropagation explained.

When a neural network is the wrong choice

⚠️ Neural networks are not the default answer

  • On ordinary table data — spreadsheets of numbers and categories — gradient boosting and decision trees usually beat neural networks, and train in seconds.
  • With a small dataset, a network with thousands of parameters will overfit immediately.
  • When you must explain the decision — for a loan or a diagnosis — a network gives you no readable reason.
Neural networks earn their keep on images, audio, and language, where the input is huge, unstructured, and full of patterns no human could hand-code.

🔑 Key Takeaways

  • A neuron is linear regression plus an activation function.
  • Without activation functions, stacking layers gains you nothing — they collapse to one straight line.
  • A neural network is neurons in layers: input, one or more hidden, output.
  • “Deep” simply means more than one hidden layer.
  • They excel on images, audio and text — and often lose to simpler models on spreadsheet data.

Further reading

3Blue1Brown’s neural networks series is the best visual explanation available, and Google’s crash course on neural networks lets you add layers in the browser and watch the decision boundary change.

Where to go next

Scroll to Top