Linear regression is where almost every machine learning course begins, and for good reason: it is simple enough to work through by hand, yet it contains nearly every idea you will meet later — parameters, a loss function, gradient descent, and evaluation. Understand this one properly and the rest gets much easier.
What linear regression actually does
Linear regression predicts a number — a house price, a temperature, an exam score — from one or more input features. It does this by assuming the relationship is roughly a straight line:
$$\hat{y} = wx + b$$
If that looks like the y = mx + c from school, it is exactly that. The difference is that you are not given w and b — the algorithm works them out from data. That is what makes it a machine learning model.
How linear regression chooses the line
Infinitely many lines could be drawn through a cloud of points. Linear regression picks one specific line, using a rule called least squares.
For each house, take the vertical distance between the real price and the line’s prediction. That distance is the residual. Square it, add up all the squares, and you have the total error:
$$\text{MSE} = \frac{1}{n}\sum_{i=1}^{n}(y_i – \hat{y}_i)^2$$
That is mean squared error, and it is the loss function linear regression minimises.
🧪 Why square the errors?
Second, squaring punishes big misses disproportionately: being wrong by 20 costs four times as much as being wrong by 10, not twice as much. That pushes the line towards avoiding large errors, which is usually what you want. When it is not what you want — because outliers are dragging the line around — use Huber loss or mean absolute error instead.
Five steps to fit a linear regression by hand
With one feature you can compute the answer exactly, no iteration required. Here is the whole procedure on four data points.
The slope formula is:
$$w = \frac{\sum (x_i – \bar{x})(y_i – \bar{y})}{\sum (x_i – \bar{x})^2}, \qquad b = \bar{y} – w\bar{x}$$
🧪 Worked example: four houses
| Area x | Price y | x − x̄ | y − ȳ | product | (x − x̄)² |
|---|---|---|---|---|---|
| 50 | 150 | −25 | −60 | 1500 | 625 |
| 60 | 180 | −15 | −30 | 450 | 225 |
| 90 | 240 | 15 | 30 | 450 | 225 |
| 100 | 270 | 25 | 60 | 1500 | 625 |
| x̄ = 75 | ȳ = 210 | Σ = 3900 | Σ = 1700 |
So the fitted line is price = 2.294 × area + 37.9. A 75 m² house predicts £210k — exactly the mean, which is always true: the least-squares line passes through (x̄, ȳ).
Check it yourself with the linear regression calculator.
Multiple linear regression: more than one feature
Real problems have several inputs. The idea does not change — each feature gets its own weight:
$$\hat{y} = w_1x_1 + w_2x_2 + \dots + w_nx_n + b$$
With two features you are fitting a plane rather than a line; beyond that, a hyperplane you cannot picture. The arithmetic stays identical, which is exactly why it is expressed with matrices — one dot product handles all features at once. This is where your linear algebra starts paying off.
⚠️ With multiple features, scale them first
The assumptions linear regression makes
Linear regression is not magic — it assumes things about your data, and when those assumptions break, the predictions quietly become unreliable.
| Assumption | What breaks it | How to spot it |
|---|---|---|
| Linearity | The real relationship curves | Plot residuals — a pattern means curvature |
| Independence | Rows influence each other (time series) | Residuals correlated in order |
| Constant variance | Errors grow with the prediction | Residual plot fans out like a cone |
| No strong collinearity | Two features carry the same information | Check the correlation matrix |
How to tell if your linear regression is any good
- R² (coefficient of determination) — the share of the variation in y your model explains. 0 means no better than predicting the average; 1 means perfect. On real data, 0.6 is often respectable.
- RMSE — average error in the original units. If RMSE is £24,000, your price predictions are typically about £24k out. Far more interpretable than R² for explaining results to someone else.
- MAE — the same idea but less sensitive to a few large misses.
Whichever you choose, measure it on held-out test data. R² on the training set will only ever flatter you.
Linear regression vs logistic regression
Common linear regression mistakes
⚠️ Four traps
- Reading correlation as cause. A positive weight on ice-cream sales when predicting drownings does not mean ice cream drowns people. Both rise in summer.
- Extrapolating beyond your data. A model fitted on 50–150 m² houses says nothing trustworthy about a 900 m² mansion.
- Ignoring outliers. Because errors are squared, a single wild point can visibly tilt the whole line.
- Trusting R² alone. Adding any feature — even random noise — never decreases R² on training data. Use adjusted R², or check the test set.
🔑 Key Takeaways
- Linear regression predicts a number by fitting a straight line, ŷ = wx + b.
- It chooses the line by minimising the sum of squared residuals — least squares.
- With one feature you can solve it exactly by hand; with many, it becomes matrix arithmetic.
- Judge it with RMSE or MAE on test data, not R² on training data.
- Check the assumptions — especially linearity and outliers — before trusting the coefficients.
Further reading
scikit-learn’s linear models guide documents ordinary least squares alongside its regularised cousins, Ridge and Lasso, and Wikipedia’s article on ordinary least squares gives the full derivation if you want the algebra behind the formula above.
Where to go next
- Logistic vs linear regression — when your answer is a category
- Overfitting and underfitting — what happens when you add too many features
- Mean squared error — the loss function in detail
- Start Here — the full learning path