Overfitting happens when a model has enough freedom to memorise noise. You can take that freedom away by using a smaller model — or you can leave the model large and make complexity expensive. Regularisation is the second approach, and it is usually the better one.
The idea behind regularisation
Normally a model minimises its loss function alone. Regularisation adds a second term:
$$\text{Total} = \text{Loss} + \lambda \times \text{Penalty}(w)$$
L2 regularisation (Ridge)
L2 penalises the sum of the squared weights:
$$\text{Penalty} = \sum_{i} w_i^2$$
Because the penalty is squared, a weight of 10 costs a hundred times more than a weight of 1. That makes L2 aggressive about very large weights and gentle about small ones — so it shrinks everything smoothly toward zero without ever quite reaching it.
This is exactly the squared L2 norm of the weight vector, which is where the name comes from.
L1 regularisation (Lasso)
L1 penalises the sum of the absolute weights:
$$\text{Penalty} = \sum_{i} |w_i|$$
The behaviour is genuinely different. With L1, the cost of shrinking a weight from 0.1 to 0 is the same as shrinking it from 10.1 to 10 — the pressure never eases off as the weight gets small. So weak weights get pushed all the way to exactly zero and stay there.
🧪 Why only L1 produces true zeros
L2: the derivative of w² is 2w. As w shrinks, the pull towards zero shrinks with it. At w = 0.001 the pull is almost nothing, so the weight drifts near zero forever without arriving.
L1: the derivative of |w| is ±1 — constant, no matter how small w gets. The pull towards zero never weakens, so it pushes the weight to exactly zero and pins it there.
That single difference is why L1 selects features and L2 does not.
Choosing between L1 and L2 regularisation
| L2 / Ridge | L1 / Lasso | |
|---|---|---|
| Penalty | Sum of w² | Sum of |w| |
| Effect on weights | Shrinks all, keeps all | Sets some to exactly zero |
| Feature selection | No | Yes, automatically |
| Correlated features | Shares weight between them | Picks one, drops the rest |
| Use when | Most features are somewhat useful | You suspect most features are useless |
If you cannot decide, Elastic Net uses both penalties at once with a mixing ratio — a sensible default when you have many correlated features.
Setting lambda
λ controls how strong the penalty is, and it is a hyperparameter — you choose it, the model does not.
- λ = 0 — no regularisation. The model is free to overfit.
- λ too small — barely any effect.
- λ too large — every weight is crushed towards zero and the model underfits, predicting roughly the average for everything.
Find it with cross-validation: try a range on a log scale — 0.001, 0.01, 0.1, 1, 10 — and keep whichever gives the best validation score. In scikit-learn, note that LogisticRegression and SVC use C, which is the inverse of λ, so a smaller C means more regularisation.
⚠️ Two things that catch people out
- You must scale your features first. The penalty is applied to raw weight sizes, so a feature measured in small units gets a large weight and is punished for it — purely because of its units. Standardise first, always.
- Do not regularise the bias. The bias term shifts the whole output and carries no complexity. Every good library excludes it by default — do not add it back by hand.
Regularisation beyond linear models
The same principle appears everywhere under different names: dropout in neural networks randomly switches off neurons so the network cannot depend on any one; early stopping halts training before memorisation begins; max_depth in a decision tree limits how many questions it can ask. All of them are the same idea — deliberately restrict the model so it generalises.
🔑 Key Takeaways
- Regularisation adds a penalty for large weights, making complexity something the model must justify.
- L2 (Ridge) shrinks all weights smoothly; L1 (Lasso) sets some to exactly zero.
- L1 selects features because its pull towards zero stays constant as weights shrink.
- λ is a hyperparameter — tune it on a log scale with cross-validation.
- Scale features before regularising, and never penalise the bias term.
Further reading
scikit-learn’s Ridge and Lasso documentation covers both penalties with built-in cross-validated variants, and Wikipedia’s article on regularisation places the technique in its wider mathematical context.
Where to go next
- Bias in machine learning — the last post in this sequence
- L1 and L2 norms — the maths the two penalties come from
- Overfitting and underfitting — the problem this solves
- Start Here — the full learning path