Feature scaling is one of those steps that seems like fussy housekeeping until it silently ruins a model. This page explains what it is, the three main methods, and — the part usually left out — how to decide which to use and when you can skip it.
Why feature scaling matters
Suppose you are predicting house prices from two features: number of bedrooms (range 1 to 5) and floor area in cm² (range 400,000 to 2,000,000). Both are useful. But numerically one is hundreds of thousands of times larger, and several algorithms take that literally.
Anything that measures distance between rows — k-nearest neighbours, k-means, anything using cosine similarity — finds that floor area dominates the arithmetic completely. Two houses differing by 3 bedrooms differ by 3 in that dimension; two differing by one square metre differ by 10,000. Bedrooms are ignored, not because they are unimportant but because their units are small.
Gradient descent suffers differently. With features on very different scales the loss surface becomes a long narrow valley rather than a round bowl, so gradient descent zigzags across the steep direction while creeping along the shallow one — converging slowly, or diverging entirely unless the learning rate is tiny.
Method 1 — normalisation (min–max feature scaling)
Squash every feature into a fixed range, usually 0 to 1:
$$x’ = \frac{x – x_{\min}}{x_{\max} – x_{\min}}$$
The smallest value becomes 0, the largest becomes 1. With bedrooms ranging 1 to 5, a 3-bedroom house becomes (3 − 1) ÷ (5 − 1) = 0.5.
⚠️ Outliers break min–max scaling
Method 2 — standardisation (z-score feature scaling)
Recentre each feature to a mean of 0 and a standard deviation of 1:
$$x’ = \frac{x – \mu}{\sigma}$$
A value of 0 now means “average”; +1 means “one standard deviation above average”. Values are not bounded, so genuine outliers stay far out — often exactly what you want. This is the same z-score from statistics.
Method 3 — robust scaling
If your data has serious outliers you do not want to remove, subtract the median and divide by the interquartile range. Because the median and IQR barely move when you add extreme values, the scaling stays sensible.
| Method | Output range | Outliers | Typical use |
|---|---|---|---|
| Normalisation | Bounded, 0 to 1 | Poor — they compress everything else | Image pixels, bounded neural net inputs |
| Standardisation | Unbounded, centred on 0 | Good — they stay distinguishable | Most classical ML: regression, SVM, PCA |
| Robust | Unbounded, centred on median | Best — barely affected | Data with heavy outliers |
Which algorithms need feature scaling
If you are using a random forest, you can skip feature scaling entirely. PCA is the case people forget: it maximises variance, so unscaled large-range features dominate every component — see step-by-step PCA with NumPy.
The feature scaling mistake that inflates your results
This matters more than the choice of method. Fit the scaler on the training set only.
The wrong order — scale the whole dataset, then split — means the mean and standard deviation were computed using test rows. Information has leaked into training and your evaluation is no longer honest. The effect is usually small, but it is exactly the invisible optimism that makes a model look better in development than in production.
🧪 The correct order, every time
- Split into training and test sets.
- Compute the mean and standard deviation from the training set only.
- Apply those numbers to transform the training set.
- Apply those same numbers again to the test set — never recompute them.
fit_transform on training data and transform alone on test data. The asymmetry is deliberate.The same rule applies in production: save the training mean and standard deviation alongside the model, because every future prediction must be scaled with identical numbers.
Two smaller traps
- Scaling the target. Usually unnecessary — and if you do it, you must reverse the transformation on predictions, or your model will confidently report that a house costs 0.34.
- Scaling one-hot encoded columns. A column that is already 0 or 1 does not need feature scaling. Standardising it produces odd values and makes coefficients harder to interpret, for no benefit.
🔑 Key Takeaways
- Feature scaling matters whenever an algorithm measures distance or is trained by gradient descent.
- Standardisation is the default; normalisation for bounded ranges; robust scaling when outliers dominate.
- Tree-based models need no scaling at all.
- Fit the scaler on training data only, then apply the same numbers to test data and to production.
- Do not scale one-hot columns, and reverse any scaling applied to the target.
Further reading
scikit-learn’s preprocessing guide documents every scaler mentioned here, and its comparison of scalers on real data shows visually how each behaves when outliers are present.
Where to go next
- Standard deviation calculator — check your feature scaling by hand
- Training data vs test data — the split that scaling must respect
- Overfitting and underfitting — where regularisation, and therefore scaling, matters
- Start Here — the full learning path