Huber loss is the clever compromise between the two great regression losses. It behaves like the smooth, fast-training mean squared error when errors are small, and like the outlier-robust mean absolute error when errors are large. A single threshold, delta, decides where it switches from one behaviour to the other. The result is a loss that trains smoothly yet refuses to be bullied by outliers — which is why Huber loss is a favourite for real-world regression on messy data.
What is Huber loss?
Huber loss looks at the size of each residual and treats it differently depending on whether it is small or large. For residuals inside a threshold delta, it squares them just like mean squared error, giving a smooth, curving penalty. For residuals beyond delta, it switches to a straight line like mean absolute error, so a huge outlier only adds its size, not its square. The two pieces are stitched together so the loss and its slope match exactly at delta, making Huber loss continuous and differentiable everywhere.
In deep-learning libraries you will often meet it under the name smooth L1 loss, a close relative used heavily in object detection. The idea is identical: be gentle and smooth near zero, be linear and forgiving far away.
The Huber loss formula
L = δ(|y − ŷ| − ½δ) otherwise
The top branch is the squared (quadratic) region; the bottom branch is the linear region. The constant in the second line is there purely to make the two pieces meet smoothly at the threshold. The single hyperparameter delta is the dividing line between “small” and “large” errors, and choosing it well is the whole art of using Huber loss.
See it: quadratic near zero, linear far out
The blue curve is Huber loss with delta = 1. Look closely: near the centre it curves like a parabola, then straightens into two lines once the error passes ±1. The orange dashed gradient rises with the error inside the threshold, then flattens to a constant — exactly the robustness that tames outliers.
The delta parameter
Why Huber loss gives the best of both
The mean squared error trains beautifully because its gradient shrinks as you approach the answer, but it overreacts to outliers. The mean absolute error ignores outliers but has a constant gradient that makes the optimiser jitter near the minimum. Huber loss takes the good half of each: the shrinking, smooth gradient of squared error for the small residuals that dominate a well-fit model, and the bounded, linear penalty of absolute error for the rare large residuals. That is why it often trains as smoothly as squared error yet generalises better on noisy data.
Huber loss vs MSE vs MAE
| Loss | Small errors | Large errors | Outlier robust? |
|---|---|---|---|
| MSE | quadratic | quadratic | no |
| MAE | linear | linear | yes |
| Huber | quadratic | linear | yes |
In short, Huber loss is the interpolation between the other two, and delta is the knob that slides it from one toward the other.
When to use Huber loss
Reach for Huber loss on regression problems where the data is mostly clean but has occasional outliers, and you want smooth optimisation without letting those outliers dominate. It is common in robust regression, reinforcement learning (where it stabilises value-function updates), and object detection under the smooth-L1 name. If your data is pristine, plain squared error is simpler; if it is extremely noisy, the MAE may be enough.
Common mistakes to avoid
Huber loss in Python
import numpy as np
def huber(y, yhat, delta=1.0):
e = y - yhat
small = np.abs(e) <= delta
return np.where(small,
0.5 * e**2,
delta * (np.abs(e) - 0.5 * delta)).mean()
# frameworks: torch.nn.HuberLoss(delta=1.0), keras loss="huber"Scikit-learn also exposes it through HuberRegressor. See the Wikipedia Huber loss article for the derivation.
Putting it into practice
A practical way to adopt the loss is to start from mean squared error, notice that a few large residuals are dominating training, and then switch, choosing delta near the point where your error histogram’s bulk ends and its tail begins. Because the loss is differentiable everywhere, you can drop it straight into any gradient-based model with no special handling, and it will usually converge about as fast as squared error while producing a model that is visibly less distorted by outliers. As always, evaluate on a held-out set, and report the more interpretable root mean squared error or mean absolute error alongside the Huber value so readers who have never met this loss can still judge the result.
Where the robust compromise shines
The approach earns its keep in three settings in particular. In reinforcement learning it is the standard choice for value-function updates, because the occasional wildly wrong target these algorithms generate would otherwise destabilise training under a purely quadratic penalty; clipping the penalty to a linear tail keeps the updates sane. In computer vision it appears throughout object-detection models under the smooth-L1 name, taming the large box-regression errors that early training produces while still giving precise, smooth gradients once the boxes are roughly right. And in classic tabular regression it is the workhorse of robust estimators that must tolerate a few mislabelled or corrupted rows without letting them steer the entire fit.
The reasoning behind the compromise
Across all three settings the underlying logic is identical: the vast majority of residuals in a healthy model are small, so you want smooth, fast-converging behaviour there, while the rare enormous residual should be acknowledged but never allowed to dominate. Setting the threshold sensibly — near the boundary between your ordinary errors and your genuine outliers — is what lets a single objective serve all of these roles at once, and it is worth revisiting that threshold whenever you rescale the target or move to a noticeably noisier dataset. A good sanity check is to plot the distribution of your residuals and mark where the threshold falls: if it sits far out in the tail, you are barely getting any robustness; if it sits inside the main hump, you are throwing away the smooth behaviour that makes optimisation easy. The right place is almost always at the shoulder between the two.
A quick mental model
If you only remember one thing, picture the penalty curve as a bowl with straight sides: gently rounded at the bottom where the everyday errors live, then flattening into two straight ramps once the errors grow large. The rounded bottom is what gives smooth, well-behaved gradients that let an optimiser settle precisely; the straight ramps are what stop a freak data point from yanking the whole model toward itself. Everything else — the threshold, the two-piece formula, the smooth-L1 nickname — is just machinery for drawing that shape. Once you can see the bowl in your head, you can predict the behaviour on any dataset: tighten the threshold and the straight ramps start closer in, buying more robustness at the cost of some smoothness; widen it and the rounded region spreads out, recovering the fast convergence of a squared penalty. Choosing well is simply deciding how much of your error range should feel gentle and how much should feel firm, and that decision is best made by looking at where your real residuals actually fall.