Loss functions are how a machine-learning model measures its own mistakes. Every model learns by making the number a loss function returns as small as possible, so the loss you pick literally defines what “good” means for your model. Choose the wrong one and even a powerful model optimises for the wrong thing. This guide maps the main loss functions — for both regression and classification — shows how they differ, and gives you a simple rule for choosing.
What loss functions do
Training a model is an optimisation loop: make a prediction, measure how wrong it is, nudge the parameters to be a little less wrong, and repeat. The thing doing the measuring is the loss function, and its gradient is the signal that tells the optimiser which way to move. Because the whole of learning is just minimising this number, the choice of loss quietly decides everything — whether the model chases outliers, whether it outputs trustworthy probabilities, and whether it converges quickly or crawls. Two families cover almost every everyday case: regression losses for predicting continuous numbers, and classification losses for predicting categories.
The main loss functions at a glance
| Loss | Task | Best when |
|---|---|---|
| MSE | regression | large errors matter most |
| MAE | regression | outliers should be ignored |
| RMSE | regression | a readable headline metric |
| Huber | regression | smooth but outlier-robust |
| Cross-entropy | classification | probabilities, most models |
| Binary cross-entropy | classification | yes/no and multi-label |
| Hinge | classification | SVMs, wide margins |
See the regression losses together
Here are three regression penalties as the prediction error grows. MSE (blue) curves upward fastest, punishing big misses; MAE (purple) rises in a straight line; Huber (green) blends the two — curved near zero, linear far out.
Regression losses
When the target is a number, the question is how to weigh a big miss against a small one. The mean squared error squares the gap, so large misses dominate and the model works hard to avoid them — great unless those large values are just noise. The mean absolute error weighs every miss in proportion, shrugging off outliers. Huber is the compromise, smooth for small errors and linear for large ones, and RMSE is simply the square root of the squared error, reported because it reads in the target’s own units.
Classification losses
When the target is a category, most models use cross-entropy, which scores the probability assigned to the correct class and punishes confident mistakes hard. Its two-class form is binary cross-entropy, also called log loss, paired with a sigmoid. Support vector machines instead use hinge loss, which cares about a decisive margin rather than a probability. The right classification loss follows from your output layer: softmax and sigmoid want cross-entropy, an SVM wants hinge.
How to choose a loss function
1. Numbers or classes? Regression → a squared or absolute loss; classification → cross-entropy or hinge.
2. Outliers? If big errors are real and costly, MSE; if they are noise, MAE or Huber.
3. Need probabilities? Yes → cross-entropy; only a decisive boundary → hinge.
Loss vs metric: a crucial distinction
Loss functions in Python (Keras)
model.compile(loss="mse") # regression
model.compile(loss="mae") # robust regression
model.compile(loss="binary_crossentropy") # yes/no
model.compile(loss="categorical_crossentropy") # multi-classPyTorch offers the same set as classes: nn.MSELoss, nn.L1Loss, nn.HuberLoss, nn.CrossEntropyLoss and more. The Wikipedia loss function article gives the broader theory.
Bringing it together
Think of the loss as the sentence you give the optimiser about what you care about. If you tell it “big misses are catastrophic” with a squared penalty, it will bend the model to avoid them; if you say “just be confidently right” with cross-entropy, it will produce calibrated probabilities. Most projects only ever need a handful of these — squared error or its robust cousins for regression, cross-entropy for classification — so once you understand this small set you can train the large majority of models with confidence. Explore each linked guide for its formula, an interactive widget, and Python code, and you will always know not just which loss to pick, but exactly why.
Custom and weighted objectives
The standard menu covers the vast majority of projects, but part of the power of modern frameworks is that you can write your own criterion whenever the defaults do not capture what you truly care about. Any differentiable function of the predictions and the targets can serve, and practitioners routinely combine several terms — for example a reconstruction term plus a penalty that encourages simplicity — into a single weighted objective, tuning the weights to balance competing goals. Weighting is also how you inject business priorities: charging more for the mistakes that hurt customers, or for errors on a class that is rare but critical. The one hard constraint is differentiability, because training needs a gradient to follow; a metric like accuracy, which jumps in steps, cannot be optimised directly, which is precisely why smooth surrogates exist and are optimised in its place. When you do craft a custom criterion, sanity-check its gradient on a tiny example first, because a subtle sign error there can send training confidently in the wrong direction.
A short history of the idea
The practice of fitting models by minimising a penalty is centuries old. Least squares, the ancestor of today’s squared-error objective, was used by Gauss and Legendre around 1800 to fit the orbits of celestial bodies from noisy observations. The probabilistic view arrived with maximum-likelihood estimation in the early twentieth century, which reframed fitting as choosing the parameters that make the observed data most probable and, for classification, leads straight to the log-based objectives used today. The margin-based family grew out of statistical learning theory in the 1990s alongside support vector machines. What deep learning added was not new penalties so much as the machinery — automatic differentiation and stochastic gradient descent — to minimise almost any differentiable criterion at massive scale. Seen this way, the modern toolbox is a long conversation between statistics and computation, and the humble choice you make when compiling a model connects directly to two hundred years of that conversation.
A quick decision checklist
When you sit down to a new problem, a short mental checklist gets you to the right objective in seconds. First, name the output: a continuous number points you to the squared-error family, a category points you to a probabilistic or margin-based criterion. Second, weigh your outliers: if the biggest misses are genuinely the most costly, keep the squared penalty; if they are mostly noise, switch to an absolute or robust alternative. Third, decide whether you need calibrated probabilities out the far end — if downstream code will threshold or rank by confidence, choose the log-based option, and if a crisp boundary is all you need, the margin criterion will do. Finally, separate this training choice from how you will report success, because the number you optimise and the number you show stakeholders are allowed to differ. Run through those four questions and you will rarely pick the wrong objective, which is more than half the battle in getting a model to learn what you actually meant. And if you are ever unsure, the safe defaults rarely let you down: squared error for regression and the log-based penalty for classification are the choices most models ship with, so starting there and only deviating when a clear reason appears is a sound, low-risk strategy for beginners and experts alike. Master this small family first, and the more exotic objectives you meet later will simply feel like variations on ideas you already understand.