Cross Entropy Loss: 1 Formula Behind Better Classifiers

The cross entropy loss is the workhorse objective for classification, the way squared error is for regression. It measures how far a model’s predicted probabilities are from the truth, and it does so with a sharp bias: it barely penalises confident correct answers but punishes confident wrong ones brutally. That single property is what pushes a classifier to output well-calibrated probabilities, and it is why almost every neural network that ends in a softmax or sigmoid is trained against it.

What is the cross entropy loss?

At heart it is beautifully simple: look at the probability your model gave to the correct answer, take its logarithm, and negate it. If the model said the right class was 99% likely, the log is almost zero and the penalty is tiny. If it said the right class was 1% likely, the log is a large negative number and the penalty is enormous. Averaged over every example, that is the cross entropy loss. It comes straight from information theory, where it measures the extra bits needed to encode the true labels using the model’s predicted distribution — but you can use it perfectly well thinking only of the negative-log-probability picture.

You will meet it under two other names. When there are just two classes it is usually called binary cross-entropy or log loss; with many classes it is called categorical cross-entropy. All three are the same idea applied to different numbers of outputs.

The cross entropy loss formula

Here y_i is 1 for the true class and 0 for the rest, and p_i is the probability the model assigned to class i. Because the true-label vector is all zeros except a single one, the sum collapses to a single term: minus the log of the probability on the correct class. That is the whole the loss for one example — everything else is just averaging and bookkeeping.

Try it: the cross-entropy calculator

Drag the slider to set the probability your model gave the correct class and watch the penalty. Notice how gentle it is near 1 and how it rockets upward as the probability falls toward 0.

Why the logarithm punishes confident mistakes

A worked example

Imagine a three-class problem where the true class is the first one. Two models both predict the right class, but with different confidence: model A outputs (0.7, 0.2, 0.1) and model B outputs (0.4, 0.35, 0.25). Both would score the same accuracy, yet their penalties differ sharply:

Accuracy cannot tell model A from model B, but the the loss rewards A for being more confident in the correct answer. That sensitivity to probability, not just the final label, is why it trains sharper, better-calibrated models than a bare accuracy target ever could.

Cross-entropy and softmax: the perfect pair

The loss almost always sits directly on top of a softmax (for single-label multi-class) or a sigmoid (for binary and multi-label). The pairing is not an accident: when you combine softmax with this loss, the gradient simplifies to the elegant “predicted probability minus true label,” which is numerically stable and fast. That is why frameworks fuse them into a single operation and ask you to pass raw scores, not probabilities.

Binary vs categorical

They differ only in how the labels are shaped and how many outputs there are; the underlying negative-log-probability idea is identical. Read the binary cross-entropy guide for the two-class case in depth.

Common mistakes to avoid

Cross entropy loss in Python

import numpy as np

def cross_entropy(p_true_class):
    p = np.clip(p_true_class, 1e-12, 1.0)   # avoid log(0)
    return -np.log(p).mean()

print(cross_entropy(np.array([0.7, 0.9, 0.6])))   # 0.29

In PyTorch use nn.CrossEntropyLoss() (it applies softmax internally) or nn.BCEWithLogitsLoss() for the binary case; in Keras, categorical_crossentropy or binary_crossentropy. See the Wikipedia cross-entropy article for the information-theory background.

Putting it into practice

In day-to-day work the loss is both your training objective and a useful diagnostic. Because it reacts to probabilities rather than hard labels, a model can improve its the loss for several epochs even while its accuracy sits still — it is becoming better calibrated before it starts flipping predictions, so watch the loss, not just accuracy, to judge progress. If the training loss falls smoothly while the validation loss turns upward, you are overfitting and it is time to stop or regularise. A sudden spike to a huge value almost always means a probability hit zero somewhere, which points to a missing clip or an unstable learning rate. And when you finally report results, remember that a good loss and a good accuracy are related but not identical: a model can be accurate yet poorly calibrated, and it is precisely that calibration gap the the loss is designed to expose and close.

From information theory to training

The idea has deep roots in information theory, where cross-entropy measures the average number of bits you would need to encode events drawn from one distribution using a code built for another. In machine learning the “true” distribution is the labels and the “other” distribution is the model’s predictions, so minimising the quantity means making the model’s predicted distribution match reality as closely as possible. It is closely tied to a cousin called Kullback–Leibler divergence, which measures how far apart two distributions are; when the true labels are fixed, minimising one is equivalent to minimising the other. You do not need any of this theory to use the objective effectively, but it explains why the metric behaves the way it does: it is fundamentally a measure of surprise, and a confident wrong answer is maximally surprising, which is exactly why it is punished so hard. Maximum-likelihood estimation, the statistical principle behind fitting most models, also reduces to this same minimisation, which is why the objective shows up almost everywhere probabilities are learned.

Reading the curve during training

Because the objective reacts to probabilities rather than to hard labels, it is a more sensitive progress signal than accuracy, and reading it well saves a lot of wasted training time. Early on it typically falls quickly as the model stops making wildly overconfident mistakes, then settles into a slow grind as it fine-tunes calibration. A validation curve that flattens and then creeps upward while the training curve keeps dropping is the classic signature of overfitting, and it is your cue to stop, add regularisation, or gather more data. Sudden spikes usually mean a probability collapsed to zero or the learning rate is too high. It also pays to remember that a lower value is not automatically a better model for your purpose: a network can achieve an excellent score by being beautifully calibrated on the common classes while quietly failing on a rare but important one, so always read it next to a class-aware metric such as recall on the class you care about.

Frequently asked questions