Naive Bayes is the fastest useful classifier you will meet, trains in a single pass over the data, and remains a genuinely strong baseline for text. It is also the one whose central assumption is openly wrong, which makes it interesting.
How Naive Bayes decides
Bayes’ theorem, in the form the classifier uses:
$$P(\text{class} \mid \text{features}) \propto P(\text{class}) \times \prod_i P(\text{feature}_i \mid \text{class})$$
What “naive” means
The naivety is one specific assumption: every feature is independent of every other, given the class.
For text, that says the word “New” tells you nothing about whether “York” follows. Obviously false. Words are deeply dependent on each other.
🧪 Why a false assumption still classifies well
Correlated features get double-counted, so the winning class often ends up with a probability like 0.9999 when the truth is nearer 0.75. The number is badly wrong — but it is still the largest number, so the prediction is still correct.
The practical consequence: trust Naive Bayes for the class it picks, never for the confidence it reports. If you need calibrated probabilities, use logistic regression instead.
The zero-frequency problem
Multiplication has a brutal failure mode. If a word never appeared in the spam training examples, its likelihood is zero — and zero times anything is zero. One unseen word vetoes the entire class, no matter how damning everything else was.
The fix is Laplace smoothing: add 1 to every count. Nothing is ever exactly zero, so no single feature can veto a class. It is one line, and it is essential rather than optional.
Which variant to use
| Variant | For | Example |
|---|---|---|
| Multinomial | Counts | Text classification — the usual choice |
| Bernoulli | Present / absent | Short documents where occurrence matters more than count |
| Gaussian | Continuous numbers | Measurements — assumes each is bell-shaped per class |
Where it genuinely shines
Text, above all: spam filtering, sentiment, topic labelling. High-dimensional sparse data is exactly where the independence assumption does least damage, and where the speed matters most.
Small datasets, because estimating a few frequencies needs far less data than fitting a flexible model.
As a baseline. It trains in seconds. If your elaborate model cannot beat Naive Bayes, that is worth knowing early.
Where it fails
⚠️ Three real limits
- Probabilities are not trustworthy. Correlated features inflate confidence badly.
- Strongly dependent features hurt it. Duplicate a column and that evidence counts twice.
- It cannot learn interactions. “A only matters when B is also present” is unrepresentable — features vote separately by construction. Trees and boosting handle that; Naive Bayes cannot.
It also needs no feature scaling, since it works with probabilities rather than distances — a small convenience worth knowing.
🔑 Key Takeaways
- Naive Bayes multiplies the prior by each feature’s likelihood and picks the biggest result.
- Training is counting frequencies, so it is extremely fast and works on small data.
- The independence assumption is false but usually harmless — the ranking survives.
- Its probabilities are unreliable; trust the class, not the confidence.
- Always use Laplace smoothing, or one unseen feature zeroes out a whole class.
Further reading
scikit-learn’s Naive Bayes guide documents all three variants and the smoothing parameter, and Wikipedia’s article works through the spam-filtering derivation in full.
Where to go next
- Gradient boosting and XGBoost — the next post in this sequence
- Covariance vs correlation — measuring the dependence Naive Bayes ignores
- Classification metrics — how to judge the result
- Start Here — the full learning path