Vanishing Gradient Problem: 5 Causes and Fixes

The vanishing gradient problem is the reason deep neural networks were nearly impossible to train for years. As error gradients propagate backward through many layers, saturating activations shrink them toward zero, so the earliest layers barely learn. This guide explains exactly why the vanishing gradient problem happens, shows it with an interactive plot, and lists the proven fixes — chief among them ReLU.

📌

In one line. The vanishing gradient problem: chained derivatives below 1 multiply toward zero with depth, starving early layers of signal.

See it: why sigmoid gradients vanish

Below is the sigmoid (blue) and its derivative (orange). The derivative never exceeds 0.25 — multiply many such numbers through a deep network and the gradient collapses toward zero. That is the vanishing gradient problem in one picture.

f(x)derivative f'(x)

What is the vanishing gradient problem?

The vanishing gradient problem is what happens when the gradients used to update a deep network’s early layers become extremely small. Backpropagation multiplies the derivative at each layer together via the chain rule. If those derivatives are less than one — as they always are for saturating activations like sigmoid and tanh — their product shrinks exponentially with depth, so the first layers receive almost no learning signal.

Why it happens

gradient ∝ ∏ f'(x) — if each f'(x) ≤ 0.25, then 0.25ⁿ → 0

With sigmoid, the maximum derivative is 0.25. Across ten layers that is 0.25¹⁰ ≈ 0.000001. The early layers barely move, so the network effectively refuses to train — the core reason the vanishing gradient problem stalled deep learning for years.

How to fix the vanishing gradient problem

💡

Five proven fixes. (1) Use ReLU or its variants — their derivative is 1 in the active region. (2) Add residual (skip) connections so gradients bypass layers. (3) Apply batch normalisation. (4) Use careful weight initialisation (He/Xavier). (5) For RNNs, use LSTM/GRU gates. ReLU alone solves most of it.

The opposite: exploding gradients

⚠️

Exploding gradients. The mirror image of the vanishing gradient problem is when derivatives greater than one multiply up and the gradient blows up to huge values. The usual cure is gradient clipping plus good initialisation.

Quick illustration in Python

import numpy as np

# product of sigmoid-max gradients over n layers
for n in [1, 5, 10, 20]:
    print(n, "layers ->", 0.25 ** n)
# 20 layers -> 9e-13  (vanished)

See the Wikipedia article for the historical background.

How to spot it

The tell-tale sign is a network whose loss barely moves while the weights in the earliest layers stay almost frozen and the later layers change normally. Logging the average gradient magnitude per layer during training makes it obvious — the numbers shrink by orders of magnitude as you move toward the input. Modern architectures bake in the cure rather than leaving it to chance: residual connections in ResNets and the gating in Transformers both give gradients a short path back to early layers, which is a large part of why very deep models train at all today. Batch and layer normalisation help by keeping each layer’s inputs in a healthy range, and good initialisation schemes set the starting scale of the weights so that signals neither shrink nor blow up on the very first forward pass.

Related activation functions

Activation Functions — the complete guide (hub).
ReLU — the main fix.
Sigmoid Function — a common cause.
Tanh — also saturates.

Frequently asked questions

What causes the vanishing gradient problem?

Saturating activations like sigmoid and tanh have derivatives below 1. Multiplying many of them through backpropagation shrinks the gradient exponentially with depth, so early layers stop learning.

How does ReLU fix vanishing gradients?

ReLU has a derivative of exactly 1 for positive inputs, so gradients pass backward at full strength instead of shrinking. This is the single biggest reason ReLU replaced sigmoid in hidden layers.

What is the difference between vanishing and exploding gradients?

Vanishing gradients shrink toward zero when derivatives are below 1; exploding gradients blow up when derivatives exceed 1. Clipping, normalisation and good initialisation address both.