Leaky ReLU: Fix Dying ReLU in 1 Step, with Code

Leaky ReLU is a simple, popular fix for the biggest weakness of ordinary ReLU: dead neurons. Instead of flattening every negative input to zero, Leaky ReLU lets negatives through with a small slope, so a neuron can always keep learning. This guide covers the Leaky ReLU formula, how it cures dying ReLU, how it compares to PReLU and ELU, and Python code.

📌

In one line. It outputs x for positive inputs and αx (small α) for negatives, so its gradient is never exactly zero.

Try it: interactive Leaky ReLU plot

The blue curve is Leaky ReLU with slope α = 0.1; notice the negative side is no longer flat, so its derivative (orange) never hits zero.

f(x)derivative f'(x)

The Leaky ReLU formula

f(x) = x if x > 0, else αx (typically α = 0.01–0.1)

Leaky ReLU is a small but important tweak to ReLU. Instead of flattening negatives to exactly zero, it lets them through with a gentle slope α. Positive inputs behave exactly like ReLU.

How Leaky ReLU fixes dying ReLU

💡

Always a gradient. A standard ReLU neuron whose input is always negative outputs zero forever and its gradient is zero — it is dead. Because it keeps slope α on the negative side, its derivative is α (not 0) there, so the neuron can always recover. That single change eliminates the dying ReLU problem.

Leaky ReLU vs ReLU vs PReLU

Function	Negative side	Learns slope?
ReLU	0	no
Leaky ReLU	αx, fixed α	no
PReLU	αx	yes (α trained)

Parametric ReLU (PReLU) is Leaky ReLU where α is learned during training rather than fixed. ELU takes a smooth-curve approach to the same goal.

Leaky ReLU in Python

import numpy as np

def leaky_relu(x, alpha=0.1):
    return np.where(x > 0, x, alpha * x)

print(leaky_relu(np.array([-3.0, -1.0, 2.0])))
# [-0.3 -0.1  2. ]

In Keras: tf.keras.layers.LeakyReLU(alpha=0.1).

When it helps in practice

The clearest signal that this variant is worth trying is a training curve that stalls with many units stuck outputting zero — a quick histogram of activations across a batch reveals it immediately. On most vision and tabular tasks the accuracy difference from plain rectified units is small, so it is best treated as a cheap insurance policy against dead neurons rather than a guaranteed upgrade. A common workflow is to reach for it only after seeing dead units, rather than by default. If you would rather not pick the negative slope by hand, let the network learn it during training, which turns the idea into Parametric ReLU. It is also worth remembering that a very large learning rate is often the real cause of dead units in the first place, so lowering the rate or adding warm-up can matter more than the exact shape of the negative side. Detailed background and benchmarks across several datasets are in the empirical evaluation of rectified activations in convolutional networks, which remains a useful reference for choosing between these closely related variants.

Related activation functions

Activation Functions — the complete guide (hub).
ReLU — the original it fixes.
ELU — a smooth alternative.
GELU — the transformer favourite.

Frequently asked questions

What problem does Leaky ReLU solve?

It solves the dying ReLU problem. By giving negative inputs a small slope instead of zero, neurons always keep a non-zero gradient and can recover.

What is a good value for the Leaky ReLU slope?

A small positive alpha, commonly 0.01 or 0.1. If you let the network learn alpha instead of fixing it, you have Parametric ReLU (PReLU).

Is Leaky ReLU always better than ReLU?

Not always. It reliably prevents dead neurons, but on many tasks plain ReLU performs just as well and is slightly cheaper. Leaky ReLU is a safe first thing to try if you see dead units.