ReLU Activation Function: 3 Reasons It Powers Deep Nets

By /

The ReLU activation function — short for Rectified Linear Unit — is the default activation for hidden layers in modern neural networks. It is almost absurdly simple: it returns the input if the input is positive, and zero otherwise. That simplicity is exactly why it trains fast and largely fixed the vanishing-gradient problem that held deep networks back for years.

📌
In one line. The ReLU activation function is f(x) = max(0, x) — it passes positive values through unchanged and clamps everything negative to zero, giving networks cheap, non-saturating non-linearity.

relu activation function graph

Try it: interactive ReLU plot

The blue line is the ReLU activation function; the orange dashed line is its derivative. Notice the derivative is a clean 0 or 1 — no tiny gradients to vanish.

ReLU f(x)Derivative f'(x)

The ReLU formula

f(x) = max(0, x)  =  { x if x > 0, else 0 }

The derivative is equally simple: 1 for positive inputs and 0 for negative inputs (undefined exactly at zero, where libraries just pick 0). Because the gradient is a full 1 in the active region, error signals flow backward through many layers without shrinking — the core reason the ReLU activation function unlocked deep learning.

How the ReLU activation function changed deep learning

Before 2012, most networks used sigmoid or tanh, and training anything more than a few layers deep was painful: gradients shrank toward zero as they propagated backward, so early layers barely learned. When AlexNet won the ImageNet competition using the ReLU activation function throughout its hidden layers, the field noticed. ReLU let gradients pass at full strength through the active half of every neuron, so very deep convolutional networks suddenly became trainable in reasonable time. Today ReLU and its variants are the default choice for the hidden layers of almost every convolutional and feed-forward architecture, and understanding it is the natural first step into every other activation function.

Why ReLU became the default

💡
Three wins. It is cheap (a single comparison, no exponentials); it does not saturate for positive inputs, so gradients stay strong; and it produces sparse activations (many exact zeros), which acts as a mild, helpful form of regularisation.

The dying ReLU problem

⚠️
Dying ReLU. If a neuron's weights push its input permanently negative, its output — and its gradient — become zero forever, and it stops learning. A large learning rate makes this worse. The fix is usually Leaky ReLU or ELU, which leak a small gradient for negative inputs.

ReLU vs sigmoid and tanh

PropertyReLUSigmoid / Tanh
Output range[0, ∞)(0,1) / (−1,1)
Saturates?Only for x < 0Both tails
Vanishing gradientLargely avoidedSevere in deep nets
Cost1 comparisonexp() call

For hidden layers, ReLU almost always wins. You still use sigmoid for a binary output and softmax for multi-class output.

ReLU in Python

import numpy as np

def relu(x):
    return np.maximum(0, x)

def relu_derivative(x):
    return (x > 0).astype(float)

print(relu(np.array([-2.0, -0.5, 0.0, 1.5, 3.0])))
# [0.  0.  0.  1.5 3. ]

In PyTorch it is torch.nn.ReLU(); in TensorFlow/Keras, activation="relu". See the Wikipedia rectifier article for the mathematical background.

Frequently asked questions

Why is the ReLU activation function better than sigmoid?
ReLU does not saturate for positive inputs, so gradients stay large and deep networks train faster. Sigmoid squashes everything into (0,1) and its gradient nearly vanishes in both tails.
Where should I use ReLU?
Use ReLU (or a variant) on the hidden layers of feed-forward and convolutional networks. Use sigmoid or softmax on the output layer, depending on the task.
What is dying ReLU?
A neuron stuck outputting zero for every input because its pre-activation is always negative. It never recovers because its gradient is also zero. Leaky ReLU and ELU prevent it.

Related activation functions

Scroll to Top