ELU Activation Function: 3 Benefits and a Live Plot

By /

The ELU activation function — the Exponential Linear Unit — behaves like ReLU for positive inputs but replaces the flat negative region with a smooth exponential curve. That negative saturation pushes a layer’s mean activation toward zero and, like Leaky ReLU, prevents dead neurons. This guide covers the ELU activation function’s formula, why it helps, ELU versus Leaky ReLU, and Python code.

📌
In one line. The ELU activation function equals x for x>0 and α(ex−1) for x≤0, saturating smoothly at −α.

elu activation function graph

Try it: interactive ELU plot

The blue curve is the ELU activation function with α = 1; unlike Leaky ReLU’s straight negative line, ELU curves smoothly to a floor of −α.

f(x)derivative f'(x)

The ELU activation function formula

f(x) = x if x > 0, else α(ex − 1)

The ELU activation function — Exponential Linear Unit — behaves like ReLU for positive inputs but replaces the flat negative region with a smooth exponential curve that saturates at −α (usually α = 1).

Why the negative saturation helps

💡
Mean activations near zero. Because the ELU activation function outputs small negative values, the average activation of a layer sits closer to zero. That acts a little like batch normalisation, speeding up learning, and — unlike Leaky ReLU — the bounded negative floor makes it more robust to noise.

ELU vs Leaky ReLU

Both fix dying ReLU, but they treat negatives differently. Leaky ReLU uses a straight line with slope α that keeps decreasing forever; the ELU activation function uses a curve that flattens to −α. ELU’s smoothness and saturation often train a touch better, at the cost of an exponential per negative input.

⚠️
Cost. The ELU activation function calls exp() for negative inputs, so it is slower than ReLU or Leaky ReLU. On large models that extra cost adds up, which is one reason ReLU-family functions remain popular.

ELU activation function in Python

import numpy as np

def elu(x, alpha=1.0):
    return np.where(x > 0, x, alpha * (np.exp(x) - 1))

print(elu(np.array([-2.0, -0.5, 2.0])))
# [-0.865 -0.393  2.   ]

In Keras: tf.keras.layers.ELU(alpha=1.0). Background is in the ELU paper.

Tuning notes

The single hyperparameter controls how deep the negative saturation goes; the default of one works well across most tasks and rarely needs changing. A closely related scaled variant, SELU, fixes the constants so that activations self-normalise toward zero mean and unit variance from layer to layer, which can remove the need for explicit normalisation in carefully constructed feed-forward networks. As always, the smooth negative branch costs an exponential per unit, so it is worth benchmarking against a plain rectifier before committing on a latency-sensitive model. In practice the choice between the smooth exponential curve and a straight leaky slope is rarely decisive; both keep neurons alive, and the larger accuracy gains almost always come from data, architecture and regularisation rather than from swapping one negative-side shape for another.

Related activation functions

Frequently asked questions

What is the ELU activation function?
The ELU activation function equals x for positive inputs and alpha*(e^x - 1) for negatives, giving a smooth curve that saturates at -alpha and pushes mean activations toward zero.
ELU vs Leaky ReLU, which is better?
Both prevent dead neurons. ELU uses a smooth, saturating curve and often trains slightly better, but it is more expensive because of the exponential. Leaky ReLU is cheaper.
What is a typical alpha for ELU?
Alpha is usually set to 1.0, which makes the negative side saturate at -1 and keeps the function continuous at zero.
Scroll to Top