GELU Activation Function: 3 Reasons Transformers Use It

By /

The GELU activation function — the Gaussian Error Linear Unit — is the activation behind modern transformers such as BERT and GPT. It is a smooth, probabilistic gate that weights each input by how likely it is to be positive, giving slightly better accuracy than ReLU in deep models. This guide covers the GELU activation function’s formula, why transformers prefer it, GELU versus ReLU, and Python code.

📌
In one line. The GELU activation function is x·Φ(x): a smooth gate that lets slightly-negative inputs pass, powering BERT and GPT.

gelu activation function graph

Try it: interactive GELU plot

The blue curve is the GELU activation function; the orange dashed curve is its derivative. Notice the gentle dip below zero — GELU is smooth everywhere, unlike ReLU’s sharp kink.

f(x)derivative f'(x)

The GELU activation function formula

GELU(x) = x · Φ(x)

The GELU activation function — Gaussian Error Linear Unit — multiplies the input by Φ(x), the cumulative distribution function of the standard normal. Intuitively, instead of a hard on/off gate like ReLU, GELU weights each input by the probability that it is greater than a random Gaussian, giving a smooth, probabilistic gate.

Why transformers use GELU

💡
Smoothness helps. The GELU activation function is differentiable everywhere and allows small negative outputs, which empirically gives better-behaved gradients and slightly higher accuracy than ReLU in deep transformers. That is why BERT, GPT and most modern language models use GELU in their feed-forward blocks.

GELU vs ReLU

ReLU has a sharp corner at zero and kills all negatives; the GELU activation function curves smoothly through the origin and lets slightly-negative inputs pass with a small weight. GELU costs more to compute, but in large models the accuracy gain is usually worth it. On small networks, ReLU is still a fine, cheaper default.

GELU activation function in Python

import numpy as np

def gelu(x):
    return 0.5 * x * (1 + np.tanh(
        np.sqrt(2/np.pi) * (x + 0.044715 * x**3)))

print(gelu(np.array([-2.0, 0.0, 2.0])))
# [-0.045  0.     1.955]

PyTorch provides it directly as torch.nn.GELU(). See the original GELU paper.

Compute cost in practice

The exact definition uses the Gaussian error function, but frameworks ship a fast tanh-based approximation that is accurate to a few decimal places and much cheaper to evaluate on a GPU. For inference on edge devices the extra cost over a simple rectifier can still matter, so some deployed models train with the smooth curve and then swap to a lighter unit for serving. On large accelerators the difference is usually negligible next to the cost of the attention layers around it, which is why nearly every large language model simply uses the smooth version everywhere. A practical habit is to match whatever the architecture you are fine-tuning was originally trained with, since mixing activations between pre-training and fine-tuning can subtly shift the distributions each layer expects.

Related activation functions

Frequently asked questions

What is the GELU activation function used for?
The GELU activation function is the default in transformer models like BERT and GPT, used in their feed-forward layers because its smoothness gives slightly better accuracy than ReLU.
How is GELU different from ReLU?
ReLU is a hard gate that zeros all negatives with a sharp corner. The GELU activation function is a smooth probabilistic gate that lets slightly-negative inputs through, and is differentiable everywhere.
Is GELU better than ReLU?
In large deep models, often yes by a small margin. For small or latency-sensitive networks, ReLU is cheaper and usually good enough.
Scroll to Top