Swish Activation Function: 3 Facts, Formula & Code

The Swish activation function is a smooth, self-gated activation defined simply as the input times its own sigmoid. Found by an automated search at Google, it often edges out ReLU on deep and mobile networks and is identical to SiLU. This guide covers the Swish activation function’s formula, what makes it special, how it compares to ReLU and GELU, and Python code.

📌

In one line. The Swish activation function is x·σ(x) — a smooth, non-monotonic curve that dips below zero before rising.

Try it: interactive Swish plot

The blue curve is the Swish activation function; notice how it dips slightly below zero before rising — a non-monotonic shape no simpler activation has.

f(x)derivative f'(x)

The Swish activation function formula

Swish(x) = x · σ(x) = x / (1 + e^−x)

The Swish activation function, discovered by a Google Brain search over activations, is simply the input times its own sigmoid. Because the sigmoid acts as a soft, learnable-looking gate on the input, Swish is described as self-gated. It is identical to SiLU (Sigmoid Linear Unit).

What makes Swish special

💡

Smooth and non-monotonic. Unlike ReLU, the Swish activation function is smooth everywhere and actually decreases slightly for small negative inputs before flattening. This small bump helps gradient flow and often gives a modest accuracy boost in deep and mobile networks such as EfficientNet.

Swish vs ReLU vs GELU

Swish and GELU are close cousins — both are smooth, self-gating curves that beat ReLU by a small margin on large models. The practical trade-off is the same as always: smooth activations cost more compute, so ReLU remains the efficient default and Swish/GELU are chosen when squeezing out extra accuracy matters.

Swish activation function in Python

import numpy as np

def swish(x, beta=1.0):
    return x * (1 / (1 + np.exp(-beta * x)))

print(swish(np.array([-3.0, 0.0, 3.0])))
# [-0.142  0.     2.858]

In Keras it is tf.keras.activations.swish. See the Swish paper for the search that found it.

Where you will meet it

This activation rose to prominence inside Google’s EfficientNet family of image models and appears throughout mobile-optimised architectures where a small accuracy gain per parameter is worth chasing. Because it is built from a sigmoid, it inherits a slightly higher compute cost than a plain rectifier, so teams weigh that against the payoff on their specific hardware and batch sizes. A practical routine is to prototype with the cheaper rectifier and switch to the smooth curve only once the architecture is settled and you are tuning for the last point of accuracy. The optional beta parameter controls how sharp the gate is: at large beta it approaches a hard rectifier, and at beta near zero it approaches a straight line, so a single family of curves spans much of the design space between the two.

Related activation functions

Activation Functions — the complete guide (hub).
GELU — its closest relative.
ReLU — the baseline it improves on.
Sigmoid Function — the gate inside Swish.

Frequently asked questions

What is the Swish activation function?

Swish is x times the sigmoid of x. It is a smooth, self-gated activation that often edges out ReLU on deep and mobile networks, and is identical to SiLU.

Is Swish the same as SiLU?

Yes. Swish with beta = 1 is exactly the Sigmoid Linear Unit (SiLU); the two names refer to the same function.

Should I use Swish instead of ReLU?

Try it when you want a small accuracy gain on a large model and can afford extra compute. For speed-critical or small models, ReLU is still the better default.