Author name: Jamal Hussein

The sigmoid vs softmax choice decides your network’s output layer, and getting it wrong quietly breaks a classifier. The short […]

The vanishing gradient problem is the reason deep neural networks were nearly impossible to train for years. As error gradients

The ELU activation function — the Exponential Linear Unit — behaves like ReLU for positive inputs but replaces the flat

The Swish activation function is a smooth, self-gated activation defined simply as the input times its own sigmoid. Found by

The GELU activation function — the Gaussian Error Linear Unit — is the activation behind modern transformers such as BERT

Leaky ReLU is a simple, popular fix for the biggest weakness of ordinary ReLU: dead neurons. Instead of flattening every

The tanh activation function — the hyperbolic tangent — squashes inputs into the range −1 to 1. It is a

The softmax function is how a neural network turns a handful of raw scores into class probabilities that add up

The sigmoid function is the classic S-shaped curve that squashes any real number into a value between 0 and 1.

The ReLU activation function — short for Rectified Linear Unit — is the default activation for hidden layers in modern