ReLU

Deciding whether a value passes through

A neuron adds up the weighted values arriving from the previous layer. But that sum doesn't go straight to the next layer. Before it's handed off, it passes through a rule, and that rule is the activation function. ReLU is the most widely used one, and what it does comes down to a single threshold. If the sum is below zero, it outputs zero. If it's above zero, it passes the value through untouched.

Without this bend, stacking layers wouldn't accomplish much. Chaining multiplication and addition alone eventually collapses into a single multiplication and addition, no matter how many layers get piled on. A bend has to sit somewhere in between for each layer to shape something new. ReLU inserts one cheap bend at zero, once per layer.

Switching half the neurons off makes it lighter, not weaker

If a layer holds thousands of neurons, roughly half of them land on a negative sum and become zero. A neuron at zero adds nothing to the next layer. That looks like a loss, but it's really a gain. Instead of every neuron murmuring a little on every input, only the neurons actually relevant to that input get a voice.

Once the irrelevant signals go quiet, the ones that remain come through more clearly. The computation gets lighter, and because different inputs switch on different neurons, a natural division of labor emerges. This property, where a large share of values sits at zero, is called sparsity.

Since the same input turns some neurons on and others off, the combination of active neurons shifts with every layer. Stitching straight segments together into a curved boundary comes out of exactly that shifting combination. A rule built on a single threshold, stacked layer after layer, ends up drawing surprisingly complex shapes.

What made deep networks actually trainable

A network learns by sending word of how wrong its answer was back from the output toward the input. At that point, each layer's activation function multiplies its own share into the signal flowing back. The smooth curved rule used earlier multiplied in a tiny value at both ends. Cross even ten layers and the signal had nearly vanished, leaving the early layers with nothing to learn from.

ReLU passes the signal through unshrunk on the side where it's switched on. Cross dozens of layers and there's still something left for the early layers to learn from. A large share of the credit for turning deep networks from a theory on paper into something that actually runs belongs to this one simple rule.

Neurons that fall asleep

There's a weakness too. If a neuron ends up producing a negative sum no matter what input it receives, its output is always zero and the signal returning to it is also always zero. There's no thread left to pull to fix it, and it stays stuck that way. This is called a dead neuron.

It tends to show up when the learning rate is set too high. One oversized adjustment can push a neuron deep into negative territory with no force left to pull it back out. When a layer accumulates too many neurons like this, that layer ends up taking up space without doing any work.

That's why variants emerged that tilt the negative side very slightly instead of flattening it all the way to zero. Leaving even a sliver of signal there gives a neuron a way to wake back up. Keeping the learning rate modest or spreading the starting values out sensibly heads off much of the problem before it starts.