Activation Function
A built-in threshold that decides whether a signal passes on
- An activation function is the last gate that decides whether a neuron's summed value gets passed on, and how strongly.
- The key is that it is not a straight line. That bend is what lets a network draw boundaries a straight line never could.
- Without it, stacking layers collapses back into a single layer. It's the piece that makes depth actually mean something.
- The most common choice squashes everything below a threshold to zero and lets anything above it pass through unchanged.
- Which function you pick changes how fast and how stable training is. Middle layers and the very last layer usually use different ones.
Contents
1The analogy
The motion-sensor light in an apartment hallway stays off by default. If someone barely moves at the far end of the hall, it ignores them. Only once they get close enough does it snap on. Once lit, it stays bright until the hallway goes quiet again.
The light has a fixed rule. Movement weaker than the threshold never trips it, no matter how many times it happens. Cross the threshold, though, and it reacts instantly. Because it ignores weak signals, a shadow flickering in the wind doesn't make it flicker too.
An activation function is that sensor's decision rule. A neuron adds up its weighted inputs, and if the total doesn't clear the threshold, nothing gets passed on. If it does clear it, the neuron sends along however much cleared it. It isn't a plain pass-through — it's a gatekeeper that filters the signal first.
2In detail
Everything below the line gets zeroed, everything above passes through
The most common choice for middle layers these days is almost embarrassingly simple. If the incoming value is below zero, it becomes zero. If it's above zero, it passes through untouched. This is called ReLU.
It works well precisely because it's simple. The computation is a single comparison, so it's fast, and large values pass through without shrinking. Because many neurons in a layer end up outputting zero, only the neurons that actually matter stay active, which lightens the load on the whole network.
Older networks used a gentler curve
For a long time, a different kind of function was popular — one that squeezed any input, however large, into a range between zero and one. The sigmoid is the classic example. Because its output sits between zero and one, it reads naturally like a probability.
The trouble showed up during training. Once a value gets even moderately large or small, the curve goes nearly flat, so changing the input barely changes the output. When the error signal gets sent back toward the earlier layers, it shrinks a little every time it crosses one of these flat stretches, until the earliest layers have almost nothing left to learn from. This is the vanishing gradient problem, and it's why this style of function got pushed out of middle layers once networks started stacking deep.
Without it, depth is wasted effort
What happens if you strip activation functions out and just chain multiplications and additions together? No matter how many times you repeat multiplying and adding, the whole chain reduces back to a single multiply-and-add. Stack ten layers or a hundred, and you can express exactly what a one-layer model can.
Add one bend in the middle, and everything changes. Each layer folds and cuts the previous layer's output before recombining it, so the more layers you stack, the more complex a boundary you can shape. That's where a network's power to separate data no straight cut could handle actually comes from.
The very last layer uses a different function
A middle layer's function exists to hand the signal off cleanly to the next layer. The last layer's function exists to shape the answer into something a person can read. The jobs are different, so the tools are too.
For a yes-or-no question, the last layer squashes the result into something that reads like a probability between zero and one. For a pick-one-of-many question, it uses softmax, which spreads the values across every candidate so they add up to one. For a problem where the answer is a number itself, like a price, no function gets applied at the end at all — the computed value just passes straight out.
The real test is whether training goes well
Whenever a new function shows up, what people actually check is whether training is faster and more stable under the same conditions. There are plenty of small variants out there — versions that leave a sliver of the negative side alive instead of zeroing it out completely, versions that smooth out the corner near the threshold.
The differences are often small. So unless there's a specific reason not to, middle layers default to the simplest, fastest option, and other candidates only get tried once training isn't going well.
3More precisely
An activation function is the nonlinear function applied at the end of a neuron's computation. Multiplying by weights, summing, and adding a bias term are all linear operations, so without this step, even many layers stacked together reduce to a single linear transformation. The reason a network can draw curved boundaries at all comes down to this nonlinearity.
The sensor-light comparison breaks down in one place. A sensor light only has two states, on and off, but the activation functions actually in use also carry through how strong the signal is once it's on. A value that clears the threshold by a lot comes out large; one that barely clears it comes out small. And a sensor light has one fixed threshold, while a neuron's turn-on point is shifted not by the function itself but by its bias term.
Training also requires one more thing from this function: its slope has to be knowable, so error can be sent backward through it. A function with a sharp corner at zero has an undefined slope right at that point, but real implementations just pick one side and move on. Because values land exactly on zero so rarely, this never causes trouble in practice.
4Try it yourself
5Common misconceptions
It's easy to think an activation function is just an on/off switch for a neuron, but actually it also decides how strongly the neuron fires once it's on.
It's easy to assume the choice of function barely matters, but actually in a deep model, picking the wrong one can stop training from progressing at all.
It's easy to think the activation function itself sets where a neuron turns on, but actually it's the bias term that shifts that point back and forth.
7One-line summary
In shortAn activation function is the threshold-gated gate that decides whether a neuron's summed value gets passed on, and that single bend is what makes stacking layers mean anything at all.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02