The neuron and why it needs a nonlinearity

A neural network is built from one repeated unit, the neuron. It computes a weighted sum of its inputs, adds a bias, and passes the result through a nonlinear function. The last step matters most. Without it, however many layers you stack, the whole network is still one linear map.

The big idea

A neuron computes two things:

z = w · x + b        the pre-activation
a = σ(z)             the activation, after the nonlinearity

Taking the second step away collapses the stack. Two layers without a nonlinearity are two matrix multiplies in a row, and a product of matrices is one matrix.

Here is a small check. Let W1 = [[1, 2], [0, 1]] and W2 = [[3, 0], [1, 1]]. Their product is W2 · W1 = [[3, 6], [1, 3]]. For the input x = [1, 1]:

two layers:  W1·x = [3, 1]  then  W2·[3, 1] = [9, 4]
one layer:   (W2·W1)·x = [3·1 + 6·1, 1·1 + 3·1] = [9, 4]

Same answer. A stack without a nonlinearity is just one layer in disguise.

An analogy: stacking sheets of tinted glass gives one sheet of glass. A bend between the sheets lets the stack do what one flat sheet cannot. The picture stops at real networks, which bend in hundreds of directions at once.

Walking through one neuron

Take inputs x = [1, 2], weights w = [0.5, −0.25], bias b = 0.1, and a ReLU as the nonlinearity, max(0, z).

z = 0.5·1 + (−0.25)·2 + 0.1 = 0.1
a = max(0, 0.1) = 0.1

Now change the input to x = [1, 4]:

z = 0.5·1 + (−0.25)·4 + 0.1 = −0.4
a = max(0, −0.4) = 0

The neuron is silent for that input. ReLU produces exactly zero whenever the pre-activation is negative.

Where this came from

Rosenblatt's perceptron (1958) used a hard threshold: output 1 if w · x + b > 0, else 0. It cannot separate classes that a single line cannot separate, and XOR is the standard example. Modern networks keep the weighted sum, replace the hard threshold with a smooth or piecewise-linear function, and stack many units.

A single neuron is also a model you already know. With a sigmoid and log loss, it is logistic regression. With a linear output and squared error, it is linear regression. A network is many of these composed together.

Common mistakes

• Believing depth alone adds power. Depth adds power only with a nonlinearity between the layers.
• Mixing up the pre-activation and the activation. Code often names both z and a. Check which one a line of code is using.
• Forgetting the bias. Without it, a neuron's decision boundary must pass through the origin.

Check yourself

1. What does a three-layer network with no nonlinearity compute? Answer: A single linear map, because a product of matrices is one matrix.
2. A neuron has w = [1, 1], b = −3, and input x = [1, 1]. What is its ReLU output? Answer: z = 1 + 1 − 3 = −1, so ReLU gives 0.

Key takeaways

• A neuron is a weighted sum plus a bias, then a nonlinearity.
• Without a nonlinearity, stacked layers collapse into one linear map.
• A single neuron with a sigmoid and log loss is logistic regression.
• Next, see how neurons stack into layers: multilayer perceptrons.

Further learning

3Blue1Brown's first deep learning chapter animates weighted sums, biases and activations layer by layer, which makes the single-neuron picture concrete.