Multilayer perceptrons and the forward pass

Stack neurons into layers, and the layers into a chain, and you have a multilayer perceptron (MLP). Each layer multiplies its input by a weight matrix, adds a bias and applies an activation. The forward pass is the flow of data through that chain. Tracking the shapes at each step tells you what the network is doing.

The big idea

Layer ℓ maps its input h of width d_(ℓ−1) to an output of width d_ℓ:

z = W · h + b        W has shape (d_ℓ, d_(ℓ−1))
h_next = σ(z)

The network starts with the input x and applies each layer in turn. The last layer often uses a different activation, or none, depending on the task.

Picture a network with 4 inputs, two hidden layers of 8 units, and 3 outputs:

The weight matrices are 8 × 4, 8 × 8 and 3 × 8. Each bias has the width of its layer's output.

Walking through a batch

In practice, you process a batch of examples at once. Each row of the input matrix is one example. This example pushes 32 flattened 28 × 28 images through two hidden layers and a 10-class output. It runs as written.

import numpy as np

def relu(z):
    return np.maximum(0, z)

rng = np.random.default_rng(0)
X = rng.normal(size=(32, 784))           # batch of 32 flattened images
W1 = rng.normal(size=(256, 784)) * 0.01  # (out, in), as in nn.Linear
W2 = rng.normal(size=(128, 256)) * 0.01
W3 = rng.normal(size=(10, 128)) * 0.01
b1, b2, b3 = np.zeros(256), np.zeros(128), np.zeros(10)

H1 = relu(X @ W1.T + b1)                 # (32, 256)
H2 = relu(H1 @ W2.T + b2)                # (32, 128)
logits = H2 @ W3.T + b3                  # (32, 10)
print(H1.shape, H2.shape, logits.shape)

Output:

(32, 256) (32, 128) (32, 10)

Each layer multiplies by the transpose of a (out, in) weight matrix. That matches the convention of PyTorch's nn.Linear. Some code stores weights as (in, out) and multiplies without the transpose. Either is fine, as long as you stay consistent.

Hidden activations are learned features. Early layers often pick up simple patterns, and later layers combine them. In an MLP on flattened pixels, that hierarchy is harder to see than in a convolutional network, but the idea is the same.

Common mistakes

• Mixing weight conventions. The shape error shows up as a matrix mismatch. See batching and broadcasting.
• Ignoring the final layer's activation. The output head depends on the task. See output heads and losses.
• Reading hidden units as named features. They are learned, and they are rarely interpretable one by one.

Check yourself

1. An MLP has 20 inputs and a hidden layer of 16 units. What is the shape of that layer's weight matrix in (out, in) form? Answer: (16, 20).
2. A batch of 64 inputs passes through a network with 5 outputs. What is the output shape? Answer: (64, 5).

Key takeaways

• An MLP is a chain of matrix multiplies, each followed by an activation.
• A batch moves through the network as one matrix per layer.
• Track shapes at every layer, and most bugs show up immediately.
• The final layer must match the task.

Further learning

Next, choose the activation for each layer: activation functions and vanishing gradients.