Deep learning
Residual connections and ResNet
A residual block adds its input back to its output, so each block only needs to learn a change. That keeps very deep networks trainable and gives gradients a path that skips the weights.
Stacking more plain convolution layers stopped helping at some depth, and training got harder. ResNet fixed this with a skip connection. Each block computes a change and adds its input back. The block only has to learn the difference, so it can default to doing nothing. Transformers use the same idea.
Concept
A residual block computes y = x + F(x), where F is a few convolutions, normalizations and ReLUs. The addition is the skip. If F outputs zero, the block is the identity.
Walking through one block
Take a scalar input x = 2, with F(x) = 0.1 from the block's layers:
y = x + F(x) = 2 + 0.1 = 2.1Now suppose the block's layers output zero. Then y = x, and the block passes its input through unchanged. The optimizer can add depth without forcing every block to do something useful from the start.
Why the gradient flows better
Differentiate the block with respect to its input:
∂y/∂x = 1 + ∂F/∂xThe 1 is a direct path. The gradient reaches earlier layers even when ∂F/∂x is small. A plain stack has no such path, so its gradient is multiplied by each layer's weights and can shrink toward zero.
Tip
A residual connection is the reason you can think of a deep ResNet as an ensemble of many shorter paths. Gradient flows along all of them.
Blocks that change shape
The addition needs x and F(x) to have the same shape. Downsampling blocks use a strided convolution inside F, which halves the grid, so the skip must match that. They use a projection shortcut, usually a 1 × 1 strided convolution, to reshape x before adding.
Landmark architectures
| Model | Year | Idea |
|---|---|---|
| LeNet | 1989 to 1998 | Convolution, pooling and fully connected layers on digits |
| AlexNet | 2012 | Deep GPU CNN with ReLU and dropout, a breakthrough on ImageNet |
| VGG | 2014 | Stacks of 3 × 3 convolutions, showing depth with simplicity |
| ResNet | 2015 | Residual connections, trained at 50 to 152 layers |
| EfficientNet, ConvNeXt | later | Scaling recipes and modernized convolutional designs |
Trade-off
Deeper or wider. Residual networks showed that depth helps when optimization works. For a fixed compute budget, scaling width and resolution together can also help, as EfficientNet does. For a new project, start from a pretrained ResNet or ConvNeXt rather than designing a backbone.
Common mistakes
- Adding a skip with mismatched shapes. The sum fails, or it broadcasts silently into the wrong shape. Use a projection when the shape changes.
- Thinking the skip makes the network shallower. The layers still run. The skip only gives the signal and gradient a shorter path.
- Assuming any residual design works. Placement of the normalization and the ReLU changes behavior. Copy a known block design first.
Check yourself
- If
F(x) = 0, what does a residual block output? The inputx, unchanged. - Why does a downsampling block need a projection shortcut? Its residual branch halves the grid, so the input has to be reshaped to match before the addition.
Key takeaways
- A residual block computes
x + F(x), so it can default to the identity. - The
+1in∂y/∂xgives gradients a direct path back through the network. - Downsampling blocks reshape the skip with a projection.
- The same residual pattern appears in transformers and most modern deep networks.
Further learning
Pretrained residual networks are the usual starting point for new vision projects: transfer learning for vision.
Related
Convolutional networks
How CNNs use small shared filters, stacked layers and residual connections to learn images, and why transfer learning from pretrained models is the default.
Backpropagation from scratch
How training computes the gradient of the loss for every weight in one backward pass. Covers computational graphs, reverse-mode autodiff and the bugs that break training.
Modern LLM architectures
Mixture of experts, GQA and MLA, long context, state space models and multimodal transformers.