Convolutional networks

Images are not bags of pixels in arbitrary order. Nearby pixels are related, and a cat should be recognized whether it sits on the left or the right. Convolutional neural networks (CNNs) build these assumptions into the model. They use small filters, share their weights across the image, and stack layers into a hierarchy of features. CNNs dominated vision from AlexNet (2012) through the ResNet era, and many production systems still use them alongside Vision Transformers.

Concepts in this topic

Read in this order. Each page builds on the one before it.

Landmark models: LeNet (1989 to 1998) used convolution, pooling and fully connected layers on digits. AlexNet (2012) brought deep GPU CNNs with ReLU and dropout to ImageNet. VGG (2014) stacked 3×3 convolutions. ResNet (2015) added skip connections and trained 50 to 152 layers. EfficientNet and ConvNeXt later scaled and modernized the design.

Common mistakes across the topic

• Calling CNNs invariant to translation. Convolution is equivariant: shifting the input shifts the feature map. Pooling and global pooling later make the output more invariant. Which layer you mean matters.
• Mismatched preprocessing. Use the same resize, crop and normalization at training and at serving. Most "model regressions" are preprocessing drift.
• Wrong normalization statistics. The ImageNet mean and standard deviation are wrong for single-channel X-rays. Recompute them for your data.

Other pieces you will meet

• BatchNorm normalizes each channel across the batch and spatial axes, then applies a learned scale and shift. It allows higher learning rates. Small batches make its statistics noisy, so GroupNorm or LayerNorm variants help. Call model.eval() at validation time, because BatchNorm then uses running statistics. At inference it can be folded into the preceding convolution.
• Augmentation is the real regularizer for vision. Random crops, flips and color jitter usually beat dropout. Some domains forbid certain flips: a flip can swap left and right anatomy in a medical image.
• Detection and segmentation keep spatial structure. Detectors such as YOLO and Faster R-CNN predict boxes and classes. Segmenters such as U-Net predict a label per pixel. The pattern is a pretrained encoder, a task head and a matching loss such as focal, Dice or IoU.
• Vision Transformers split the image into patches and treat them as a sequence. They need more data than CNNs, but they compete well at scale. Patch size plays the role of the first stride.

Key takeaways

• Convolution applies a small filter at every position, with shared weights.
• Output size depends on kernel size, stride and padding. Channels are a separate design choice.
• Stacking layers builds a hierarchy: edges, then parts, then objects.
• Residual connections make very deep CNNs trainable, a pattern used far beyond vision.
• Start from a pretrained backbone unless your domain is very different and your data is very large.

Further learning

3Blue1Brown on what a convolution is. The signal-processing intuition carries over:

CS231n: Convolutional Neural Networks for Visual RecognitionCourse materials and notes for Stanford class CS231n: Deep Learning for Computer Vision.cs231n.github.io Deep Residual Learning for Image Recognition (ResNet)Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the lay…arXiv.org