Neural networks

A neural network is a stack of simple functions: linear maps interleaved with elementwise nonlinearities. Its parameters are tuned so the whole composition maps inputs to outputs. Backpropagation, GPUs and transformers are ways to fit and scale that composition. This topic covers the composition itself.

If you already know linear and logistic regression, you know one layer of a network. Depth is what changes the game. Stacking nonlinear layers lets the model build features that no single linear map could construct.

Concepts

Capacity and overfitting

More parameters mean more capacity. Capacity without enough data or regularization leads to overfitting: the training loss falls while the validation loss rises. Weight decay, dropout, early stopping and more data all still apply to MLPs. See training neural networks well.

For tabular data, gradient-boosted trees often beat MLPs out of the box. Neural nets earn their cost when the inputs have structure, such as images, text or audio, or when the dataset is very large.

When not to use an MLP

• Tabular data of modest size. Gradient boosting, covered in trees and ensembles, usually wins.
• Images. Use convolutional networks, because nearby pixels matter.
• Variable-length text. Use sequence models or transformers, because weight sharing across positions matters.
• A calibrated probability from a few linear features. Logistic regression is enough, and it is easier to debug.

MLPs shine as heads, fusion layers, and as blocks over fixed-size dense vectors, such as embeddings of categorical features. The feed-forward half of every transformer layer is also an MLP.

Batch size

Batch size does not change the architecture, which is the map from one input to one output. It changes the noise in the gradient, and that noise changes the learning rate that works best. Mini-batch SGD and batch size covers this.

Common mistakes

• Skipping the nonlinearity. Without it, the stack is one linear map.
• Choosing a loss that does not match the head. Use cross-entropy with softmax, and MSE with a linear output.
• Training an MLP on pixels or tokens by default. Architectures with the right inductive bias usually learn faster.

Key takeaways

• A neuron is a weighted sum and a bias, followed by a nonlinearity.
• An MLP is a chain of matrix multiplies, with activations between them.
• ReLU or GELU for hidden layers. The output head and loss depend on the task.
• Depth builds layered features, and width adds capacity in parallel.
• Parameter counts give you the memory cost before training starts.

Further learning

3Blue1Brown's visual series is the fastest intuition upgrade for what networks compute and how they learn:

Chapter 2 connects networks to gradient descent on a loss surface:

Understanding Deep Learning (Simon Prince)Understanding Deep Learningudlbook.github.io Neural Networks and Deep Learning (Nielsen)Neural networks and deep learningneuralnetworksanddeeplearning.com