Mini-batch SGD and batch size

Computing the exact gradient over millions of examples on every step is too slow. Mini-batch stochastic gradient descent estimates the gradient from a small random subset instead. The estimate is noisy, but on average it points the right way. The noise often helps the model generalize.

The big idea

There are three ways to choose the data for each gradient:

• Full batch. Use every example. The gradient is exact, but each step is expensive.
• Stochastic (one example). Very cheap, and very noisy.
• Mini-batch. Average the gradient over B random examples. This is the default.

g ≈ (1/B) · sum over the batch of ∇Lᵢ(θ)
θ ← θ − η · g

Why the average is trustworthy

If the examples are sampled uniformly, the mini-batch gradient is an unbiased estimate of the full gradient. On average, it equals the true gradient.

Its noise shrinks with the batch size. The variance falls like 1/B, so the standard deviation falls like 1/√B. Quadruple the batch, and the noise is halved:

batch 64  → noise ∝ 1/√64  = 0.125
batch 256 → noise ∝ 1/√256 = 0.0625

Walking through the trade-offs

For large GPU training, follow the recipe for your architecture. When you change the batch size, adjust the learning rate. A common starting point is the linear scaling rule: multiply the learning rate by the same factor as the batch, with a warmup period. Some teams scale by the square root instead. If you multiply the batch by 8, try a learning rate multiplier between about 2.8 (the square root of 8) and 8. Then compare validation metrics, not only training speed.

Common mistakes

• Changing the batch size and leaving the learning rate alone. The two are linked. Adjust them together.
• Judging a batch size by training speed alone. A faster run can generalize worse. Compare validation results.
• Assuming the noise is always bad. Some noise helps the optimizer escape sharp minima.

Check yourself

1. The batch grows from 64 to 256. How does the gradient noise change? Answer: It halves, because the noise scales with 1/√B and 256 / 64 = 4.
2. You double the batch size. What should you consider changing? Answer: The learning rate, often raised with a warmup. Then check the validation metrics.

Key takeaways

• A mini-batch gradient is an unbiased but noisy estimate of the full gradient.
• The noise falls with the square root of the batch size.
• Batch size, learning rate and warmup are linked. Change them together.
• Validate every batch-size change before trusting it.

Further learning

Next, handle ravines and uneven gradients: momentum and Adam.