Gradient clipping and accumulation

Two practical tools show up in almost every transformer training loop. Gradient clipping stops a rare spike from destroying the run. Gradient accumulation fakes a larger batch when the GPU cannot hold one.

The big idea

import torch
from torch import nn

p = nn.Parameter(torch.tensor([3.0, 4.0]))  # grad we will set to this vector
p.grad = torch.tensor([3.0, 4.0])           # L2 norm = 5
torch.nn.utils.clip_grad_norm_([p], max_norm=1.0)
print(p.grad.tolist())

Output:

[0.6, 0.8]

The vector (3, 4) had norm 5, so it was scaled by 1/5. Typical steps barely change. Rare spikes get crushed. Log the pre-clip norm. A sudden climb warns of instability. max_norm=1.0 is a common default in RNN and transformer training.

Walking through accumulation

Accumulation simulates a larger batch when memory is tight:

for micro in microbatches:
    loss = compute(micro) / N
    loss.backward()   # grads accumulate
optimizer.step(); optimizer.zero_grad()

Effective batch size = micro_batch × N × num_gpus. Keep the loss averaging correct so the effective learning-rate meaning stays stable. If you forget to divide by N, you have scaled the gradient — and the step — by N.

BatchNorm statistics see only the micro-batch. That is another reason transformers prefer LayerNorm. See normalization layers.

A picture: clipping is a speed limiter on a downhill road. Accumulation is taking the same trip in several cars and averaging the directions before you turn the wheel. The picture breaks if the cars (micro-batches) are so small that BatchNorm's "class average" is noise.

Warmup plus cosine (or a known step schedule) still beats a constant LR for long runs. See conditioning and schedules. Mismatching "scheduler step per batch" versus "per epoch" is how you accidentally use 100× fewer decay steps than you intended.

Common mistakes

• Forgetting to divide the micro-batch loss by N. The effective LR jumps by N.
• Clipping after the optimizer step. The spike already landed.
• Using BatchNorm with tiny micro-batches and blaming Adam. The batch statistic is the noise.

Check yourself

1. Pre-clip grad norm is 100, max_norm=1. What happens to each gradient? Answer: Every grad is scaled by 0.01.
2. You accumulate 8 micro-batches and do not divide the loss by 8. What did you just do to the step size? Answer: You scaled the gradient, and the update, by 8.

Key takeaways

• Clipping caps the global grad norm and saves runs from rare spikes.
• Log the pre-clip norm. A sudden climb is an instability warning.
• Accumulation fakes a larger batch. Divide the loss by the number of micro-batches.
• BatchNorm still sees the micro-batch. LayerNorm does not care.

Further learning

Next: neural-net regularization.