RNNs and backprop through time

Language, audio and time series are sequences: the next element depends on what came before. A recurrent neural network (RNN) keeps a hidden vector that summarizes the past, then updates it as each new token arrives. Unrolled in time, that loop is a deep feed-forward net with shared weights across steps.

The big idea

An RNN updates a hidden state h_t:

h_t = tanh(W_h h_{t−1} + W_x x_t + b)
y_t = W_y h_t + c

The same W_h and W_x are used at every step. That is the inductive bias: "the rule for combining the past with the present does not change with time."

To train, run forward for T steps, then backprop through the unrolled graph. Truncated BPTT limits T to save memory. The sequential dependence is painful on GPUs, which want huge parallel matrix multiplies.

Walking through a tiny character RNN

The objective is still next-token prediction. A character-level toy with vocabulary 27 (a–z plus space) and context 1 is a 27×27 table of transition probabilities. An RNN is the same game with a learned hidden state instead of a count table.

Here is a sketch that trains next-character prediction on a tiny corpus. After a few hundred steps it memorizes local patterns such as "hell" and "worl".

import torch
import torch.nn as nn

text = "hello world " * 200
chars = sorted(set(text))
stoi = {c: i for i, c in enumerate(chars)}
data = torch.tensor([stoi[c] for c in text], dtype=torch.long)

class CharRNN(nn.Module):
    def __init__(self, vocab, hidden=64):
        super().__init__()
        self.emb = nn.Embedding(vocab, hidden)
        self.rnn = nn.LSTM(hidden, hidden, batch_first=True)
        self.head = nn.Linear(hidden, vocab)

    def forward(self, x, state=None):
        h, state = self.rnn(self.emb(x), state)
        return self.head(h), state

B, T = 16, 32
model = CharRNN(len(chars))
opt = torch.optim.AdamW(model.parameters(), lr=1e-3)
loss_fn = nn.CrossEntropyLoss()
ix = torch.randint(0, len(data) - T - 1, (B,))
x = torch.stack([data[i : i + T] for i in ix])
y = torch.stack([data[i + 1 : i + T + 1] for i in ix])
logits, _ = model(x)
loss = loss_fn(logits.reshape(-1, len(chars)), y.reshape(-1))
print(round(loss.item(), 3))

The loss prints a number near ln(vocab), typically around 2, before any real training. Scaling context and data is what later demands transformers. The objective does not change.

At inference an RNN generates token by token while carrying (h, c) across calls. State size stays constant, which is streaming-friendly. A transformer decoder instead carries a KV cache that grows with generated tokens.

The GPU argument with numbers

Hidden size H = 1024, sequence length T = 512. An LSTM step is on the order of 8 H² FLOPs per token, so about 8 H² T FLOPs in total, but wall clock is roughly T sequential kernel launches.

A transformer layer does attention ~T² H and MLP ~8 T H² with all tokens in parallel. For moderate T, total FLOPs can be higher, yet GPU utilization is far better, so training tokens/sec wins.

Common mistakes

• Thinking the hidden state is a lossless memory. It is a fixed-size summary. Long histories get crushed.
• Unrolling the full corpus as one graph. Truncate BPTT or you run out of memory.
• Padding without packing. Padded RNN steps waste compute unless you use packed sequences.

Check yourself

1. Why is an RNN forward pass hard to parallelize on a GPU? Answer: h_t depends on h_{t−1}, so steps cannot run at the same time.
2. What is BPTT? Answer: Backprop on the unrolled RNN, with weights shared across time.

Key takeaways

• An RNN compresses history into h_t with shared weights at every step.
• BPTT is backprop on that unrolled chain. Truncation caps memory.
• The sequential dependency wastes GPUs compared with attention.
• The language-modeling objective stays next-token cross-entropy.

Further learning

Next: vanishing gradients and LSTMs.

Karpathy's makemore intro builds language modeling from counts upward: