MDPs, returns and values

Supervised learning gets the right answer for each input. Reinforcement learning (RL) gets a reward, a single number that may arrive only at the end of a long sequence of decisions. Language model training works this way. A whole completion is scored as good or bad, and the model has to work out which of its earlier choices mattered. The vocabulary below is what every RL paper assumes.

The loop

state s_t  →  agent picks action a_t  →  environment returns s_{t+1} and reward r_{t+1}

Repeat until the episode ends. For a language model, the state is the text so far, an action is the next token, and the episode is the prompt plus the completion.

Returns

The return G_t is the total reward from step t onward, with later rewards discounted by a factor γ between 0 and 1:

G_t = r_{t+1} + γ · r_{t+2} + γ² · r_{t+3} + …

Walk through a three-step episode with rewards 0, 0, 1 and γ = 0.9:

G_2 = r_3                     = 1
G_1 = r_2 + γ · r_3           = 0 + 0.9 = 0.9
G_0 = r_1 + γ · r_2 + γ² · r_3 = 0 + 0 + 0.81 = 0.81

The earlier a step is, the more the final reward has to be discounted to reach it. That is the credit problem in one picture: the step that set up the win gets only a slice of it.

For single-turn language tasks, people often use γ = 1, since the episode is short and the whole completion is scored at the end.

Values

• The state value V(s) is the expected return from state s if the policy keeps acting as it does.
• The action value Q(s, a) is the expected return if the agent takes action a in s first, then follows the policy.

The Bellman equation ties them together. A state's value equals the expected reward plus the discounted value of where you land next:

V(s) = E[ r + γ · V(s′) ]

That recursion is why dynamic programming and temporal-difference methods work. For LLM training, people often skip learning a value for every token and use the full return of each completion. The idea of a baseline value still shows up, as shown in policy gradients.

Common mistakes

• Assuming γ is always below 1. Discounting is for long tasks where distant rewards should count less. A single-turn completion can use γ = 1.
• Mixing up V and Q. V averages over the actions the policy might take. Q fixes the first action.
• Reading a reward as a label. A reward is a score for what happened, and it may be noisy or hackable.

Check yourself

1. With γ = 0.5 and rewards 0, 0, 1, what is G_0? Answer: 0.25, because 0.5² · 1.
2. How does Q differ from V? Answer: Q fixes the first action and then follows the policy, while V averages over the policy's choices.

Key takeaways

• RL learns from rewards in an MDP loop: state, action, next state, reward.
• A return is the discounted sum of future rewards.
• Values estimate expected return. Q adds the first action.
• Rewards can arrive late, so credit has to flow back through the discount.

Further learning

Next, see how a policy is improved by following the return in policy gradients.