Guide

LLM fundamentals

Self-consistency and best-of-n

Sample several times and pick with a vote or a verifier. You buy robustness with n times the output tokens, and only when the answer is checkable.

22 of 155 · 3 min read

When a single sample is brittle, generate more than one. Self-consistency takes several reasoning paths and keeps the majority final answer. Best-of-n generates n candidates and picks with a verifier: unit tests, a rubric, or an LLM judge. Both multiply cost by n. Neither helps if every sample shares the same mistake.

Concept

Before you start: chain-of-thought is the model writing intermediate steps before the answer. Self-consistency is majority vote over several of those paths (Wang et al.). Best-of-n is the same fan-out with a checker instead of a vote.

The big idea

If you can fire five samples concurrently, wall-clock latency stays close to one sample while cost scales with n. If your provider tokens-per-minute budget is tight, best-of-n can trip rate limits even when a single-stream path was fine. Budget tokens for the fan-out.

Example

Pass@1 on a coding task is 40% at T=0.2. With n=5 independent samples and a unit-test picker, pass@5 might reach ~70% depending on diversity. You paid about 5× output tokens. Compare that cost to upgrading the model tier. Sometimes best-of-n on a mid model beats one shot on a frontier model for verifiable tasks.

When answers are not automatically checkable, majority vote can still entrench a shared mistake. Use judges carefully and spot-check against humans.

Trade-off

Self-consistency buys robustness on math-like tasks but multiplies output cost and latency. Prefer it when you have a cheap checker (unit tests, exact match). Prefer a single low-temperature call with tools when the task is retrieval-bound. Prefer a stronger model when n would have to be huge to close the gap.

Reasoning / "thinking" models frequently ignore or fix temperature and top-p. Read the provider note before tuning yourself into a no-op.

Walking through a coding pick

You have a function to implement and a unit test. Settings: T=0.2, n=5, picker = "first candidate whose tests pass."

  1. Fire five completions in parallel.
  2. Run the tests on each.
  3. Return the first pass. If none pass, return a structured failure, not a random leftover.

Cost is about five output sequences. The 40% to ~70% lift is only a win if the tests are real. If the picker is "ask another model which looks nicest," you have added a judge with its own error rate.

For classification, skip the fan-out. Read logprobs of the label tokens at T=0.

Common mistakes

  • Best-of-n without a checker. A vote can lock in a popular wrong answer.
  • Forgetting TPM. Five parallel samples can 429 a path that was fine at n=1.
  • Using this on retrieval-bound tasks. More samples of the same empty context will not find the doc.

Check yourself

  • Pass@1 is 40% and you have unit tests. Why might n=5 on a mid model beat one frontier call? The tests pick a correct candidate. You paid 5× output on the cheaper model.
  • When does majority vote make things worse? When most paths share the same mistake and there is no independent checker.

Key takeaways

  • Self-consistency is a majority vote over sampled reasoning paths. Best-of-n is fan-out plus a verifier.
  • Both cost about n times the output tokens. Parallelism keeps wall-clock close to one sample.
  • Use them when the answer is checkable. Otherwise a shared mistake can win the vote.
  • Record n, temperature and the picker next to the prompt version.

Further learning

Self-Consistency Improves Chain of Thought Reasoning (arXiv:2203.11171)Chain-of-thought prompting combined with pre-trained large language models has achieved encouraging results on complex reasoning tasks. In this paper, we propose a new decoding strategy, self-consistency, to replace t…arXiv.org

Back to the map: sampling and decoding.

Decoding controls

Max tokens, stop sequences, penalties and logprobs. Temperature 0 removes sampling noise. It does not make two API calls bit-identical.

LLM-as-a-judge

Using models to grade models: when a judge is the right grader, how to design one, and why an uncalibrated judge is a vibes function.

Chain-of-thought and extended thinking

When asking the model to reason step by step helps, how native thinking differs from prompted reasoning, and how to keep the reasoning out of the answer.