Eval noise and sample size

LLM outputs change from run to run, and small eval sets are noisy. A move from 72% to 75% on 40 cases can be luck. This page shows how to size that noise, so you know when a change is real.

The big idea

Each case is a pass or a fail. A pass rate is therefore a proportion: the share of cases that passed. Any proportion measured on a sample wobbles. The size of the wobble is the standard error:

standard error = sqrt( p × (1 − p) / n )

p = pass rate (for example 0.70)
n = number of cases

A swing of about one standard error is normal. Treat smaller moves as noise until you have more cases.

Think of it like polling a crowd. Ask 40 people and the answer swings a lot. Ask 200 and it settles.

Walking through the arithmetic

Start with a pass rate of 70%.

p = 0.70, n = 40
  p × (1 − p) / n = 0.21 / 40  = 0.00525
  standard error  = sqrt(0.00525) ≈ 0.072   → about ±7 points

p = 0.70, n = 200
  p × (1 − p) / n = 0.21 / 200 = 0.00105
  standard error  = sqrt(0.00105) ≈ 0.032   → about ±3 points

Five times the cases cut the wobble roughly in half. The noise falls with the square root of n, so more cases help, but each gain costs more.

A real comparison

Say faithfulness on 100 cases goes from 64 to 71 after a prompt change. That is a 7-point gain.

Each rate has about 5 points of noise at this size. For a rate of 0.64 with n = 100, the standard error is sqrt(0.64 × 0.36 / 100) ≈ 0.048, about 5 points. The gap between two rates has roughly sqrt(5² + 5²) ≈ 7 points of noise. A 7-point gap is about one noise unit. That is suggestive, not proof.

Re-run the suite, add more faithfulness cases, or read which IDs flipped from pass to fail before you declare a win.

Per-category rates matter

A global number can hide a drop. Take a support bot with three categories: retrieval (80 cases), tool use (60) and refusal (40). The baseline pass rates are 78%, 70% and 92%. After you swap the model, they are 81%, 58% and 94%.

baseline overall = (0.78×80 + 0.70×60 + 0.92×40) / 180 = 78.4%
new overall      = (0.81×80 + 0.58×60 + 0.94×40) / 180 = 76.2%

The overall number fell by about 2 points, and tool use fell by 12. The 12-point drop matters more than the overall figure, because a broken tool breaks refunds. Block the upgrade, read the newly failing tool cases, and fix them first.

Practical rules

• Do not celebrate or panic over 2 to 3 point moves on a small suite.
• For high-stakes comparisons, add cases, or run each case several times and average.
• Compare two systems on the same inputs.
• Read rates by category, and make sure each category has enough cases.

Common mistakes

• Trusting a 3-point gain on 40 cases. That is about one standard error of noise.
• Looking only at the overall number. The category that matters most can fall while the total holds.
• Tuning against the set you report on. Keep a held-out slice so the number stays honest.

Check yourself

1. A pass rate moves from 72% to 75% on 40 cases. What do you do? Treat it as likely noise. Re-run, add cases, or inspect which IDs changed.
2. Why does a larger n shrink the noise? n sits in the denominator under the square root, so more cases mean a smaller standard error.

Key takeaways

• A pass rate is a proportion, and proportions wobble.
• At 40 cases, a swing of about 7 points is normal.
• At 200 cases, the wobble is about 3 points.
• Compare by category, with enough cases in each, on the same inputs.

Further learning

Back to the full picture in evaluating LLM apps. For how judges are compared in pairs, see LLM-as-a-judge.