Guide

Math toolkit

Uncertainty and sample size

A metric from a finite test set is an estimate, not a fact. The law of large numbers, the central limit theorem and the bootstrap tell you how much to trust it.

21 of 187 · 2 min read

A model gets 40 clicks out of 1,000 impressions. Is the 4% rate real, or could it be noise? The honest answer depends on sample size. Three ideas give you a range instead of a single number.

Two limit theorems

Concept

The law of large numbers (LLN) says a sample average settles on the true expectation as the sample grows. The central limit theorem (CLT) says that for a large sample, the sample mean is roughly Normal with variance σ² / n, even when the raw data are not Normal.

The LLN is why large eval sets give steadier metrics. The CLT is why you can write a formula for a confidence interval at all.

Walking through a click rate

Here p̂ = 40 / 1,000 = 0.04. The standard error of the estimate is:

SE = √( p̂(1 − p̂) / n ) = √( 0.04 · 0.96 / 1000 ) = √0.0000384 ≈ 0.0062

A rough 95% interval is p̂ ± 1.96 · SE:

1.96 · 0.0062 ≈ 0.012
interval ≈ [0.028, 0.052]

So the true rate is probably between 2.8% and 5.2%. A competitor at 4.5% is not clearly better.

If you quadruple the impressions to 4,000 at the same rate, the width halves. The width scales with 1 / √n, so you need four times the data to halve it.

The bootstrap

When a metric has no simple variance formula, such as AUC or BLEU, use the bootstrap:

  1. Resample the eval set with replacement, to the same size.
  2. Recompute the metric on the resample.
  3. Repeat many times, then read the 2.5th and 97.5th percentiles.

The spread of those results is your uncertainty estimate.

Common mistakes

  • Reporting one number with no sample size. Give the N or an interval, so readers can judge the noise.
  • Calling a small lift a win. A one-point AUC gain on 200 examples is often noise.
  • Checking an A/B test over and over. Stopping at the first good-looking result inflates false wins. Use a proper A/B framework that accounts for repeated looks.

Check yourself

  1. Why does a bigger eval set give a steadier metric? Answer: the spread of the sample mean shrinks with 1 / √n, by the LLN and CLT.
  2. When would you use the bootstrap instead of a formula? Answer: when the metric, such as AUC or BLEU, has no simple variance formula.

Key takeaways

  • The LLN says averages settle as samples grow.
  • The CLT makes sample means roughly Normal, so an interval is estimate ± 1.96 · SE.
  • The interval width shrinks with √n.
  • The bootstrap estimates uncertainty for metrics that have no formula.

Further learning

For how sample size fits into model evaluation, read evaluating models. The section overview is probability, statistics and information.

Evaluating models

Choose metrics that match the decision, set thresholds from costs, check calibration, and guard against leakage. A high offline score means little until it holds up on real decisions.

Entropy and cross-entropy

Entropy measures how surprising a distribution is on average. Cross-entropy measures how well a model's guesses match reality, and it is the classification loss.

Evaluating LLM apps

A fixed set of inputs plus a way to score outputs, so you can tell whether a change helped.