Benchmarking without lying

A micro-benchmark times a small piece of code in isolation. It answers questions like "is this hash map faster than that one?" They are easy to write and easy to get wrong. A benchmark that is wrong usually looks fast, which is the worst kind of wrong, because you will ship the change on the strength of it.

The big idea

A benchmark measures what the compiler and CPU actually ran, not what you wrote. Each pitfall below makes the timed code different from the code you meant to test.

Walking through dead-code elimination

Suppose you time sum_squares(1000) and the result is never used. An optimizing compiler can see that the answer has no effect, remove the call, and report a time near zero.

benchmark: sum_squares(1000)   0.3 ns per op

That number is the loop overhead, not the work. Assign the result to a variable the compiler must keep. In Go, that is a package-level sink variable. In JMH, it is a Blackhole. In Google Benchmark, it is DoNotOptimize.

Use the harness for your language

Each language has a harness that handles iteration counts, warmup and timing. Use it instead of a hand-rolled timing loop.

These are sketches. The Go one assumes a sampleOrder helper that builds a realistic fixture, and the Python one assumes a sample_order helper. Run the Go benchmark with go test -bench=Encode -benchmem -count=10, then compare runs with benchstat.

func BenchmarkEncodeOrder(b *testing.B) {
	order := sampleOrder(500) // built outside the timed loop
	for b.Loop() {            // Go 1.24+: manages the timer and keeps results alive
		if _, err := json.Marshal(order); err != nil {
			b.Fatal(err)
		}
	}
}

import orjson

def test_encode_order(benchmark):
    order = sample_order(500)
    result = benchmark(orjson.dumps, order)  # handles rounds and warmup
    assert result                            # use the result

On the JVM, use JMH, the harness written by JVM performance engineers. Hand-rolled System.nanoTime() loops are almost always wrong there. In Go, benchstat tells you whether the difference between two runs is real.

Noise: read the spread, not one number

Run each version many times. Compare the two sets of results with a statistical tool. Eyeballing two numbers is not enough.

old: 102 ns, 98 ns, 105 ns, 101 ns   (mean ≈ 101 ns)
new:  99 ns, 104 ns, 97 ns, 103 ns   (mean ≈ 101 ns)

These runs show no real difference. A 3% gain inside 5% noise is not a result.

Common mistakes

• Trusting one run. Run many times and report the spread.
• Benchmarking tiny inputs. Use the size production sees, so cache misses are real.
• Letting the compiler delete the work. Consume every result.
• Using the benchmark to decide whether code matters. Profile the real service first.

Check yourself

1. A benchmark reports 0.3 ns for a function that should take microseconds. What happened? The compiler most likely removed the work, because the result was unused.
2. Version A averages 101 ns and version B averages 101 ns, with a 5% spread. What do you conclude? No real difference. The gap is inside the noise.

Key takeaways

• A micro-benchmark measures the code the compiler actually ran. Check that your results are plausible.
• Warm up, consume results, use realistic data and compare with statistics.
• Use JMH on the JVM, testing.B with benchstat in Go, or pytest-benchmark in Python.
• A faster function is only worth shipping when a profile of the real system says it matters.

Further learning

The async-profiler project is the standard tool for profiling the JVM under realistic load, the setting where micro-benchmarks fall short.

async-profiler: sampling CPU and heap profiler for JavaSampling CPU and HEAP profiler for Java featuring AsyncGetCallTrace + perf_events - async-profiler/async-profilerGitHub

Next, return to the hub, performance and profiling, for a full worked investigation.