Closed-loop vs open-loop load

How you drive load changes which percentiles you see. Closed-loop keeps N requests in flight. Open-loop fires requests on a timer, often Poisson at rate λ, whether or not earlier ones have finished. They answer different questions.

The big idea

Closed-loop is a call center with N agents who pick up the next ticket as soon as they finish the last. Open-loop is a public API: the phone keeps ringing at λ even when every agent is busy. Under overload, the open-loop queue grows without bound. That is exactly what you want to see before production does.

The analogy breaks because the "agents" here are GPU batch slots, and the client in a closed-loop test is itself holding those slots. Latency includes queueing the client induced.

Walking through the two drivers

Closed-loop (fixed concurrency). When one request finishes, another starts. Easy to reason about. Natural for "how many concurrent users can we hold." The throughput you measure is bounded by N and by how fast requests complete.

Open-loop (arrival process). Requests arrive regardless of completions. This matches public APIs and gateways better. When λ exceeds goodput, queues grow and latency percentiles explode.

In a real report, plot throughput on x and TTFT p95 on y, and draw a horizontal SLO line (for example 1500 ms). The rightmost point under the line is your operating capacity for that config.

Common mistakes

• Only closed-loop tests. You never see unbounded queue growth, so autoscaling looks fine until production.
• Open-loop with no cap and a short run. Percentiles are noise until the queue has had time to grow.
• Comparing the two as if they were the same N. Concurrency N is not the same as QPS = N.

Check yourself

1. You want the knee of the latency–throughput curve while tuning vLLM. Which driver? Answer: closed-loop concurrency sweep.
2. You want to see what happens when QPS exceeds goodput. Which driver? Answer: open-loop, so the queue can grow without the client holding slots.

Key takeaways

• Closed-loop keeps N in flight. It is the fast way to find the curve.
• Open-loop arrives at rate λ. It is the honest way to test overload and autoscaling.
• The operating point is the rightmost point under the SLO line.

Further learning

Next, run the concurrency sweep and read the three regions of the curve.