Keystroke latency budget

The target for suggestions is about 100 ms at p99, measured from the keystroke to the pixels on screen. A fast server can still miss it. On a mobile network, the trip to the nearest edge is often the largest term. So the client and the edge do much of the work.

Think of the budget like a check-in deadline. Each stage takes a slice of time, and the slices must add up to less than the total.

The big idea

Three things spend the budget:

• Debounce: wait a short pause after typing before sending a request.
• Network: the round trip from the phone to the nearest edge.
• Lookup: either a cache hit at the edge or a trip to the origin trie.

The biggest lever is avoiding the origin. A popular prefix served from the edge or the browser costs a few milliseconds instead of a long round trip.

Walking through it

Here is the budget with typical numbers from the design.

Target p99:                100 ms

Path A, edge cache hit
  debounce                  40 ms
  client to edge RTT        35 ms
  edge cache hit             5 ms
  total                     80 ms   fits

Path B, edge miss, trie at origin
  debounce                  40 ms
  client to edge RTT        35 ms
  origin trie lookup        15 ms
  extra round trip          20 ms
  total                    110 ms   misses

Path B has no edge hit, so the 5 ms cache step is replaced by the origin lookup and an extra hop. That makes the total 110 ms, which is over budget. Origin speed alone cannot save it, so the design invests in edge TTLs and client caches.

To fix Path B, you could cut debounce from 40 ms to 30 ms. That brings the total to exactly 100 ms, but with no margin. Caching is the better fix.

The client tricks

These are the cheapest wins:

• Debounce for 30 to 50 ms, so mid-word typists do not send every character.
• Cancel in-flight requests when a newer prefix replaces them.
• Cache results by prefix on the device, and show cached results while the fresh ones load.
• Require a minimum length, such as two characters, before calling the network.
• Prefetch likely next prefixes when the user often continues a word.

Common mistakes

• Optimizing the trie walk while ignoring the network. The walk is a few microseconds. The round trip is tens of milliseconds.
• Setting debounce too long. Suggestions then feel late even when the server is fast.
• Sending one-character prefixes to the origin. Single letters are the hottest keys, so this is where most origin load comes from.
• Caching the empty prefix for a long time. Trends change, so keep those entries on a short TTL.

Check yourself

1. Using the numbers above, what single change brings Path B to exactly 100 ms?

Answer: Cut debounce from 40 ms to 30 ms, since 30 + 35 + 15 + 20 = 100.

2. Why does requiring two characters before a request help so much?

Answer: Single-letter and empty prefixes are the most common keys, so skipping them removes a large share of origin traffic.

Key takeaways

• Measure the budget from keystroke to pixels, not just server time.
• Popular prefixes must be served from the edge or the browser cache.
• Debounce trims requests but adds delay, so keep it small.
• Cancel outdated requests and show cached results while refreshing.

Further learning

Return to the search autocomplete case study for the full design, or watch the walkthrough linked there.