Batch vs online inference

Some predictions can wait an hour. Others must return before a page loads. The choice between batch and online inference sets your infrastructure, your latency budget and how much can go wrong at serving time.

The big idea

Batch is simpler. You can replay it, debug it and rerun it, and nothing depends on a live feature store. The cost is staleness: a nightly score is up to a day old when a user acts on it.

Online is required when the best features exist only at request time, such as the contents of a shopping cart. Every online feature becomes a runtime dependency, so each one needs an owner and an outage plan.

Walking through a latency budget

Suppose the product needs a 200 ms response at the 99th percentile. Authentication and the database take 80 ms. That leaves 120 ms for the model. If fetching features takes 40 ms, the model itself has 80 ms.

That arithmetic decides the model size, how fast the feature store must be, and whether you should precompute scores instead.

Choosing a mode

1. Does the best feature exist only at request time? If yes, go online.
2. Is a score that is hours old still useful? If yes, go batch.
3. Otherwise, use nearline, updating scores as events arrive.

Packaging the model

Model packaging matters as much for serving as the choice of mode. Pickle files are fragile across library versions. ONNX, TorchScript and model servers such as Triton are more portable. Pin the dependency versions. Store the training commit SHA and the dataset snapshot ID next to the artifact.

Common mistakes

• Building an online path "just in case". Every online feature is a dependency that can fail at 3 a.m.
• Assuming batch scores are fresh. A nightly score can be a day old.
• Leaving the latency budget until launch. Work out the budget before choosing the model size.

Check yourself

1. A product shows recommendations when a page loads, from features that change weekly. Batch or online? Answer: Batch. The features barely change within a day, so scores can be precomputed and looked up.
2. A fraud check needs the items currently in the cart. Batch or online? Answer: Online, because the cart exists only at request time.

Key takeaways

• Batch is simpler and easier to replay. Online is required when features exist only at request time.
• The latency budget is a sum of parts, and the model gets what is left.
• Each online feature is an operational dependency.
• Package the model with pinned versions and a record of its training inputs.

Further learning

Next, check that training and serving compute the same features: training-serving skew and point-in-time features.