Serving classic ML models

Not every score needs a Transformer. Fraud models, ranking features, recommendations and pricing often still ship as gradient-boosted trees or small MLPs.

• Prefer CPU with ONNX Runtime. TensorRT-LLM is irrelevant, and a GPU would sit idle.
• Latency budgets are often sub-5 ms p99. Colocate with the feature store or the app.
• NVIDIA Triton Inference Server shines here: one daemon, many model backends (ONNX, TensorRT, PyTorch, FIL for trees), shared dynamic batching and scheduling policies per model.

Putting XGBoost on a GPU Triton instance "for consistency" usually raises latency and cost versus a local CPU library.

Walking through batching

Batch GBDTs only if feature-row batching is natural (e.g. scoring 500 candidates). For single-row online paths, batching windows must stay tiny or you lose the latency race to a dumb local library.

This is a sketch of a Triton model repository, not a runnable server:

model_repository/
  fraud_xgb/1/model.onnx
  embed_mini/1/model.onnx
  config.pbtxt   # instance_group, dynamic_batching, max_batch_size

One gateway, many model types is fine. One autoscaling signal for all is not. Scale embedding Deployments on batch queue depth, diffusion on GPU util or queue time, LLMs on KV and waiting queue as in autoscaling, and classic ML on p99 score latency.

Teams often ask whether embeddings can share the LLM’s H100s “when idle.” Sometimes yes for nightly indexing. Rarely yes for online query embedding next to chat decode:

• LLM decode wants large KV arenas. Embedding batches want contiguous free memory for big padded tensors. They fight.
• A chat traffic spike starves the embedder and blows RAG p95.
• Failure domains differ: a bad LLM graph capture restart should not take down search.

Export p99 score latency and feature-null rates. Bad features look like model bugs. Without per-model_name tags, on-call pages “GPU high” with no idea which of five colocated models is guilty.

The big idea

A pocket calculator does not need a data-center GPU. Forcing it onto one makes the answer later and the bill higher. The picture breaks when you score 500 candidates at once: then a tiny batch window on CPU (or a small GPU) can be the right call, as long as p99 stays in budget.

Common mistakes

• Putting XGBoost on an H100 so every model "lives in Triton on GPU."
• One HPA on CPU util for LLM, diffusion and trees together.
• Ignoring null features. The model looks broken. The store is empty.

Check yourself

1. Why is a GPU usually the wrong host for a GBDT fraud score? Answer: the model is tiny and the SLO is sub-5 ms; PCIe and batching add delay a local CPU library does not.
2. When can embeddings share an LLM GPU? Answer: offline or nightly indexing with preemption, not online query embed beside decode.

Key takeaways

• Classic ML belongs on CPU ONNX or Triton FIL, not on an H100 chat pool.
• Batch rows only when the product already scores many candidates.
• Match scale signals to the model type.
• Share GPUs only for preemptible offline work.

Further learning

NVIDIA Triton Inference ServerThe Triton Inference Server provides an optimized cloud and edge inferencing solution. - triton-inference-server/serverGitHub

Back to the non-chat serving map.