Quality and safety
Serving fine-tunes
Managed APIs trade control for speed. Open-weight LoRA trades ops for isolation. Plan for base-model upgrades either way.
A finished adapter is not a product. You still have to serve it, version it, and redo the work when the base model is replaced. The hosting choice is managed API vs open-weight LoRA. The maintenance choice is how you survive the next base-model release.
Concept
Before you start: you need a held-out win on quality, or on cost and latency at matched quality. See fine-tuning data. If the bake-off is still "prompt plus routing wins," do not stand up GPUs.
Managed API vs open-weight LoRA
| Managed fine-tuning APIs | Open-weight LoRA | |
|---|---|---|
| Ops | Upload data, train, call a custom model id | GPUs, checkpoints, serving stack |
| Data | Leaves your boundary (check terms) | Can stay in-VPC |
| Iteration | Fast without ML infra | More knobs: rank, targets, schedule |
| Serving | Provider scales it | Multi-LoRA or merged weights |
| Cost shape | Per-training plus per-token on the custom model | GPU time to train and to serve |
Orders of magnitude as of when this was written: managed SFT jobs often land in the tens to hundreds of dollars for modest datasets. Serving a fine-tuned mid-size model can undercut frontier API prices if volume is high. Self-host LoRA training on one high-memory GPU is routine for 7B to 13B. Larger models need multi-GPU or QLoRA.
Always compare total cost: engineering time, eval maintenance, and the day the base model is deprecated.
Tip
Run a bake-off before committing infra: best prompted frontier vs best prompted mid vs fine-tuned small, all on the same eval suite, with dollars per 1,000 requests and p95 latency. Many "we must fine-tune" projects die when a better prompt plus routing wins.
Multi-LoRA and a light sweep
For LoRA SFT, people actually touch rank r (often 8 to 64), which modules get adapters, learning rate, epochs, and max sequence length matching production. Watch validation loss. Data fixes beat exotic schedules.
Multi-LoRA keeps one base in memory and swaps adapters per request or tenant. Merge an adapter into the base when there is a single global tune. See multi-tenant serving.
Versioning and churn
Treat base_model + adapter_id + data_revision + eval_suite_id as the release. Roll back like any bad deploy.
Frontier and open bases improve every few months. Budget to re-distill or re-SFT on the new base. Old adapters do not automatically transfer. If you fine-tune only once a year, a prompted newer base may overtake you for free.
Prefer fine-tunes that encode stable behavior (format, taxonomy) over ones that encode weekly policy text. Policy belongs in RAG.
Even after a successful fine-tune, keep an escape hatch. Hard queries escalate to a larger model. The fine-tune handles the fat middle. The frontier model handles the long tail. See latency and cost.
Combining RAG and fine-tuning
The usual production pattern is both: retrieve evidence, then generate with a model that was SFT'd to cite, abstain and format correctly. Fine-tuning without retrieval still will not know Tuesday's price change. Retrieval without behavioral tuning may still wander off-schema. Evaluate the compound system, retrieval plus weights plus prompt, as one unit.
Common mistakes
- Skipping the bake-off. You spend six weeks on LoRA and discover two better few-shots were within 1 point.
- Treating the adapter as permanent. The next base release orphans it.
- Encoding weekly policy in weights. Next Tuesday's price change will be wrong until you re-train.
Check yourself
- You have no ML infra and a 400-row classification set. Managed API or open-weight LoRA? Managed API. Upload, train, call a custom model id.
- Why re-train when the base model changes? Adapters do not automatically transfer. A prompted new base may already beat last year's tune.
Key takeaways
- Managed APIs trade control for speed. Open-weight LoRA trades ops for isolation.
- Compare total cost, including the day the base is deprecated.
- Version base, adapter, data and eval suite together.
- Combine RAG and fine-tuning: evidence in context, style in weights.
Further learning
Back to the decision ladder. For serving many adapters on one base, see multi-tenant serving.
Related
Fine-tuning vs prompting vs RAG
When to change weights instead of the prompt or the retrieved context, and how to decide with evals.
Fine-tuning data
Quality and coverage of the examples dominate the training run. Match production inputs and freeze a held-out eval first.
Multi-LoRA and multi-model serving
Serving many fine-tuned adapters and models on shared GPUs efficiently.