Guide

LLM fundamentals

Why benchmarks lie

Leaderboards answer "is this model generally strong?" They do not answer whether it extracts your invoice fields at 99% with your schema.

5 of 155 · 2 min read

Leaderboards like LMArena (human preference battles) and aggregators like Artificial Analysis (quality, speed, price) are excellent for orientation. They answer "is this model generally strong?" They do not answer "will it extract our invoice fields at 99% with our schema?"

Treat leaderboards as a shortlist generator. The decision document cites your eval numbers, p50/p95 latency and projected monthly token spend.

Concept

Before you start: you need a task-specific eval and the worksheet. This page is why you should not stop at the trophy slide.

Why public suites fail products

  • Your distribution is weird: internal jargon, PDF layouts, tool schemas.
  • Arena Elo rewards style and sycophancy as much as correctness.
  • Contaminated or saturated academic suites stop discriminating.
  • Latency and cost at your prompt lengths never appear on the trophy slide.

A beauty contest on generic chat, then shipping the winner into a structured extraction pipeline, is the usual failure. Chat preference and schema adherence are weakly correlated.

Walking through a mismatch

LMArena says Model F beats Model M on human preference. Your invoice suite says Model M hits 99% exact field match and Model F hits 94% because it adds prose around the JSON. Preference Elo picked the chattier model. Your parser picked M.

Arena Elo:     F  >  M     (style, sycophancy, "helpfulness")
Your suite:    M  99% field match
               F  94% field match   (prose around JSON)
Ship:          M

Use Arena and Artificial Analysis to notice that F and M are both worth scheduling an eval run. Then run the suite. That is the whole relationship.

Common mistakes

  • Pasting a leaderboard screenshot into a design doc as the decision.
  • Optimizing for Elo when the product contract is a schema.
  • Never measuring tool parse success. A generally "strong" model that emits invalid JSON at 5% will dominate on-call.

Check yourself

  1. What do you cite in the decision document: Arena Elo, or your eval plus latency and spend? Your eval, p50/p95 latency and projected monthly token spend. Elo is a shortlist.
  2. Why can a higher-Elo model lose on invoice extraction? Arena rewards style. Your parser rewards valid fields. Those are weakly correlated.

Key takeaways

  • Leaderboards shortlist. Your eval decides.
  • Arena Elo is not schema adherence.
  • Your jargon, PDFs and tool schemas are not on the trophy slide.
  • Measure tool parse success as a first-class number.

Further learning

Artificial AnalysisComparison and analysis of AI models and API hosting providers. Independent benchmarks across key performance metrics including quality, price, output speed & latency.artificialanalysis.ai LMArena leaderboardlmarena.ai

Back to the map in the LLM landscape.

The LLM landscape and model choice

Frontier vs open-weight models, reasoning models, and a process for picking one with your own evals.

Choosing a model

Prove the product on a strong model, freeze an eval, then step down the ladder. Your suite decides, not a leaderboard.

Evaluating LLM apps

A fixed set of inputs plus a way to score outputs, so you can tell whether a change helped.