Evaluating multi-agent systems

A fluent final brief can hide a bad plan, a dropped citation or two workers that disagreed. Evaluate the product output first, then slice the failure. "Did each agent call tools" is not a quality metric.

The big idea

Score these, in this order:

1. End-to-end quality — a rubric or an LLM judge, calibrated on human grades, for the final artifact.
2. Coverage — did the plan assign the subquestions that matter?
3. Handoff fidelity — do worker summaries preserve citations and caveats? Inject a known fact in a worker transcript and test whether it survives.
4. Efficiency — tokens, wall time, duplicate fetches.
5. Coordination incidents — conflicting claims in the final answer.

Score contradiction rate: the final answer asserts A while a worker finding said not-A. That metric catches merge bugs that "sounds fluent" judges miss.

Trace at the boundaries: the plan, each worker's trajectory, the merge input and the merge output. Without that, you cannot tell bad research from bad summarization.

Walking through a landmine

Take the checkout teardown. Put "Adyen requires company-id on step 2" only in worker W2's raw tool result. The worker's handoff must keep it. The orchestrator's comparison table must keep it.

If the final brief omits it, do not retune the crawler first. Check the handoff schema and the merge prompt.

Anthropic's write-up of their research product is the best public guide. Themes that transfer:

• The orchestrator plans and delegates. Sub-agents explore in parallel with isolated contexts.
• Compression of results back to the orchestrator is a first-class design problem.
• Evaluation is end-to-end quality of the final research brief, not tool-call counts.
• Token cost is an explicit product constraint.
• Simple patterns beat elaborate graph frameworks that have no owner of state.

If your own system cannot yet cite sources reliably with one agent, more agents will not fix that. They will hide the failure inside a prettier merge. Fix retrieval and single-agent faithfulness first. Then use workers to buy parallelism.

Common mistakes

• Scoring only the final prose. You cannot see a dropped caveat.
• No boundary traces. You then guess whether research or merge failed.
• Adding agents to fix citations. Faithfulness is a single-agent problem first.

Check yourself

• A worker found a caveat and the final brief dropped it. Which two scores should move? Handoff fidelity and, if the final claim now contradicts the worker, contradiction rate.
• Why is "did each agent call tools" a weak metric? A busy, well-traced run can still produce a wrong brief.

Key takeaways

• Evaluate the final artifact, then slice coverage, handoff, efficiency and contradictions.
• Inject known facts and test that they survive the merge.
• Trace the plan, each worker and the merge. Pair that with ordinary evals and LLMOps.
• Do not scale to many agents until one agent is faithful.

Further learning

How we built our multi-agent research system (Anthropic)On the the engineering challenges and lessons learned from building Claude's Research systemanthropic.com

Back to the map: multi-agent systems.