Direct answer
Production agents need traces plus offline evals that can block merges. Pick tools by job: eval/CI specialists, LangChain-native trace suites, or OpenTelemetry self-host stacks. Hybrid is normal. See AI Agent Evaluation & Observability.
Minimum operating model
- Define failure modes: wrong tool, bad handoff, policy breach, cost spike
- Instrument traces across agents, tools, guardrails, and handoffs
- Curate a golden dataset from real failures
- Run offline scorers on every PR that touches prompts or tools
- Sample online production traces with budgeted scorers
- Cap cost per run and per tenant
Tooling map (2026)
- Eval/CI gates: Braintrust-class workflows that post results on PRs
- LangChain-native: LangSmith for deep graph/agent traces
- OTel / self-host: Phoenix or Langfuse
- SDK-native: OpenAI Agents SDK spans, optionally mirrored to OTel
Rule: if quality cannot fail a merge, you do not have evals. You have reports.
