LLM Evals and Observability in Production

Traditional monitoring answers whether the service responded. It does not answer whether the answer was any good, and for an AI feature that is the only question that matters. A model returning confident nonsense in 200ms looks perfect on every dashboard you already have.

Build the eval set from real failures

Synthetic eval sets test what you imagined. Production failures test what actually breaks. Every complaint, every thumbs-down and every support escalation is a test case somebody has already written for you.

• capture the full input — retrieved context included, not just the user's message
• record the output and why it was wrong, in the reporter's own words
• keep the set small enough to run on every change, and grow it only for genuinely new failure modes
• freeze it — an eval set that drifts is a benchmark you cannot compare across time

Fifty well-chosen cases beat five thousand generated ones. The value is in coverage of failure modes, not in volume.

Grade the things that have right answers

• did it call the correct tool with the correct arguments — deterministic, cheap, catches most agent regressions
• is the required fact present in the output — a string or structured assertion
• did it refuse when it should have refused — a fixed set of cases that must decline
• is the output well-formed — schema validation, not a model judging JSON

Reserve model-graded evaluation for the genuinely subjective, and when you use it, pin the grader model and its version. A grader that silently upgrades underneath you makes every historical score meaningless.

Trace the whole chain, not the final call

# one user question, traced end to end
request_id 8f31c2
  retrieval      docs=8   ms=120   tokens_in=0
  rerank         docs=3   ms=40
  llm_call_1     tokens_in=3200  tokens_out=180   ms=900
  tool:lookup    ms=210   status=ok
  llm_call_2     tokens_in=3900  tokens_out=240   ms=1100
  total          ms=2370   cost=$0.021

Most incidents filed as "the model got it wrong" are retrieval incidents. Without the retrieval step visible in the trace you will spend a day editing prompts to fix what is actually a chunking problem.

Cost is a production signal

Token spend is the closest thing to a leading indicator this work offers. A prompt change that adds context, a retrieval change that returns more documents, an agent that has started looping — all of them show up as cost per request well before they show up as a complaint.

• track cost per request and per feature, not just the monthly invoice
• alert on distribution shift rather than the mean; the tail is where loops live
• cap tokens and tool calls per request, and log hitting the cap as an error
• record the model version in every trace, so a provider-side change is visible to you

Ship prompt changes the way you ship code

Prompts are code with worse tooling. Version them, review them, and run the eval set before they go out. When a change improves the average but breaks four previously passing cases, that is a regression regardless of what the average says.

The purpose of an eval set is not the score. It is the list of things that used to work, so that you find out when they stop.