Skip to content
Applied AISeptember 24, 2026 · 5 min read

Evaluating an LLM feature when you have no labelled data

A practical evaluation path for teams shipping a first AI feature into a workflow that has never been measured.

Start from the queue, not the corpus

There is almost always a record of the problem: support tickets, review notes, escalations, a spreadsheet someone keeps. Pull 300 real examples and sort them by frequency and cost. That set is your evaluation set, and it will be more honest than anything you can invent.

Define three failure types, not a single score

Grade each case for retrieval failure (wrong or missing source), reasoning failure (right source, wrong conclusion) and presentation failure (correct content, unusable answer).

The three failures have different fixes — chunking, prompting, interface — and collapsing them into one accuracy number hides which one you are actually improving.

Grade in pairs, and record the disagreements

Two graders per case, independently. Where they disagree, the disagreement is a policy gap, not noise. Resolve it in writing once and the next thousand cases become cheap.

Track cost and latency alongside quality

A 4-point quality gain that doubles latency per session may be a worse product. Log retrieval count, tokens, time to first token and completion cost per request from the first day, so trade-offs are visible before they are argued about.

Freeze the set, then extend it

Keep a frozen evaluation slice to compare releases fairly, and a living slice drawn from production traffic that grows weekly. The frozen slice answers whether a change helped; the living one answers whether the product still works on this month's questions.