Start from the queue, not the corpus
There is almost always a record of the problem: support tickets, review notes, escalations, a spreadsheet someone keeps. Pull 300 real examples and sort them by frequency and cost. That set is your evaluation set, and it will be more honest than anything you can invent.
Define three failure types, not a single score
Grade each case for retrieval failure (wrong or missing source), reasoning failure (right source, wrong conclusion) and presentation failure (correct content, unusable answer).
The three failures have different fixes — chunking, prompting, interface — and collapsing them into one accuracy number hides which one you are actually improving.
Grade in pairs, and record the disagreements
Two graders per case, independently. Where they disagree, the disagreement is a policy gap, not noise. Resolve it in writing once and the next thousand cases become cheap.
Track cost and latency alongside quality
A 4-point quality gain that doubles latency per session may be a worse product. Log retrieval count, tokens, time to first token and completion cost per request from the first day, so trade-offs are visible before they are argued about.
Freeze the set, then extend it
Keep a frozen evaluation slice to compare releases fairly, and a living slice drawn from production traffic that grows weekly. The frozen slice answers whether a change helped; the living one answers whether the product still works on this month's questions.
