How to Eval Your AI Agent: Goldens and LLM-as-Judge

Problems:

• agent hallucinates or gives bad results
• agent doesn't have enough info to give the best answer
• you don't know which model you should use
• you need to know which mcp servers you should add
• should the quality threshold be lower or higher
• you don't know which temperature, top_k, top_p, presence_penalty, repetition_penalty, tool_choice to set

EVALUATION AGAINST GOLDEN DATASET

Golden Dataset - is manually collected dataset of problem-solution pairs with some context on resolution (+ optional metadata).

Why is it needed?

• This is the best way to benchmark/evaluate quality of your agent on real data.

How to use it?

• We can run agent on this data and compare actual<>golden resolution.

Do's

• Only real data
• Subject Matter Expert verified
• Covers ~90%+ of scenarios
• Quality > Quantiny
• Happy-Path cases 60%; Edge cases 20%; Ask for clarification 10%; Adversarial inputs 5%;
• Keep ~5-10% of golden samples hiden and never optimise for them - it's your test cases.

Don'ts

• Never use golden examples in system prompt or few-shot examples
• Ensure Agent does NOT have access to tools to get golden answers (logs mcp, slack mcp, etc) - it will just read the answer

How Golden should look like? Basically, it should have all the data you need to find why specific case fails

• User Input
• Golden Output
• Context (tools called, message history)
• Metadata (difficulty of the test case, date, link to input, etc)

How Evaluation Benchmark should look like?

• Explicitly defined metrics (e.g. resolution_score 0-1, sources_score 0-1, etc) and criteria for them

Before Evaluation

• Run few same cases multiple times - capture data variance

CI/CD integration

• define fail threshold in your CI/CD to capture drift
• git track and update your data