Writing Evals for LLMs: A Practical Guide

Most AI engineers evaluate their systems the same way: they try a few inputs they care about, see outputs that look reasonable, and move on. This is not evaluation. It's hoping.

The gap between "it looks fine when I test it" and "it works reliably across the full distribution of real inputs" is where AI systems go wrong. Closing that gap is what evals are for — and it's one of the most underdeveloped skills in the field right now.

Why evals are the real differentiator

You can build an impressive-looking AI system without any formal evaluation. You can demo it, share it on GitHub, and get stars. What you cannot do is trust it.

Companies that ship AI features to production need engineers who can answer: "How do we know this is working?" Not with intuition, but with evidence. That evidence comes from a well-constructed eval pipeline. Engineers who can build one are rare and get hired quickly. Engineers who can't are building systems that will embarrass them in production.

The evaluation spectrum

Evals exist on a spectrum from cheapest-and-weakest to most-expensive-and-strongest:

• Vibe checks — manual inspection of a handful of outputs. Necessary but not sufficient. Tells you almost nothing about generalisation.
• Exact match / unit tests — for tasks with deterministic correct answers (structured extraction, classification, code generation). Fast, cheap, reliable.
• Reference-based metrics — BLEU, ROUGE, BERTScore for generation tasks. Faster than human eval, but often poorly correlated with actual quality.
• LLM-as-judge — use a separate model (GPT-4o, Claude Opus) to evaluate outputs against criteria. Scalable and often well-calibrated, but introduces its own failure modes.
• Human evaluation — the ground truth. Expensive and slow, but necessary to calibrate everything else.

A production eval pipeline uses multiple layers of this spectrum: cheap automated checks run on every commit, LLM-as-judge runs on a sampled subset, human eval runs periodically to validate that the automated metrics are still trustworthy.

Building your test set

The quality of your evals is bounded by the quality of your test set. A weak test set makes everything else irrelevant.

A good test set has three properties:

1. Representative. It covers the actual distribution of inputs your system will see in production — not just the easy cases you thought of first. If 30% of real queries are ambiguous or multi-hop, 30% of your test set should be too.
2. Verified. Every ground-truth answer in your test set was verified by a human who knows the correct answer. Synthetic ground truth is a starting point, not a finish line.
3. Growing. Every time your system fails on a real input, that input goes into the test set permanently. Your eval suite should accumulate failures over time, not stay static.

Start with 50–100 examples. That's enough to get meaningful signal early. Grow it from there as you discover failure modes.

Planted failures: the most underused technique

A planted failure is a test case that your system should fail on — and if it doesn't fail, something is wrong.

For a RAG system, a planted failure might be a question whose answer is not in the corpus. The correct behaviour is to say "I don't know" or "this information isn't available." If your system answers confidently, it's hallucinating. That's a bug.

For a classification system, a planted failure might be a deliberately ambiguous or out-of-distribution input. If your classifier assigns it to a category with high confidence, your confidence calibration is broken.

Planted failures tell you whether your system knows what it doesn't know. This is one of the most important properties a production AI system can have, and almost no one tests for it explicitly.

LLM-as-judge: how to do it right

Using a language model to evaluate language model outputs is powerful but fragile. The common failure modes:

• Position bias. LLMs tend to prefer whichever answer comes first when comparing two options. Always randomise order or evaluate each independently.
• Verbosity bias. Longer answers often get higher scores, even when concise answers are better. Specify in your evaluation prompt that length is not a quality signal.
• Prompt sensitivity. Small changes to your evaluation prompt produce large changes in scores. Lock your eval prompt once you've calibrated it against human labels.
• Self-preferencing. GPT-4o tends to rate GPT-4o outputs higher; Claude tends to rate Claude outputs higher. Use a different model family than your target for evaluation where possible.

The right way to use LLM-as-judge: define specific rubric dimensions (faithfulness, relevance, completeness), score each independently on a 1–5 scale with explicit criteria for each score, and validate periodically against human labels to confirm the judge is still calibrated.

Metrics that actually matter

For a RAG pipeline, track:

• Retrieval recall@k — what fraction of relevant documents appear in the top-k retrieved chunks?
• Answer faithfulness — what fraction of claims in the generated answer are grounded in the retrieved context?
• Answer correctness — does the answer match the ground truth (for questions with factual answers)?
• No-answer rate on unanswerable queries — when the answer isn't in the corpus, does the system say so?

For an agent, track: task completion rate, tool call accuracy, and error recovery rate (when a tool call fails, does the agent recover correctly?)

The specific metrics matter less than picking them deliberately, defining them precisely, and tracking them consistently over time. A metric you measure consistently is more valuable than a sophisticated metric you measure once.

The regression testing mindset

Every time you make a change to your system — new prompt, different model, changed retrieval strategy — you run your full eval suite and compare results to the previous version. Any regression (a metric getting worse) is a bug, even if your vibe check says the new outputs look better.

This is where most teams fall down. They improve one thing, eyeball the outputs, feel good, and ship. Two weeks later a use case that used to work quietly breaks. With a regression suite, you catch this before it ships.

Build the habit early: eval before every significant change, keep scores in version control, and treat regressions as blockers.

---

TryCrucible has an Evals & Testing challenge category where you build and run eval pipelines against real datasets, including planted failures. Your eval design is scored as part of the submission. See evals challenges →