LLM Evals: A Practical Framework for Testing AI Systems That Actually Work

If you're building AI systems and not writing evals, you're flying blind. You'll know your system mostly works because you ran it a few times and the output looked right. What you won't know is which inputs break it, whether a prompt change made things better or worse, or what happens when a real user sends something you didn't anticipate.

Evals — short for evaluations — are the test suite for LLM-based systems. This guide covers what they are, how to write them, which scoring methods to use for which situations, and how to build a regression harness you can actually run.

What LLM evals are (and aren't)

An eval is a test case that assesses whether your AI system produces the right output for a given input. Unlike traditional unit tests, the "right output" is often not a single exact string — it's a property of the response (is it faithful to the source? does it correctly classify the input? does it follow the format?). This is what makes LLM evaluation hard, and why most teams skip it.

The three things evals are not:

• Not vibes-checking. Running your system manually on 5 inputs and deciding it looks okay is not an eval. It's a demo. Demos don't catch regressions.
• Not identical to benchmarks. Public LLM benchmarks (MMLU, HellaSwag, etc.) measure general model capability. Your evals measure whether your specific system — with your prompts, your retrieval, your tools — does what you need it to do for your users.
• Not optional at production scale. Every change to a prompt, model version, retrieval strategy, or tool definition is a potential regression. Without evals, you have no way to know if a change helped or hurt.

The eval dataset: what to put in it

Start with 20–50 test cases. More is better, but 20 well-designed cases will tell you more than 200 cases that all probe the same thing.

A good eval dataset covers:

Happy path cases. The standard inputs your system is designed to handle. These should pass consistently, and if they don't, you have a fundamental problem before you even think about edge cases.

Edge cases. The inputs that are at the boundary of what your system handles: ambiguous questions, inputs that require multi-step reasoning, inputs that look like they should match but shouldn't. Most production failures come from edge cases that weren't anticipated.

Adversarial cases. Inputs designed to make your system fail: prompt injection attempts, out-of-domain questions, intentionally misleading context. For any system with real users, adversarial cases must be in the eval set.

Regression cases. Every time you catch a bug in production, add it to the eval set. The goal is that once a bug is fixed, it never reappears without being detected.

For a RAG system, each test case should include:

• The input query
• The expected answer (or expected properties of the answer)
• The documents that should be retrieved (for measuring retrieval recall)
• Whether the answer should be "I don't know" (for testing faithfulness — the system should not hallucinate)

Scoring methods: which one to use when

The choice of scoring method depends on what property you're measuring and whether you can define "correct" precisely.

Exact match. Compare output to expected output directly. Works when output is a category label, a structured JSON object, a boolean, or any other deterministic value. Fastest and cheapest; use it whenever you can. If your system should output {"sentiment": "positive"} and it outputs {"sentiment": "neutral"}, exact match tells you that clearly.

def exact_match(expected: str, actual: str) -> bool:
    return expected.strip().lower() == actual.strip().lower()

Contains / regex match. Check whether the output contains a required element. Useful when the output format is flexible but must include specific content. "Does the response mention the correct policy number?" is a contains check.

Semantic similarity. Embed both the expected and actual outputs, compute cosine similarity. Useful when two phrasings mean the same thing and exact match would incorrectly mark them as wrong. Threshold typically set at 0.85+.

from openai import OpenAI
import numpy as np

client = OpenAI()

def cosine_similarity(a: list[float], b: list[float]) -> float:
    a, b = np.array(a), np.array(b)
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

def semantic_match(expected: str, actual: str, threshold: float = 0.85) -> bool:
    embeddings = client.embeddings.create(
        model="text-embedding-3-small",
        input=[expected, actual],
    )
    sim = cosine_similarity(
        embeddings.data[0].embedding,
        embeddings.data[1].embedding,
    )
    return sim >= threshold

LLM-as-judge. Use a language model to evaluate the output against a rubric. The most powerful method for evaluating open-ended responses where neither exact match nor semantic similarity captures the relevant quality. Expensive but necessary for evaluating faithfulness, coherence, and instruction-following.

def llm_judge(question: str, context: str, answer: str) -> dict:
    prompt = f"""You are evaluating an AI system's answer for faithfulness.

Question: {question}

Context provided to the system:
{context}

System answer:
{answer}

Evaluate:
1. Is the answer faithful to the context? (Does it only claim things supported by the context?)
2. Is the answer complete? (Does it address the question fully?)
3. Is the answer hallucinating? (Does it state facts not in the context?)

Respond in JSON: {{"faithful": true/false, "complete": true/false, "hallucinating": true/false, "reason": "..."}}"""

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"},
    )
    return json.loads(response.choices[0].message.content)

When to use each method:

• Classification tasks → exact match
• Extraction tasks → contains / regex
• Paraphrase-heavy generation → semantic similarity
• Open-ended Q&A, faithfulness, instruction-following → LLM-as-judge
• Production RAG systems → all four, for different dimensions

Building the regression harness

An eval that you run once is useful. An eval harness you run on every change is transformative.

import json
from pathlib import Path
from dataclasses import dataclass

@dataclass
class EvalResult:
    case_id: str
    passed: bool
    score: float
    details: dict

def run_eval_suite(eval_cases: list[dict], system_fn) -> list[EvalResult]:
    results = []
    for case in eval_cases:
        actual_output = system_fn(case["input"])
        passed, score, details = evaluate_output(
            expected=case["expected"],
            actual=actual_output,
            scoring_method=case.get("scoring", "exact"),
        )
        results.append(EvalResult(
            case_id=case["id"],
            passed=passed,
            score=score,
            details=details,
        ))
    return results

def print_summary(results: list[EvalResult]) -> None:
    total = len(results)
    passed = sum(1 for r in results if r.passed)
    avg_score = sum(r.score for r in results) / total
    print(f"Passed: {passed}/{total} ({passed/total:.0%})")
    print(f"Average score: {avg_score:.2f}")
    for r in results:
        if not r.passed:
            print(f"  FAIL [{r.case_id}]: {r.details.get('reason', 'no reason')}")

Run this before and after any prompt change. Track the results over time. If pass rate drops from 94% to 86% after a change, you know immediately — rather than finding out when a user reports a problem.

The three most valuable evals to start with

If you're building evals for the first time, start here:

1. Retrieval recall@k. For any RAG system, measure how often the relevant document is in the top-k retrieved results. If the answer isn't in the context, the model cannot produce a correct answer, no matter how good it is. This is the most important eval for any retrieval system, and it's entirely measurable without an LLM judge.

2. Faithfulness. Does the system's answer claim things that aren't in the retrieved context? Use LLM-as-judge with a clear rubric. Hallucination is the primary trust problem in production RAG systems. Measure it explicitly.

3. Instruction-following. Does the output match the required format? If your system prompt specifies "respond in JSON", does it? If it says "always cite your sources", does it? These are exact or regex checks — cheap to run, and they catch a whole class of regressions that would otherwise slip through.

What good eval coverage looks like

A system with strong eval coverage has:

• 50+ test cases covering happy path, edge cases, and regression cases
• Automated scoring for at least 80% of cases (the rest are sampled for human review)
• A harness that runs in under 5 minutes on the full suite
• A passing threshold defined before changes are made, not after
• Results tracked over time so trends are visible

This isn't complicated infrastructure. It's a JSON file of test cases, a script that runs them, and a results table. The engineering effort is writing the test cases carefully and committing to running them consistently.

The engineers who build AI systems that work reliably in production all have this habit. The ones who are constantly firefighting a system that behaves unpredictably don't.

---

TryCrucible's Evals challenge tests your ability to write a real eval harness — test cases, scoring functions, and a pass/fail report — against a hidden test set. Completing it produces a scored artifact on your public profile that proves you can do this in practice. See the Evals challenge →