rag// prompt challenge · ~10 min · no setup needed
RAG Relevance Judge
In RAG pipelines, the retriever returns chunks — but not all of them are useful. Write a system prompt that makes an LLM judge whether a document chunk is relevant to a user's query, outputting a structured verdict with reasoning. The LLM will be given the query and three chunks below as its user message.
Test input — your prompt will receive this
QUERY: "How do I handle rate limiting when calling the OpenAI API?" CHUNK 1: Title: OpenAI API Rate Limits — Best Practices Content: "When you hit a 429 Too Many Requests error from the OpenAI API, the recommended approach is exponential backoff with jitter. Start with a 1-second delay, double it on each retry, add random jitter of ±20%, and cap at 60 seconds. Use the Retry-After header if present. For production workloads, use a token bucket or sliding window rate limiter on your side to stay within your tier's TPM and RPM limits before requests even leave your server." CHUNK 2: Title: OpenAI Pricing Page — Updated June 2026 Content: "GPT-4o is priced at $5.00 per 1M input tokens and $15.00 per 1M output tokens. Batch API requests receive a 50% discount. Volume discounts are available for usage above $10,000/month. All prices are in USD and exclude taxes. Prices may change with 30 days notice." CHUNK 3: Title: Building a Document Q&A System with LangChain Content: "To build a document Q&A system, start by chunking your documents into 512-token segments with 50-token overlap. Generate embeddings using text-embedding-3-small and store them in a vector database like Pinecone or pgvector. At query time, embed the user question and retrieve the top-5 chunks by cosine similarity, then pass them to GPT-4o as context."
This is the exact user message that will be sent to the LLM alongside your system prompt.
See what a good response looks like
// example of an ideal LLM output for this test input
CHUNK 1 — RELEVANT The chunk directly addresses the query. It explains how to handle 429 rate limit errors from the OpenAI API, covering exponential backoff with jitter, the Retry-After header, and client-side rate limiting strategies (token bucket / sliding window). CHUNK 2 — NOT RELEVANT The chunk covers OpenAI token pricing and volume discounts. It contains no information about rate limiting errors, retry strategies, or request throttling. CHUNK 3 — NOT RELEVANT The chunk describes how to build a RAG Q&A system using LangChain and vector databases. While it involves OpenAI, the topic is document retrieval architecture, not rate limit handling.
Your system prompt
Free preview — 1 attempt, no account needed. Sign up free for up to 3 attempts and a scored profile.
0/2000
Ready for a full challenge?
Build a real RAG pipeline locally and get a scored artifact on your profile.