Production-Grade RAG Evaluation That Actually Catches Failures

Shipping RAG is easy; keeping it reliable is hard. Evaluation has to cover retrieval, grounding, safety, latency, and cost together. This is the checklist I use when I’m asked to harden a RAG stack for real users.

1) Treat Retrieval As A First-Class Surface

• Coverage (recall@k): Build a labeled set of queries with gold documents. Track recall@k and MRR, not “it feels good.”
• Freshness: Measure how quickly a new doc shows up in the top-k after ingestion. Policy changes must propagate in minutes, not days.
• Hybrid relevance: Run dense + sparse + reranker. Log per-query which mode wins so you can tune weights instead of guessing.
• Drift detection: Alert on embedding similarity drops or reranker score shifts when catalogs/policies change. Keep a drift dashboard.
• Context hygiene: Deduplicate near-identical passages before they reach the model; keep token budgets sane.

2) Groundedness And Safety, Not Just BLEU

• Groundedness: Score whether each cited snippet is actually used in the answer. Track hallucination rate per surface (<3% for customer-facing).
• Safety: Run a fast toxicity + PII classifier on outputs before they reach the user. Reject/repair if tripped.
• Source compliance: Validate that answers only cite allowed sources (latest policy version, correct region). Fail the request if compliance breaks.
• Answer structure: Enforce JSON/markdown schemas with validators; force citations per sentence for regulated domains.

3) Golden Test Suites That Match The Business

• Build a golden intent set: high-value flows, edge cases, seasonality, “gotcha” policy questions.
• For each intent, store: query, expected facts, allowed sources, disallowed sources, target answer style, and unacceptable behaviors.
• Run this suite on every model, retriever, or index change. Fail the build if groundedness or recall regresses beyond a threshold.
• Maintain a red-team slice with adversarial prompts (negations, prompt injection, ambiguous phrasing).

4) Automated Raters Where Humans Don’t Scale

• Use LLM-as-a-judge for groundedness/helpfulness, but pin it to retrieved snippets so it cannot hallucinate a grade.
• Calibrate the judge with a human-rated seed set (50–100 examples). Track correlation and refresh monthly.
• Add rule-based raters for schema correctness: JSON validity, required keys, numeric ranges, allowed enums.
• Score citation density (citations per fact) to catch under-cited answers.

5) Latency, Cost, And Capacity As First-Class Metrics

• Track P50/P95 separately for retrieval, rerank, and generation. Users feel the tail.
• Enforce cost SLOs per request: tokens + vector reads + reranker calls. Reject or shrink context if it exceeds budget.
• Add pre-flight context budgeting: estimate tokens before calling the model; trim to a capped budget with priority ordering.
• Monitor throughput and saturation: vector DB QPS, reranker concurrency, model TPS. Alert before hitting rate limits.

6) Observability Patterns I Always Add

• Log: query, embedding model version, index version, top-k doc IDs, chosen context, model version, cost, latency, safety score.
• Attach a trace ID and store a replayable trace so support can reproduce an issue without re-hitting production APIs.
• Capture why a doc was included (similarity score, reranker score, rule match). It speeds up debugging.
• Build drill-down dashboards: hallucination by intent, recall@k by collection, P95 latency by node.

7) Rollout Strategy That Avoids Surprises

• Shadow new retrievers/generators side-by-side; compare groundedness, recall, latency, and cost on live traffic.
• Canary to a small slice with auto-rollback on metric regression. Keep feature flags for retriever, reranker, and generator versions.
• Require golden-suite pass + shadow parity before canarying. Block deploys if compliance or safety fails.

8) Data And Labeling Workflow

• Build a continuous labeling loop: sample failed traces, have SMEs label groundedness/hallucination, fold back into eval sets.
• For B2B, let customer SMEs tag “allowed/disallowed” sources to enforce compliance in evals.
• Track label freshness: retire stale intents when policies change; add new ones for seasonal launches.

9) Example Reference Stack

• Retrieval: Hybrid (BM25 + dense) with cross-encoder reranker; deterministic filters for jurisdiction/product/locale.
• Indexing: Background jobs + streaming updates; schema versioned; doc lineage for audits.
• Eval: Golden suites + LLM judge + safety classifier + latency/cost dashboards; alerts on drift.
• Ops: Feature flags, replayable traces, canaries, and alerting on grounding, recall, P95 latency, hallucination rate, and cost SLOs.

The goal: a RAG system you can change safely—measurable, debuggable, compliant, and fast enough for production. If you want this wired into your stack, let’s talk.