Apr. 13, 2026
22 minutes read
Share this article
Last Updated April 2026
LLM apps fail in ways ordinary software tests do not catch. A request can return a 200 status code, stay within latency budgets, and still produce a fabricated answer, an unsafe recommendation, or a response that quietly misses the user’s goal. Teams building these systems need a testing model that treats language quality, safety, and business usefulness as first-class release criteria, especially when delivery depends on a broader machine learning and AI operating model rather than a single prompt.
That is why evaluation has to be designed into the product from the start, not added after launch. In practice, the strongest teams pair LLM-specific checks with a broader custom software delivery approach so quality gates, release workflows, and rollback decisions are defined before the application reaches users. Once the product moves beyond a prototype, the operational side also matters, and many organizations discover they need the same discipline associated with LLMOps for AI operations management to keep testing, monitoring, and versioning aligned.
LLM evaluation is the structured measurement of whether an application meets defined success criteria. Those criteria can include answer correctness, policy compliance, relevance, factual grounding, tone, latency, cost, and task completion.
A practical evaluation system has three parts:
Some scores are expressed on a 0 to 1 scale, but the number alone is never the whole story. A score can summarize performance, yet release decisions still require human judgment about whether the score measures the right thing and whether the application is failing in places the average hides.
Traditional software tests assume deterministic behavior. The same input should produce the same output every time unless something breaks. LLM apps do not work that way. Small prompt changes, retrieval differences, model updates, or tool-call timing can alter the answer even when nothing appears broken at the infrastructure layer.
This creates five recurring problems:
For that reason, a useful evaluation focuses on the application as a system, not only on the base model.
Users return to an AI product only if it behaves consistently enough to feel dependable. That does not mean every answer must be identical. It means the answer should remain within an acceptable quality band across normal usage, edge cases, and repeated prompts.
Hallucinations matter most when users treat the answer as actionable. In support, operations, finance, healthcare, legal workflows, or internal knowledge systems, a plausible but false answer can create rework, compliance exposure, or reputational harm.
Bias, toxic language, privacy leakage, and policy violations are not edge concerns. They belong in the release process. Teams dealing with customer data, regulated workflows, or sensitive requests often need evaluation criteria that align with privacy-by-design requirements in generative AI applications, not only generic model quality scores.
A strong evaluation program answers product questions, not just research questions. Did the assistant deflect more tickets correctly? Did the knowledge tool reduce handle time? Did the coding assistant increase completion speed without increasing security risk? Metrics only matter when they help decide whether the app is ready, improving, or regressing.
The fastest way to build a weak evaluation suite is to start with generic metrics. A customer support assistant, a contract summarizer, a text-to-SQL tool, and an agent that calls external systems do not fail in the same way.
Before choosing metrics, define:
A text-to-SQL assistant may need execution accuracy, schema adherence, and permission compliance. A RAG assistant may need retrieval relevance, faithfulness to context, citation formatting, and refusal behavior when evidence is weak. An agent may need tool selection accuracy, parameter correctness, and successful multi-step completion.
The cleanest evaluation programs use two modes, not one.
Offline evaluation uses pre-production data such as curated examples, golden datasets, historical tickets, synthetic cases, or human-annotated samples. It is the right choice for:
Offline testing is useful because teams can rerun the same cases after every change. That makes it easier to compare versions and catch regressions.
Online evaluation uses live traffic, production traces, user feedback, and real interactions after release. It is the right choice for:
Offline evaluation tells teams whether a release looks ready. Online evaluation tells them what actually happens after real users arrive.
Offline evaluation without online monitoring misses production behavior. Online evaluation without offline regression testing makes it hard to improve safely. The two should share as much logic as possible so the same evaluator can score both staged and live interactions.
Many evaluation failures start with the dataset, not the scoring method. If the examples do not represent real traffic, the results will not predict real behavior.
A practical dataset should include:
Golden datasets are especially useful. These are high-quality, reviewed examples that establish a benchmark for recurring tasks. They do not need to cover every possible input. They need to cover the cases that matter most for release confidence.
Synthetic examples can help fill gaps, but they should not dominate the suite. When the test set becomes too polished or too predictable, the application may appear stronger than it really is.
Most teams benefit from separating metrics into layers.
These measures whether the answer is good from the user’s point of view.
Reference-based metrics such as BLEU, ROUGE, and METEOR can still help with constrained generation tasks, especially when a close target output is available. They are much less useful for open-ended answers where several responses may be equally good.
These measures whether the system stays inside defined boundaries.
These checks often deserve the same status as security testing. Teams already thinking through AI security risk reviews usually find it easier to integrate them into release governance.
These measures whether the application is usable at scale.
Operational metrics do not tell you whether the answer was good, but they do tell you whether the product is sustainable.
These matter when the application includes multiple model calls or external actions.
RAG systems need evaluation at two layers:
Useful RAG checks include:
A system can retrieve the wrong context and still produce a fluent answer, or retrieve the right context and still summarize it poorly. That is why the two layers should be measured separately.
| Metric | What it measures | Layer | How it’s typically scored |
|---|---|---|---|
| Relevance | Whether the answer addresses the user’s actual question | Quality | LLM-as-a-judge or human rubric |
| Correctness | Whether the answer is factually accurate | Quality | Reference comparison or human review |
| Completeness | Whether the answer covers all required elements | Quality | Rubric-based or checklist scoring |
| Faithfulness | Whether the answer stays grounded in retrieved context | RAG | LLM-as-a-judge against source documents |
| Context recall | Whether retrieval surfaces all relevant documents | RAG | Reference-based comparison |
| Retrieval relevance | Whether retrieved chunks match the query intent | RAG | Embedding similarity or LLM scoring |
| Abstention quality | Whether the system refuses appropriately when evidence is weak | RAG / Safety | Rule-based or LLM-as-a-judge |
| Toxicity | Whether output contains harmful, offensive, or unsafe language | Safety | Classifier or LLM-as-a-judge |
| Bias and fairness | Whether outputs treat groups consistently | Safety | Paired testing or human audit |
| Privacy leakage | Whether outputs expose sensitive or restricted data | Safety | Rule-based pattern matching |
| Prompt injection success | Whether adversarial inputs manipulate system behavior | Safety | Adversarial test set |
| Refusal quality | Whether the system refuses the right requests correctly | Safety | Golden set comparison |
| Task completion rate | Whether multi-step workflows reach a successful end state | Workflow | End-to-end test suite |
| Tool selection accuracy | Whether the agent selects the right tool for the task | Workflow | Reference comparison |
| Tool parameter correctness | Whether tool arguments are valid and complete | Workflow | Schema validation or rule-based check |
| Latency | Response time from request to completion | Operational | Infrastructure instrumentation |
| Token usage | Tokens consumed per request | Operational | API logging |
| Cost per request | Compute and API cost per interaction | Operational | Usage-based billing logs |
| Error rate | Frequency of failed or malformed responses | Operational | Compute the API cost per interaction |
No single evaluator is enough for every task. The strongest setups combine three categories.
These are deterministic and useful when the output must obey clear constraints. Examples include:
These are often the first line of defense because they are cheap, fast, and reproducible.
LLM-as-a-judge is useful when quality depends on open-ended interpretation, such as relevance, helpfulness, faithfulness, clarity, or policy adherence. It scales better than a full manual review, especially for conversational and generative use cases.
Still, it is not a magic answer. A judge prompt, a judge model, and a scoring rubric can all introduce noise. That is why automated judgments should be calibrated against human review, especially before they become release gates.
Human review remains necessary for:
Human review does not need to score every sample forever. It needs to establish whether automation is measuring the right thing.
| Evaluator type | Best for | Time and cost | What it misses |
|---|---|---|---|
| Rule-based checks | Schema validity, format compliance, keyword presence, SQL syntax, required field verification | Very fast, near-zero cost, fully reproducible | Cannot assess open-ended quality, relevance, or nuanced policy adherence |
| LLM-as-a-judge | Relevance, helpfulness, faithfulness, tone, policy adherence, open-ended quality at scale | Moderate cost, scalable, consistent when rubric is well-designed | Can inherit model biases, requires calibration, not reliable without a clear rubric |
| Human evaluation | Subjective quality, brand-sensitive content, high-risk outputs, rubric design, calibrating automated scoring | High cost, not scalable for full production volume | Cannot cover full traffic at scale; reviewer fatigue affects consistency |
| Reference-based metrics (BLEU, ROUGE, METEOR) | Constrained generation tasks with a close target output, translation, structured summarization | Fast and cheap | Weak for open-ended answers where multiple responses are equally valid |
| Embedding similarity | Semantic closeness between output and reference, retrieval relevance checks | Fast, scalable | Constrained generation tasks with a close target output, translation, and structured summarization |
The practical combination for most production teams: Rule-based checks as the first line of defense for format and policy violations. LLM-as-a-judge for quality and relevance at scale. Human review to calibrate the judge and cover high-risk outputs. Reference-based metrics only where a close target output exists.
The evaluation tooling ecosystem has matured quickly. Here is how the leading options break down by function.
Promptfoo is one of the most widely adopted open-source tools for offline LLM evaluation. It allows teams to define test cases in YAML or JSON, run them against multiple models or prompt variants simultaneously, and compare results side by side. It supports LLM-as-a-judge scoring, custom rubrics, and CI integration — making it practical for regression testing before every release.
Braintrust provides a hosted evaluation platform with experiment tracking, dataset management, and LLM-as-a-judge scoring. It is designed for teams that prefer a managed environment over a self-hosted setup, and it integrates with common LLM providers and orchestration frameworks.
Weights & Biases (W&B) extended its experiment-tracking capabilities to include LLM evaluation through its Weave product. Teams already using W&B for MLOps can track prompt versions, evaluation runs, and quality scores alongside model training experiments — which is especially useful when a workflow combines fine-tuning and prompt generation.
Ragas is purpose-built for evaluating retrieval-augmented generation pipelines. It measures faithfulness, answer relevance, context precision, and context recall separately — addressing both the retrieval and generation layers of a RAG system. It integrates with LangChain and LlamaIndex and can run automated scoring using an LLM judge.
TruLens provides evaluation and tracing for LLM applications, including RAG pipelines. It instruments the full application chain — retrieval, prompting, and generation — and scores each step, making it easier to determine whether a quality problem originates in the retriever or the model.
LangSmith (from LangChain) provides tracing, dataset management, prompt versioning, and evaluation for LLM applications. It captures the full chain of inputs, retrieved context, tool calls, and outputs — making it useful for both offline testing and production monitoring. Its annotation queues support human review workflows alongside automated scoring.
Langfuse is an open-source alternative to LangSmith that provides tracing, scoring, and dataset management. It works across LLM providers and orchestration frameworks and is a strong choice for teams that need observability without vendor lock-in.
Arize AI provides production monitoring for LLM systems — logging inputs, outputs, latency, token costs, and safety events at scale. It supports drift detection, segment-level quality analysis, and integration with offline evaluation pipelines, which makes it practical for teams that want to connect pre-release testing with post-release monitoring.
Helicone provides lightweight production observability focused on cost, latency, and usage analytics. It is faster to set up than Arize and works well for teams that need visibility into token costs and basic quality logging before building a more comprehensive evaluation stack.
Guardrails AI and NVIDIA NeMo Guardrails provide runtime policy enforcement — blocking disallowed outputs, enforcing topic boundaries, validating output schemas, and routing edge cases to human review. These tools complement evaluation by acting as real-time filters rather than post hoc measurements, and they are especially important in customer-facing or regulated deployments where policy violations carry direct business consequences.
Abstract evaluation frameworks are easier to apply when they are grounded in a specific system. Here is what a practical evaluation setup looks like for one of the most common enterprise LLM use cases: a RAG-based assistant that answers customer support questions by retrieving from a product knowledge base.
Before choosing any metric, the team defines what good and bad look like:
The dataset combines four source types:
| Faithfulness to the retrieved context, answer relevance, completeness, and tone adherence | Metrics used |
|---|---|
| Retrieval | Retrieval relevance, context recall |
| Generation | Faithfulness to retrieved context, answer relevance, completeness, tone adherence |
| Safety | Prompt injection resistance, refusal quality for out-of-scope questions |
| Operational | Latency per request, token cost per query |
| Metric | Minimum threshold to release |
|---|---|
| Faithfulness | ≥ 0.85 |
| Answer relevance | ≥ 0.80 |
| Retrieval relevance | ≥ 0.78 |
| Prompt injection resistance | 100% pass on adversarial set |
| Latency (p95) | ≤ 2.5 seconds |
| Human approval on golden set | ≥ 90% |
After release, LangSmith traces every conversation. Arize monitors for quality drift segmented by query type, customer tier, and knowledge base version. Any session in which faithfulness falls below 0.75 or a refusal is triggered is flagged for human review within 24 hours. Token cost per query is tracked weekly against a budget threshold, and retrieval relevance is reviewed whenever the knowledge base is updated.
This setup is not the only valid approach. The right metrics, thresholds, and tools depend on the use case, the risk profile, and the team’s existing infrastructure. What matters is that the choices are explicit, documented, and tied to the business definition of success rather than to generic benchmark performance.
The most effective teams use eval-driven development. Instead of changing prompts or swapping models and then asking whether the result seems better, they define the scorecard first and optimize against it.
A practical workflow looks like this:
This is especially important in agentic systems, where a single prompt change can alter tool use, latency, and downstream correctness all at once. Teams working with agent guardrails such as permissions, tool scopes, and audit trails often discover that evaluation becomes easier once the allowed action boundaries are explicit.
After launch, the goal shifts from pre-release validation to continuous verification. Production monitoring should combine traces, quality scores, and business context.
Track at least these categories:
The point is not to create an enormous dashboard. The point is to make quality degradations explainable. A drop in answer quality should be traceable to a concrete cause, such as a retriever change, a prompt revision, a model upgrade, or a document ingestion issue.
In regulated environments, teams often map these controls to familiar governance structures, such as NIST terminology, but the operational value comes from making release decisions auditable, not from the label alone.
Before a production release, teams should be able to answer yes to the following:
A mature evaluation program does not mean the app never fails. It means the team can detect, explain, and reduce failure with discipline.
At a minimum, maturity includes:
As systems become more capable, the same logic applies. The structure just expands from single responses to full workflows. Whether the application answers a question, summarizes a document, retrieves evidence, or executes multi-step actions, the central rule stays the same: measure what the user and the business actually need, then make those measurements part of every release decision.
LLM evaluation is the structured process of assessing whether a large language model application meets defined success criteria before and after release. Unlike traditional software testing, LLM evaluation cannot rely on deterministic output matching — the same input can produce different outputs, and fluent language can mask factual errors or policy violations. A practical evaluation system defines what success means for the specific use case, tests against a representative dataset, scores outputs across quality, safety, and operational dimensions, and connects those scores to release and monitoring decisions.
Testing an LLM application requires combining multiple evaluation methods rather than relying on a single approach. Rule-based checks validate format, schema, and policy compliance quickly and cheaply. LLM-as-a-judge scoring assesses open-ended quality dimensions like relevance, helpfulness, and faithfulness at scale. Human review calibrates automated scoring and covers high-risk or brand-sensitive outputs. Offline evaluation against curated datasets catches regressions before release. Online monitoring of production traces detects quality drift and unexpected failures after launch. The strongest setups use all of these in combination, with shared logic between the pre-release and production evaluation layers.
LLM-as-a-judge is an evaluation technique that uses a language model to score the outputs of another language model against a defined rubric. Instead of matching output to a reference answer word for word, a judge model assesses dimensions like relevance, faithfulness, completeness, or tone using structured scoring criteria. It scales better than full human review and works well for open-ended generation tasks where multiple answers could be equally valid. The key requirements are a well-designed rubric, a capable judge model, and calibration against human review before the judge is used as a release gate.
RAG systems need evaluation at two separate layers: retrieval quality and generation quality. On the retrieval side, the key metrics are retrieval relevance (whether retrieved chunks match the query intent), context recall (whether all relevant documents were surfaced), and context precision (whether retrieved chunks are focused rather than noisy). On the generation side, the key metrics are faithfulness (whether the answer is grounded in the retrieved context rather than fabricated), answer relevance, and abstention quality (whether the system refuses appropriately when the evidence is insufficient). Tools like Ragas and TruLens handle both layers and integrate with common RAG frameworks.
The right metrics depend on the use case, but most production LLM applications need metrics across at least four layers. Quality metrics — relevance, correctness, faithfulness, completeness — measure whether the answer is good from the user’s perspective. Safety metrics — toxicity, bias, privacy leakage, prompt injection resistance, refusal quality — measure whether the system stays within defined boundaries. Operational metrics — latency, token usage, cost per request, error rate — measure whether the product is sustainable at scale. Workflow metrics — task completion rate, tool selection accuracy, step success rate — apply when the application includes multi-step reasoning or tool use. Starting with a small set of release-critical metrics and expanding deliberately produces more reliable results than tracking everything at once.
Offline evaluation runs against pre-production data — curated examples, golden datasets, historical tickets, or synthetic cases — before a release. It is the right approach for prompt comparison, regression testing, CI gates, and controlled experiments. Online evaluation monitors live production traffic after release, using real interactions, user feedback, and production traces. It is the right approach for detecting quality drift, surfacing unexpected edge cases, and understanding how the application performs under real user behavior. Both are necessary: offline evaluation without production monitoring misses real-world failures, and production monitoring without regression testing makes it difficult to improve safely.
Evaluating LLM apps is not about chasing a single perfect score. It is about building a repeatable system that defines quality, tests it before release, watches it after launch, and connects technical signals to user and business outcomes. Teams that do this well treat evaluation as part of product engineering, not as a last-mile QA task.
When evaluation covers offline testing, online monitoring, safety checks, workflow analysis, and human calibration, LLM apps become easier to improve and safer to operate. That is the standard required for production performance, and it is the difference between an AI feature that demos well and one that holds up under real use.
Leandro is a Subject Matter Expert in Backend at Coderio, where he focuses on modern backend architectures, AI-assisted modernization, and scalable enterprise systems. He contributes technical thought leadership on topics such as legacy system transformation and sustainable software evolution, helping organizations improve performance, maintainability, and long-term scalability.
Leandro is a Subject Matter Expert in Backend at Coderio, where he focuses on modern backend architectures, AI-assisted modernization, and scalable enterprise systems. He contributes technical thought leadership on topics such as legacy system transformation and sustainable software evolution, helping organizations improve performance, maintainability, and long-term scalability.
Accelerate your software development with our on-demand nearshore engineering teams.
