Share:𝕏LinkedInRedditFacebookCopy link

Testing Prompts Before They Go Live

By Ivo Donker — compiled with AI assistance (Claude & Gemini)

Adjusting a prompt often feels surprisingly simple: you add a line, tweak some wording, and in the API provider's playground the response immediately looks better. Yet this is one of the biggest pitfalls when developing AI applications. A prompt that works perfectly for that one test case can silently disrupt the processing of hundreds of other user inputs.

Software engineering has had strict procedures for testing code before it goes to production for decades. Exactly the same necessity applies to prompt engineering. Anyone who changes instructions for a Large Language Model (LLM) is effectively modifying the application's logic. In this article, we cover a structured approach to validating, measuring, and gradually rolling out prompt changes.

Why the playground isn't enough

Manually trying out a prompt in an interactive playground gives a false sense of security. The main reasons why this isn't sufficient for production applications are:

To prevent unexpected prompt errors in production, a systematic approach is essential. Before you get to testing, it's also crucial that you keep track of changes through strict prompt version control.

Building a representative test set

The foundation of a reliable testing process is a high-quality test set (also called an evaluation dataset or golden dataset). This set must be a realistic mirror of what the system will face in practice.

An effective test set for prompts consists of three categories of cases:

  1. Standard cases (Happy Path): Common, representative questions or tasks that cover the vast majority of daily volume.
  2. Edge cases: Input with extreme lengths, missing information, unusual punctuation, spelling errors, or multilingual fragments.
  3. Historical regression cases: Specific examples of input that have led to incorrect output, hallucinations, or wrong formats in the past.

Practical tip: Continuously expand your test set based on anonymized production logs. As soon as a user reports an error or rates a response as 'bad', add that specific input to the test set.

What do you record for each test case?

In traditional software testing, you often compare the output against an exact expected value. With LLMs, that is rarely possible or desirable, because the model can phrase the same concept in dozens of different ways. Instead of a fixed answer, you record the following for each test case:

Hard checks versus soft judgments

Evaluating a prompt's output requires a combination of automated, deterministic checks and more qualitative evaluations.

Type of evaluation Method Examples of criteria
Hard checks (Automated) Code-based validation (Python/TypeScript) Valid JSON structure, presence of required fields, length restrictions, absence of forbidden words.
Soft judgments (Qualitative) Model-as-a-Judge or human sampling Degree of relevance, politeness of tone, correctness of the summary compared with the source text.

Hard automated checks

Hard checks are executed with standard program logic. They are cheap, fast, and run without any doubt. If a prompt must produce a response in a specific format, it helps to apply techniques for enforcing output formats. The test then simply checks whether the output passes through a parser like JSON.parse() or a Pydantic model.

// Example of an automated hard check on prompt output
function valideerOutput(rawOutput) {
  try {
    const parsed = JSON.parse(rawOutput);
    
    // Check for required fields
    if (!parsed.samenvatting || !parsed.categorie) {
      return { success: false, reason: "Required fields missing" };
    }
    
    // Check for length restriction
    if (parsed.samenvatting.length > 500) {
      return { success: false, reason: "Summary exceeds maximum length" };
    }

    return { success: true };
  } catch (e) {
    return { success: false, reason: "Invalid JSON structure" };
  }
}

Soft qualitative judgments

For things like 'clarity' or 'correctness', rule-based checks are not sufficient. Here you often use a stronger model as an evaluator (LLM-as-a-Judge) with a very specific evaluation prompt, or have the team review samples. This process runs more smoothly when the team has made clear agreements about reviewing prompts in the team.

Dealing with variance: running multiple runs

Because an LLM is non-deterministic, a single test round on your test set gives an incomplete picture. A prompt can coincidentally score 100% on the test set, but fail on 15% of cases on a second attempt due to slightly different word choices.

Therefore, always run critical test cases multiple times (for example, 3 to 5 times per input). Then calculate the pass rate per test case. A test case only passes when it meets all hard checks in at least 80% or 90% of runs. This clearly reveals the so-called 'flakiness' of your prompt.

A/B testing: putting the new variant next to the old one

Never judge a new prompt on its own. The crucial question is not only: "Is the new prompt good?", but above all: "Is the new prompt better than the version that is currently live, without causing regressions?"

Run the full test set through both the current production prompt (Variant A) and the proposed prompt (Variant B). Compare the results directly side by side at the same level of input. Within our network, you can use the interactive prompt A/B testing tool at https://benchmark.llmnet.nl/ to score two variants side by side in a structured way and map performance differences.

Agreeing on an acceptance threshold

Before you analyze the test results, the team must have established a clear acceptance threshold. This prevents discussions about whether or not to go live from being driven by gut feelings.

A realistic acceptance framework includes, for example, the following rules:

Phased rollout and the rollback plan

Even the most comprehensive test set never covers 100% of reality. When a prompt change meets the acceptance threshold, immediately rolling out to 100% of users is still a risk.

Therefore, apply a phased rollout (canary release):

  1. First route 5% to 10% of live traffic to the new prompt version.
  2. During this phase, actively monitor for error messages, increased latency, or a spike in the number of retries.
  3. If the picture is stable, gradually scale up to 25%, 50%, and finally 100%.

Essential: Make sure your application code is prepared for a quick 'rollback'. Because a prompt must have version control separate from the application code, you need to be able to switch back to the previous prompt version within seconds via a configuration change, without needing a new code deploy.

Checklist for reviewing a prompt change

Use this compact checklist before a prompt change gets the status 'production ready':