When working with Large Language Models (LLMs) as an individual developer or researcher, the temptation to adjust prompts ad-hoc is strong. A word here, an instruction there, until the output looks good in the playground. However, once AI applications form the core of production applications, this "cowboy approach" is no longer sustainable. Prompts must be handled, reviewed, and approved in a structured way within a team.
In this article, we discuss why you should review prompts with the same discipline you use to review software code (such as Python or JavaScript). We cover setting up test sets, separating subjective taste from measurable quality, and we provide a concrete template that you can integrate into your pull requests today.
A prompt is not just a piece of text; it is declared logic. Where traditional code defines the how, a prompt defines the what. A seemingly innocent adjustment in a prompt can have far-reaching consequences for the reliability and security of the application. That is why it is essential to work with proper prompt version control.
Within a team, the following problems often arise if prompts are not structurally reviewed:
A fundamental rule when reviewing prompts is: you cannot judge a change without evidence. Because the output of an LLM is non-deterministic (even at a temperature of 0, absolute consistency is not guaranteed over time), a reviewer cannot simply look at the prompt and say with 100% certainty: "This will work."
For every significant prompt change, the developer must run the prompt against a gold standard: the test set. A good test set for prompts contains:
During a review, the author must not only provide the new prompt, but also a diff (difference) of the generated output compared to the previous prompt version, based on the agreed test set. You can read more about structured testing of LLM outputs on our sister platform in the guide on LLM quality assurance and validation.
One of the biggest pitfalls during a prompt review is the discussion about taste. Phrases like "I don't think this answer sounds friendly enough" or "Maybe it should be a bit shorter" are disastrous for progress. They lead to endless iterations without a clear end goal.
To review effectively in a team, subjective taste must be converted into measurable rubric criteria. When assessing common prompt errors, we apply objective standards.
| Subjective Feedback (Incorrect) | Measurable Feedback (Correct) |
|---|---|
| "The tone is too boring." | "The output lacks the prescribed brand identity: use a maximum of 2 sentences per paragraph and address the reader as 'you'." |
| "The answer is too long." | "The output exceeds the strict limit of 150 words specified in the system instruction." |
| "It feels like the model is hallucinating." | "The model claims facts in paragraph 3 that are not in the provided RAG (Retrieval-Augmented Generation) context." |
To further curb discussions about subjectivity, mature AI teams often deploy another LLM (often a heavier, slower model like GPT-4 or Claude 3 Opus) to score the output of the modified prompt. This principle, where an independent prompt evaluates the quality of another prompt on a scale of 1 to 5 based on established criteria, provides a quantitative basis for the human reviewer.
A prompt never lives in a vacuum; it is inextricably linked to the underlying model. When OpenAI, Google, or Anthropic updates a model (for example, from version 0613 to 1106), the semantics of your prompt can suddenly be interpreted differently. What previously yielded a perfect JSON output might now result in JSON with unwanted markdown wrappers.
A good team practice is to always pin prompt versions to a specific model version. When a model upgrade is required, this is treated as a major breaking change. The existing prompts must go through the entire test set, and the review process focuses specifically on "model drift": the subtle ways in which the new model version deviates from the old one. You can find more in-depth information on this phenomenon in our overview article on the impact of model updates on production environments.
If everyone is responsible, no one is responsible. Within a development team, there must be clarity about the ownership of prompts. In practice, we often see the following division of roles:
During a pull request (PR) for a core prompt, there must be at least an approval from the person managing the application logic, and—in the case of content changes—the domain expert.
A prompt without documentation is legacy code from the moment it is committed. To properly review and maintain a prompt, contextual information is essential. This documentation ideally belongs in a central prompt library or at least in a docstring above the code where the prompt is called.
A complete prompt documentation always contains:
To ensure the review process runs smoothly in practice, we recommend setting up a standard Pull Request template in platforms like GitHub, GitLab, or Azure DevOps. Below you will find a lightweight template specifically designed for prompt changes.
pull_request_template.md or make it a specific issue template within your repository.
Building robust AI applications requires us to move away from the idea that prompts are magical pieces of text that we iteratively adjust in a web interface. By treating prompts as a fundamental part of your codebase—including version control, objective test sets, clear ownership, and formal peer reviews—you increase the stability of your application and reduce stress within the development team. Implementing a review template as described above is an excellent first step toward a mature AI development process.