Prompting for smaller and local models: what's different

By Ivo Donker - 3 August 2026

When an application is migrated from a large-scale hosted language model to a smaller or locally run model, existing prompts often no longer produce the desired result. Prompts that execute complex, composite instructions flawlessly on large models cause smaller parameter systems to produce incorrect formats, missed constraints, or outright faulty reasoning. This difference isn't just about the total number of parameters — it comes down to fundamental differences in instruction-following ability, effective context length, and the robustness of the internal representations.

Successfully deploying smaller models requires a different approach to prompt engineering. Where large models fill in missing details on their own and process complex logic within a single input, smaller models require a tighter architecture in which the application takes over logic from the prompt.

Why large prompts fail on small models

The cause of disappointing performance in smaller models can be traced back to three main factors: reduced instruction-following capacity, a more limited effective context, and more brittle reasoning chains.

Reduced instruction-following ability

Large language models are trained with extensive RLHF and instruction-tuning phases on enormous datasets. This allows them to parse and prioritize complex, layered instructions. Smaller models have less capacity to maintain multiple constraints at once. When a prompt contains four different rules about tone, format, excluded topics, and length, a small model will often ignore one or two of these requirements.

Shorter effectively usable context

Although the theoretical context window of a small model can sometimes be 32k or 128k tokens, the precision of the attention mechanism degrades faster as the input grows. Information located in the middle of a long prompt gets taken into account less reliably. When setting up context management for small models, minimizing input length is therefore more critical than for larger variants.

More brittle reasoning chains

Complex logic in which step A directly affects step B leads to deviations more quickly in small models. As soon as the model makes an incorrect assumption in one of the intermediate steps, that error gets locked into the rest of the generated text. The margin of error compounds, making the final conclusion unreliable.

Concretely adapting the instruction structure

To get a small model to perform reliably, the prompt must be structured with minimal ambiguity. This requires rethinking how instructions are presented to the system.

The key adjustments in practice are:

• One task per call: Don't combine summarization, sentiment analysis, and translation in a single prompt. Split them into three separate requests. With a strategy of a model per task you retain maximum control over the quality of each step.
• Splitting instructions with clear delimiters: Use explicit markdown headers or XML tags to physically separate the instruction, the source text, and the expected output format in the input.
• Showing examples instead of describing rules: A small model learns faster from a concrete input-output pair than from an abstract description of the desired result.
• Avoiding negations: Sentences like "Don't use jargon" or "Don't answer in English" often backfire. Smaller models process the negating word poorly and end up focusing attention on the very term being negated. Phrase the rule positively instead: "Use simple language" or "Answer only in Dutch".

The weight of examples (few-shotting)

Compared to large models, small models rely heavily on the patterns present in the input. A general explanation of few-shot prompting shows that examples steer the output, but with small models they determine almost the entire structure of the answer.

The optimal number of examples for small models is usually between two and four. Too few examples (a single example) provide insufficient grip on the pattern, while more than four examples take up valuable context space without a proportional increase in accuracy.

The selection of examples is essential. Choose examples that have exactly the same structure as the expected final result. If the output format must follow a specific key-value structure, each example should contain exactly the same keys in the same order. Deviations in the examples lead directly to inconsistencies in the generated output for small models.

Guaranteeing structured output without relying on the promise

A common mistake is relying on a small model's promise that it will produce JSON or XML based on a textual instruction. Small models regularly forget closing brackets, add introductory text ("Here is the JSON:"), or change field names partway through generation.

To reliably obtain structured data from smaller models, additional measures are needed. See the guide on enforcing output formats for a broader overview, combined with the following specific steps:

1. Repeating the schema right before generation: Place the expected JSON schema not at the beginning of the prompt, but right at the end, just before the point where the model needs to start writing.
2. Applying constrained decoding or grammars: Use techniques such as GBNF (GGML BNF) or JSON schema constraints in the runtime environment. These force the inference process, at the token level, to select only tokens that comply with the grammar. This prevents syntax errors entirely.
3. Validation and recovery in the application layer: Build a fallback mechanism into your code. When the parsed output fails a JSON schema check, have a quick validation routine surface the error. Then send a short repair prompt to the model containing only the faulty output and the validation error.

Chain-of-thought: benefits and risks with small models

Having a model reason explicitly ("think step by step") can improve the quality of the final answer. With small models, however, this technique carries specific risks.

If the reasoning chain becomes too long, the model can drift away from the original question. This is caused by hallucinating intermediate steps. The model then follows its own generated noise instead of the instruction. For small models, chain-of-thought primarily helps with short, clear arithmetic or categorization problems, but it often works against you in open-ended text generation.

A robust alternative to a long internal reasoning chain is to split the reasoning step across the application. Have the model extract only the facts from the text in call 1. Have the model make a decision based on those extracted facts in call 2. This prevents a single faulty intermediate step from contaminating the entire generation.

The effect of quantization on prompt stability

In practice, local models are almost universally run in a quantized form (such as INT4, Q4_K_M, or IQ3_XS) to reduce memory footprint. A technical explanation of this can be found on the page about quantization explained.

Quantization directly affects prompt sensitivity. When weight precision is reduced from 16-bit to 4-bit, the model loses some of its ability to distinguish subtle nuances in instructions. This has the following practical consequences:

• A prompt that works flawlessly on the unmodified FP16 version of a model can suddenly start ignoring formatting rules on a Q4 version.
• Sensitivity to word order in the prompt increases.
• The stability of output formatting decreases at lower bit depths.

It's therefore essential to always test and optimize prompts on the exact quantization version and inference engine used in the final production environment. Testing on a non-quantized API gives a false picture of performance on a local system.

Measuring instead of assuming: your own evaluation set

Public benchmarks give a general picture of a language model's capabilities, but they say little about how a specific model responds to your specific prompts and company-specific data.

Building your own evaluation set is the only way to determine whether a prompt change actually delivers an improvement. Within the self-evaluation framework you put together a collection of at least 50 to 100 representative input examples, including the desired output.

Evaluate the results against hard criteria: does the model follow the JSON structure, are the required fields present, and does the answer length stay within the set limits? By automating this evaluation on every change to the prompt or model, you prevent changes that look like an improvement from causing regressions elsewhere in the application.

Further reading

• Few-shot prompting in practice
• Enforcing and validating output formats
• Architecture with a model per task
• Quantization of local models explained
• Setting up your own framework for evaluating models