Managing the context window: what fits, what doesn't, and what to do about it

By Ivo Donker — compiled with AI assistance (Claude & Gemini)

Modern language models boast context windows ranging from hundreds of thousands to even millions of tokens. On paper, the line between memory and storage seems to have vanished: dump entire codebases, transcripts, and PDF files into a single prompt and let the model do the work. In a production environment, however, this approach immediately runs into three hard limits: exponentially rising latency, soaring API bills, and a measurable loss of reasoning capability due to attention degradation. Managing a context window effectively is therefore not about filling it to capacity, but about strict budgeting.

In practice, a context window acts as the working memory (RAM) of a language model. Anything outside the window does not exist for the model during that specific inference round; everything inside it competes for the attention of the transformer's attention layers. In this article, we analyze how to structure tokens, where the real limits lie, and which architectural patterns are essential to keep applications fast, reliable, and cost-effective.

The anatomy of the context window: token distribution

A context window is not a homogeneous space where you can randomly dump text. Each inference call consists of a composite payload that must fit within the maximum allowed token limit (the context limit) as well as the maximum output length (max completion tokens). To build reliable applications, we divide the context into four distinct segments:

When the reserved output buffer is not explicitly budgeted for, a common production error occurs: the model stops mid-JSON object because the total sum of input and output hits the model's hard ceiling. Also check out the core principles of fundamental context window management to see how memory models are theoretically constructed.

Attention loss and the 'Lost in the Middle' phenomenon

Just because a model technically accepts 128,000 tokens does not mean it processes all 128,000 tokens with equal attention. Repeated empirical research shows that transformer models exhibit a strong recency bias and primacy bias . Information placed at the very beginning of the prompt (the system prompt) and information at the absolute end (the most recent user input) is retrieved with high precision. However, crucial data placed in the middle of a large context payload regularly fades in the attention maps.

This phenomenon, commonly known as Lost in the Middle, is amplified as the complexity of the reasoning task increases. A simple factual retrieval task (needle-in-a-haystack) often scores 99% in synthetic tests, but as soon as the model must synthesize two inconsistent facts from the middle into a logical conclusion, accuracy drops drastically. Blindly stretching the context thus introduces silent errors that are difficult to reproduce.

Furthermore, the computational complexity of attention mechanisms scales quadratically ($O(N^2)$), or in optimized variants linearly to sub-quadratically, relative to context length $N$. Even with FlashAttention and modern KV-cache optimizations, Time to First Token (TTFT) increases significantly once prompts exceed tens of thousands of tokens. For those looking to calculate the financial impact of large payloads, the overview on the true cost of long context windows provides a clear calculation framework.

Classic management strategies: pruning versus sliding

To prevent a multi-turn dialogue or a data-intensive task from uncontrollably bloating the window, we use several programmatic strategies. The two most fundamental methods are hard truncation and the sliding window.

1. Hard truncation (FIFO pruning)

With First-In, First-Out (FIFO) pruning, the application simply removes the oldest turns from the conversation history once a threshold is reached. The system prompt is always preserved at position zero, but turns 1 and 2 disappear when turn 8 arrives. The advantage is simplicity and predictability; the disadvantage is abrupt memory loss: an agreement or entity mentioned in the first minute of the conversation is immediately unreachable.

2. Token-based sliding window with weights

Instead of deleting entire messages, a more advanced sliding window measures exact token counts and maintains a minimal set of anchors. An implementation of this might, for example, reserve 4,000 tokens for history, giving system instructions priority 1, the last two turns priority 2, and truncating intermediate turns based on age.

def assemble_context(system_prompt: str, history: list[dict], 
                     rag_chunks: list[str], max_tokens: int = 8000) -> list[dict]:
  reserved_output = 1000
  budget = max_tokens - reserved_output - count_tokens(system_prompt)
  
  # 1. Reserveer ruimte voor relevante RAG context (maximaal 50% van overgebleven budget)
  rag_budget = int(budget * 0.5)
  selected_chunks = []
  rag_tokens = 0
  for chunk in rag_chunks:
    t = count_tokens(chunk)
    if rag_tokens + t <= rag_budget:
      selected_chunks.append(chunk)
      rag_tokens += t
      
  # 2. Vul resterend budget met meest recente gesprekshistorie
  history_budget = budget - rag_tokens
  selected_history = []
  for message in reversed(history):
    t = count_tokens(message["content"])
    if history_budget - t >= 0:
      selected_history.insert(0, message)
      history_budget -= t
    else:
      break
      
  # 3. Bouw uiteindelijke payload
  context_payload = [{"role": "system", "content": system_prompt}]
  if selected_chunks:
    context_payload.append({
      "role": "system", 
      "content": "Relevante context:\n" + "\n---\n".join(selected_chunks)
    })
  context_payload.extend(selected_history)
  return context_payload

Hierarchical compression and semantic summarization

When a dialogue needs to be sustained over a long period without losing crucial details, a sliding window falls short. This is where hierarchical compression provides a solution. Instead of bluntly discarding older turns, they are asynchronously summarized by a lighter, faster model as soon as a specific token threshold is exceeded.

The process follows three fixed steps:

1. Segmentation: As soon as the conversation history reaches 2,000 tokens, the oldest 1,500 tokens are isolated.
2. Structured extraction: A background task extracts explicit entities, decisions made, user preferences, and open questions into a compact JSON or bullet-point format.
3. Context injection: The summary is placed directly below the system prompt as a continuous "memory block," after which the original 1,500 tokens are deleted from the active window.

Prefix Caching: Optimizing speed and cost structure

Since the introduction of automatic and explicit prefix caching (also known as prompt caching) by leading model providers, the physical layout of the context window is directly linked to API costs. Prefix caching means that the computed KV cache of identical text fragments at the beginning of the prompt is reused at the server level for subsequent requests.

To take full advantage of this technique, the order of elements in the context window must be strictly deterministic from static to dynamic:

[1. Statische Systeemprompt & Tools]   -> 100% Cachable (Verandert nooit)
[2. Vaste Few-Shot Voorbeelden]        -> 100% Cachable (Verandert zelden)
[3. Geheugensamenvatting / Klantprofiel]-> Deels Cachable (Wijzigt per sessie)
[4. RAG-documenten / Dynamische Data]   -> Incidenteel Cachable
[5. Huidige Gebruikersinput (Turn N)]   -> Nooit Cachable (Volledig dynamisch)

If you place dynamic variables, such as the current timestamp or arbitrarily ordered RAG documents, at the top of the system prompt, you break the cache for all downstream tokens. By consistently placing static definitions first, token costs for large prompts drop by 50% to 80% and latency is reduced to a fraction of an uncached call.

Context management within agentic workflows and tool loops

Managing the context window becomes exponentially more complex as soon as we transition from static prompts to agents that invoke tools autonomously. Anyone looking to understand how agent architectures evolve compared to static chats can consult the explanation on how autonomous AI agents actually work . In an agentic loop, the model repeatedly executes actions, receives tool outputs, and reasons about the next step.

Without active context management, an agentic loop grinds to a halt within a few iterations. A database query that returns 200 rows of raw JSON can consume 15,000 tokens in a single blow. If the loop runs for ten iterations, the context window explodes, leading to hallucinations or crashes. For a deeper analysis of iterative loop errors and compounding tool outputs, the article on debugging agentic loops provides practical guidance.

Within agent systems, we therefore apply three specific filtering techniques:

• Tool Output Masking: Clean up raw API and SQL responses prior to injecting them into the context window. Remove unused JSON fields, metadata, and redundant headers.
• Intermediate Scratchpad Pruning: As soon as an agent completes an intermediate step (such as validating a file), the detailed intermediate step can be replaced with a single-line status confirmation (Status: Validatie succesvol). The full log does not need to remain in the window.
• Sub-agent Isolation: Delegate data-intensive search tasks to a sub-agent with its own isolated context window. This sub-agent reads through 50 pages and returns only a concise 200-token answer to the primary agent.

Measuring and evaluating: quantifying context quality

Context management should not be a matter of gut feeling. Every decision to add, truncate, or summarize context has measurable consequences for both response precision and operational costs. To systematically evaluate which context length yields the highest accuracy at an acceptable latency, the guide on A/B testing for prompts and contexts is a logical starting point.

In a robust evaluation pipeline, we measure four core metrics across different context configurations:

Practical matrix: which strategy fits which scenario?

There is no universal context configuration that works optimally for every type of application. The chosen architecture depends on the interaction pattern, data dynamics, and the required state retention period:

Production Architecture Guidelines

Effectively designing an LLM application requires treating the context window as a precious, scarce resource. While large context windows offer fantastic flexibility for occasional complex analyses, relying on them as the foundation for continuous production workflows will inevitably lead to slow, expensive, and error-prone systems without strict management.

The core rules for managed context windows can be summarized into four principles: structure payloads deterministically to maximize caching, protect attention focus by aggressively filtering out noise, segment complex tasks across isolated sub-contexts, and continuously measure the relationship between context size, accuracy, and latency. By applying this engineering discipline, the application will maintain stable performance regardless of the volume of data flowing through it.