# Skeleton-of-Thought: parallel reasoning through prompts

[Skip to content](#lm-inhoud)Network/[NL](/en/skeleton-of-thought-parallel-redeneren-via-prompts)EN[Hubhub.llmnet.nlCompare models on task, language, cost and license.](https://hub.llmnet.nl/en/)[Communitycommunity.llmnet.nlPrompt techniques, patterns and system prompts.](https://community.llmnet.nl/en/)[APIapi.llmnet.nlLLMs in production: rate limits, routing, structured output.](https://api.llmnet.nl/en/)[Consultancyconsultancy.llmnet.nlRolling out AI in an organization, pilot to production.](https://consultancy.llmnet.nl/en/)[Newsnieuws.llmnet.nlAI developments, explained for the Netherlands.](https://nieuws.llmnet.nl/en/)[Benchmarkbenchmark.llmnet.nlMeasure AI quality yourself, on your own tasks.](https://benchmark.llmnet.nl/en/)[Careersvacatures.llmnet.nlAI roles, salaries and career paths in the Netherlands.](https://vacatures.llmnet.nl/en/)[Learnleren.llmnet.nlAI concepts in plain language, beginner to builder.](https://leren.llmnet.nl/en/)[Guidegids.llmnet.nlRun AI privately on your own Mac, PC, NAS or home server.](https://gids.llmnet.nl/en/)[Directorydirectory.llmnet.nlMapping the AI ecosystem: tools, models, companies.](https://directory.llmnet.nl/en/)[Radarradar.llmnet.nlSignals from X, research and communities for indie developers.](https://radar.llmnet.nl/en/)[Appsapps.llmnet.nlReviews of AI apps and open-source repos, with tips for builders.](https://apps.llmnet.nl/en/)[llmnet.nl — main site](https://llmnet.nl/en/)[](https://x.com/intent/post?url=https%3A%2F%2Fcommunity.llmnet.nl%2Fen%2Fskeleton-of-thought-parallel-redeneren-via-prompts&text=Skeleton-of-Thought%3A%20parallel%20reasoning%20through%20prompts)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcommunity.llmnet.nl%2Fen%2Fskeleton-of-thought-parallel-redeneren-via-prompts)[](https://www.reddit.com/submit?url=https%3A%2F%2Fcommunity.llmnet.nl%2Fen%2Fskeleton-of-thought-parallel-redeneren-via-prompts&title=Skeleton-of-Thought%3A%20parallel%20reasoning%20through%20prompts)[](#)[](https://x.com/intent/post?url=https%3A%2F%2Fcommunity.llmnet.nl%2Fen%2Fskeleton-of-thought-parallel-redeneren-via-prompts&text=Skeleton-of-Thought%3A%20parallel%20reasoning%20through%20prompts)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcommunity.llmnet.nl%2Fen%2Fskeleton-of-thought-parallel-redeneren-via-prompts)[](https://www.reddit.com/submit?url=https%3A%2F%2Fcommunity.llmnet.nl%2Fen%2Fskeleton-of-thought-parallel-redeneren-via-prompts&title=Skeleton-of-Thought%3A%20parallel%20reasoning%20through%20prompts)[](#)

 
# Skeleton-of-Thought: parallel reasoning through prompts

 By Ivo Donker — compiled with AI assistance (Claude & Gemini)

 The generation speed of large language models is fundamentally limited by the autoregressive decoding process: every new token requires a forward computation over all preceding tokens. When an application needs to generate an extensive analysis, a technical report, or a multi-step plan, this linear process inevitably results in noticeable delay for the end user. Skeleton-of-Thought (SoT) breaks this linear chain by splitting the generation process into two phases: first the model sketches a concise answer skeleton, after which an orchestrator has the individual points expanded in parallel through simultaneous API calls.

 Where sequential methods cause latency to grow as the desired output gets longer, with Skeleton-of-Thought the wait time scales mainly with the length of the longest individual sub-point. This produces significant speed gains for structured prompts, but at the same time introduces specific challenges around coherence, token consumption, and redundancy. In this article we cover the mathematical basis, the exact prompt templates, a complete implementation, and the architectural trade-offs of this technique.

 
## The mechanics of autoregressive latency versus parallelism

 Modern transformer models generate text token by token. For an answer of $N$ tokens, the total inference time is roughly $T_{totaal} = T_{prefill} + N \times T_{decode}$, where $T_{prefill}$ is the time to process the input prompt and $T_{decode}$ is the time per generated token. Because $T_{decode}$ is dominated by memory bandwidth on the GPU, the wait time remains linearly proportional to the length of the answer. Anyone generating 1200 tokens at a speed of 40 tokens per second has to wait 30 seconds for completion.

 Skeleton-of-Thought fundamentally changes this dynamic by exploiting the inherent independence of sub-sections in many informative answers. Instead of invoking one long generation, the system splits the task into two separate process steps:

 
 
- Skeleton generation (Skeleton Stage): The model receives the main instruction and generates only a numbered list of key points or sub-topics, usually limited to 30 to 80 tokens.
 
- Point expansion (Expansion Stage): For each point in the skeleton, a separate, parallel API call is started. Each call receives the original user question, the full skeleton, and the specific point to be expanded as context.
 

 This reduces the total latency to $T_{totaal} = T_{skelet} + \max(T_{punt_1}, T_{punt_2}, \dots, T_{punt_k}) + T_{samenvoeging}$. Instead of the sum of all expansions, the application only pays the latency of the longest individual sub-point, plus the negligible time for merging the text blocks.

 
## The difference with sequential reasoning chains

 To understand when Skeleton-of-Thought should be deployed, a comparison with traditional prompt strategies is essential. Consult [the decision guide for prompt techniques](https://community.llmnet.nl/en/prompttechniek-kiezen) to determine whether a task benefits from zero-shot, few-shot, or multi-step reasoning. Where classic techniques such as Chain-of-Thought deliberately place reasoning steps one after another, Skeleton-of-Thought is based on breadth exploration.

 In a strictly sequential reasoning chain, step $B$ builds directly on the outcome of step $A$. If, for example, we're working through mathematical derivations or logical deductions, it's not possible to compute step 3 before step 2 is complete. See also the deeper theoretical background on how a model arrives at a conclusion step by step in the article on [reasoning via Chain-of-Thought](https://leren.llmnet.nl/en/chain-of-thought). Skeleton-of-Thought, by contrast, is designed for tasks with an orthogonal information structure: topics where the sub-sections can be developed in parallel without directly needing each other's intermediate results.

 
 
 
 
 Property | 
 Standard Zero-Shot / CoT | 
 Prompt Chaining | 
 Skeleton-of-Thought (SoT) | 
 

 
 
 
 Execution model | 
 Single sequential | 
 Multiple sequential (pipeline) | 
 Hybrid: 1 sequential + $K$ parallel | 
 

 
 Latency scaling | 
 Linear ($O(\sum L_i)$) | 
 Linear ($O(\sum L_i)$) | 
 Sub-linear ($O(L_{skelet} + \max L_i)$) | 
 

 
 Dependency between steps | 
 High (full autoregressive context) | 
 Very high (output $n$ is input $n+1$) | 
 Low (points are conceptually orthogonal) | 
 

 
 API calls | 
 1 | 
 $K$ sequential | 
 $1 + K$ concurrent | 
 

 
 Token cost | 
 Lowest (no context duplication) | 
 Medium | 
 Higher (original prompt repeated $K$ times) | 
 

 
 
 

 
## Prompt design: The skeleton phase and the expansion phase

 The effectiveness of Skeleton-of-Thought depends entirely on the precision of the prompts. If the skeleton prompt is phrased too broadly, the sub-points end up overlapping. If the expansion prompt contains too little context, the model loses the overarching storyline.

 
### 1. The skeleton prompt (Skeleton Prompt)

 This prompt forces the model to produce a compact, numbered list without introductory pleasantries or deeper explanations. The shorter the skeleton, the faster the first phase completes.

 [SYSTEEM]
Je bent een gespecialiseerde planner. Jouw enige taak is om een beknopt antwoordskelet
te ontwerpen voor de vraag van de gebruiker.

Regels:
1. Geef uitsluitend een genummerde lijst van 3 tot maximaal 6 kernpunten.
2. Elk punt mag maximaal 3 tot 7 woorden bevatten.
3. Geen inleiding, geen conclusie, geen toelichting per punt.
4. Zorg dat de punten elkaar niet inhoudelijk overlappen.

[GEBRUIKER]
Vraag: Welke architecturale maatregelen zijn noodzakelijk om een RAG-systeem te beveiligen tegen data-lekken?

[ASSISTENT]
1. Strikte document-level autorisatie bij retrieval
2. Encryptie van vector-indices en embeddings
3. Input-sanitisatie tegen indirect prompt injection
4. Context-filtering vóór generatie
5. Output-validatie en datalek-monitoring

 
### 2. The expansion prompt (Point-Expansion Prompt)

 The orchestration system then fires $K$ requests at the model API simultaneously. Each worker receives the full context, but is explicitly instructed to work out only its assigned sub-point.

 [SYSTEEM]
Je bent een technische schrijver die meewerkt aan een parallel gegenereerd rapport.
Je werkt aan één specifiek onderdeel van een vooraf vastgesteld skelet.

Context van het totale document:
Oorspronkelijke vraag: Welke architecturale maatregelen zijn noodzakelijk om een RAG-systeem te beveiligen tegen data-lekken?
Volledig skelet:
1. Strikte document-level autorisatie bij retrieval
2. Encryptie van vector-indices en embeddings
3. Input-sanitisatie tegen indirect prompt injection
4. Context-filtering vóór generatie
5. Output-validatie en datalek-monitoring

Jouw taak:
Werk uitsluitend punt 3 uit: "Input-sanitisatie tegen indirect prompt injection".

Instructies voor jouw uitvoer:
- Schrijf direct de inhoudelijke alinea's voor dit specifieke punt.
- Begin NIET met een titel, nummering of herhaling van het punt.
- Schrijf geen inleiding of afsluiting voor het hele document.
- Houd de toon zakelijk, feitelijk en diepgaand technisch.

 
## A working Python implementation with asyncio

 The code below demonstrates how to build a Skeleton-of-Thought pipeline using Python's asyncio and a generic OpenAI-compatible API client. Instead of linear chains that wait for successive steps — as described in the guide on [breaking down complex tasks via prompt chaining](https://community.llmnet.nl/en/prompt-chaining) — this script shows how concurrent coroutines drastically minimize total wait time.

 import asyncio
import re
import time
from openai import AsyncOpenAI

client = AsyncOpenAI()

async def generate_skeleton(query: str) -> list[str]:
 system_prompt = (
 "Je bent een planner. Geef uitsluitend een genummerde lijst van 3 tot 5 "
 "korte kernpunten (max 7 woorden per punt) als antwoordskelet. "
 "Geen introductie of extra tekst."
 )
 response = await client.chat.completions.create(
 model="gpt-4o-mini",
 messages=[
 {"role": "system", "content": system_prompt},
 {"role": "user", "content": query}
 ],
 temperature=0.2,
 max_tokens=150
 )
 raw_text = response.choices[0].message.content.strip()
 
 # Parse genummerde regels: "1. Punt" -> "Punt"
 points = []
 for line in raw_text.split("\n"):
 match = re.match(r"^\d+[\.\)]\s*(.+)$", line.strip())
 if match:
 points.append(match.group(1).strip())
 return points

async def expand_point(query: str, skeleton: list[str], point_idx: int, point_title: str) -> str:
 skeleton_formatted = "\n".join(f"{i+1}. {p}" for i, p in enumerate(skeleton))
 system_prompt = (
 "Je bent een technisch expert. Werk uitsluitend het aangewezen punt uit "
 "van het onderstaande skelet. Schrijf direct de inhoudelijke tekst. "
 "Herhaal de titel of het nummer niet."
 )
 user_prompt = (
 f"Originele vraag: {query}\n\n"
 f"Volledig skelet:\n{skeleton_formatted}\n\n"
 f"Werk nu punt {point_idx + 1} uit: '{point_title}'."
 )
 
 response = await client.chat.completions.create(
 model="gpt-4o-mini",
 messages=[
 {"role": "system", "content": system_prompt},
 {"role": "user", "content": user_prompt}
 ],
 temperature=0.5,
 max_tokens=400
 )
 return response.choices[0].message.content.strip()

async def skeleton_of_thought_pipeline(query: str) -> str:
 t0 = time.perf_counter()
 
 # Stap 1: Genereer het skelet
 skeleton = await generate_skeleton(query)
 t_skeleton = time.perf_counter()
 print(f"Skelet gereed ({len(skeleton)} punten) in {t_skeleton - t0:.2f}s")
 
 # Stap 2: Parallel uitwerken via asyncio.gather
 tasks = [
 expand_point(query, skeleton, idx, point)
 for idx, point in enumerate(skeleton)
 ]
 expansions = await asyncio.gather(*tasks)
 t_expanded = time.perf_counter()
 print(f"Alle deelpunten parallel uitgewerkt in {t_expanded - t_skeleton:.2f}s")
 
 # Stap 3: Assemblage
 assembled_document = [f"# Analyse: {query}\n"]
 for idx, (point, content) in enumerate(zip(skeleton, expansions)):
 assembled_document.append(f"## {idx + 1}. {point}\n{content}\n")
 
 final_output = "\n".join(assembled_document)
 print(f"Totale verwerkingstijd: {t_expanded - t0:.2f}s")
 return final_output

# Voorbeeld-aanroep
# asyncio.run(skeleton_of_thought_pipeline("Hoe schaal je een multi-tenant vector database?"))

 
## Quality and consistency: The absence of cross-point context

 The most important architectural trade-off of Skeleton-of-Thought is the loss of autoregressive conditioning context between the sub-points. In a standard model call, paragraph 4 "knows" exactly what was written in paragraphs 2 and 3; the model aligns phrasing, builds up arguments, and avoids repeating the same examples.

 Because with SoT all expansions happen simultaneously in isolated context windows, specific quality defects can occur:

 
 
- Style breaks and tonal stutter: Worker 1 uses a formal academic style with passive sentence constructions, while Worker 3 switches to an active, direct instructional tone.
 
- Internal redundancy: Different workers independently define the same acronym or re-explain the same basic concept in their introductory sentences.
 
- Contradictions: For open-ended questions, Worker 2 might recommend a technology that Worker 4 labels as unsafe or outdated, without the document making this nuance explicit.
 

 When consistency and reliability of reasoning are an absolute priority, it can be necessary to combine parallel expansions with multiple sampling, as documented in the article on [self-consistency prompting for reasoning steps](https://community.llmnet.nl/en/self-consistency-prompting-redeneringsstappen). This evaluates multiple parallel paths to select the most coherent outcome.

 
## Assembly strategies: From concatenation to recombination

 After completing the parallel expansion calls, the fragments need to be merged into a coherent final result. There are three common methods for shaping this aggregation:

 
### 1. Direct concatenation (Deterministic Stitching)

 This is the fastest and cheapest method: the orchestrator pastes the titles from the skeleton and the expanded paragraphs one after another into a Markdown or HTML structure. This costs 0 extra tokens and 0 milliseconds of inference time, but doesn't resolve any style differences or duplicate definitions. This pattern is excellent for reference works, modular documentation, and structured checklists.

 
### 2. Lightweight reconciliation (Smoothing Pass)

 The assembled text is run through a fast, small language model with the instruction to smooth transitions between paragraphs and remove duplicate introductions only, without altering the substantive facts. This adds a small amount of sequential latency (about 1 to 2 seconds), but significantly improves editorial readability.

 
### 3. Hierarchical recombination

 For very large reports, the skeleton itself can be recursive: a master skeleton defines chapters, after which sub-skeletons are generated and expanded in parallel. The aggregation then proceeds tree-structure-wise, generating a summarizing conclusion per section.

 
## When should you use Skeleton-of-Thought, and when not?

 Skeleton-of-Thought is not a universal replacement for traditional prompting. It's a specialized pattern that excels under specific conditions and fails under others.

 
 Rule of thumb: Use Skeleton-of-Thought when the output needs to be long (>800 tokens), the structure lends itself to enumeration or sectioning, and the time to the complete answer (Time-to-Last-Token) is business-critical.

 

 Excellent use cases:

 
 
- Extensive overview reports: Comparisons of market segments, analyses of legislation, or overviews of best practices.
 
- System architecture documents: Where frontend, backend, database, security, and deployment can be described in parallel.
 
- Educational guides and curricula: Where each chapter covers a clearly delineated topic.
 
- Brainstorming and option analyses: Generating five different scenarios in parallel along with their pros and cons.
 

 Unsuitable use cases:

 
 
- Mathematical proofs and logical deduction: Where each step mathematically depends on the previous one.
 
- Complex software codebases: Where functions in file $B$ depend on the exact types and declarations just generated in file $A$.
 
- Creative storylines and prose: Where narrative tension building and subtle foreshadowing require a continuous contextual flow.
 

 
## Cost, rate limits, and token overhead in production

 The time savings of Skeleton-of-Thought come with a clear financial and infrastructural price. Instead of one prompt context, we send the initial question and the skeleton to the LLM provider $K$ times. With a skeleton of 5 points, the base context is duplicated five times in the input tokens.

 Moreover, a sudden burst of 5 to 10 concurrent requests per end user places heavy load on the API rate limits (both Requests-Per-Minute and Tokens-Per-Minute). When multiple users trigger an SoT pipeline simultaneously, the orchestration layer needs to have robust concurrency limits, semaphores, and retry mechanisms with exponential backoff to prevent HTTP 429 errors.

 Finally, prompt caching plays a crucial role in modern infrastructures. Because the $K$ parallel expansion calls largely share identical system prompts and document contexts, API gateways with automatic prefix caching can significantly reduce the cost and processing time of this redundant input. By consistently placing the static context at the front of the prompt window, the financial downside of Skeleton-of-Thought in production environments is largely neutralized.