Agent observability: seeing what your agent did and why

When an LLM serves as an autonomous reasoning engine that independently plans API calls, retrieves context, and makes decisions, traditional software monitoring falls short. A simple HTTP status code or response time reveals nothing about the internal logic, intermediate reasoning steps, or unexpected paths the model took. Agent observability transforms this opaque decision-making chain into a fully inspectable tree structure.

For classic stateless API calls, logging input, output, latency, and token counts is sufficient. As soon as an application evolves into an agent featuring cyclic loops, dynamic memory mutations, and varying tool selections, a non-deterministic system emerges. Anyone wanting to understand why an agent gets stuck after seven iterations or executes an incorrect database update needs to be able to reconstruct every decision point. Those looking to explore the theoretical foundation behind this shift from static prompts to dynamic systems can read the overview article on AI agents for a solid introduction to autonomous architectures.

The fundamental difference between LLM logging and Agent observability

Classic LLM logging focuses on individual transactions: a prompt is sent to the model, and a response is returned. We measure call duration, record the model version used, and count input and output tokens to verify billing. This form of logging answers the question: how much did this specific call cost, and how fast did the model complete it?

Agent observability answers an essentially different question: which reasoning steps led to this specific tool call, how did the system's internal state change, and where did the behavior diverge from the intended path? An agent rarely executes a single generation; it traverses a chain of observations, thoughts, actions, and state updates. If an intermediate step fails, a robust agent attempts to take an alternative path. Without in-depth observability, it is impossible to determine whether an error stemmed from a flawed tool description, a hallucinated parameter, a failed parsing step, or an issue within the source system.

In addition, agent observability requires insight into context dynamics across multiple steps. As an agent calls more tools, the context history grows exponentially with payload data, error messages, and intermediate results. Observability must clarify how this accumulation of tokens affects reasoning quality and at what point stale data leads the agent astray.

The three pillars: Traces, Spans, and Graph State

To make complex agentic workflows measurable, modern observability builds upon concepts from distributed tracing, augmented with AI-specific metadata. The structure rests on three interconnected layers:

• Traces (the overarching session): A trace represents the complete lifecycle of a task, from the initial user query to the final outcome. It contains the high-level objective, total duration, cumulative token costs, and the final status.
• Spans (individual operations): Within a trace are multiple nested spans. A span represents a discrete action: a single LLM call, the execution of a local Python function, a vector search in a database, or an external API transaction. Spans contain exact start and end times, input parameters, raw responses, and error codes.
• Graph State & Memory Snapshots: In addition to the linear timeline, observability records the state of the agent's memory and variables before and after each span. This captures mutations in the working context, such as updated priority lists or intermediate search results.

By consistently linking these layers with unique identifiers (such as a trace_id, parent_span_id and session_id), a developer can visualize a chaotic execution history as a clear tree structure or a directed graph. When agents are designed as deterministic state transitions, it helps to understand how control mechanisms shift; therefore, consult the guide on graph engineering to see how structured control loops simplify traceability.

Logging tool interactions and argument inspection

The most vulnerable link in an agent workflow is the interaction with external tools. Based on a function description, the model generates a structured JSON object that is subsequently executed by the runtime application. Errors manifest here in various ways: syntactically invalid JSON, missing mandatory parameters, correctly formatted but logically impossible values (such as a negative price), or the selection of a completely irrelevant tool.

Observability therefore requires that both the generated JSON payload and the raw output of the tool are captured down to the byte level. If a database query yields zero results, it must be visible in the span whether this was due to an erroneous SQL WHERE clause from the agent or an actual absence of data in the table.

When the agent consistently selects the wrong function, the root cause often lies in ambiguous schemas; consult the article on effective tool descriptions to discover how precise parameter documentation prevents faulty calls.

Implementation model: An OpenTelemetry-compatible trace schema

To prevent vendor lock-in and enable standardized data exchange, modern systems adopt OpenTelemetry-compliant structures with LLM-specific semantic conventions. The JSON artifact below illustrates a representative span in which an agent decides to invoke an external retrieval tool, including its reasoning step (thought), parameters, response latency, and token distribution.

{
  "trace_id": "tr-8f4b2c9a-20260815-9941",
  "span_id": "sp-agent-decide-003",
  "parent_span_id": "sp-root-task-001",
  "name": "agent_reasoning_and_tool_call",
  "start_time_unix_nano": 1786800000000000000,
  "end_time_unix_nano": 1786800001450000000,
  "attributes": {
    "llm.vendor": "anthropic",
    "llm.model": "claude-3-7-sonnet",
    "llm.temperature": 0.2,
    "llm.usage.prompt_tokens": 1420,
    "llm.usage.completion_tokens": 185,
    "llm.usage.total_tokens": 1605,
    "agent.cycle_index": 3,
    "agent.thought": "Gebruiker vraagt orderstatus van #4921. Ik roep order_lookup aan.",
    "agent.tool_name": "order_service_lookup",
    "agent.tool_arguments": "{\"order_id\": 4921, \"include_tracking\": true}",
    "agent.tool_call_valid": true,
    "gen_ai.system": "anthropic",
    "gen_ai.request.model": "claude-3-7-sonnet"
  },
  "events": [
    {
      "name": "tool_execution_started",
      "timestamp_unix_nano": 1786800001460000000
    },
    {
      "name": "tool_execution_completed",
      "timestamp_unix_nano": 1786800001890000000,
      "attributes": {
        "tool.status": "success",
        "tool.response_length_bytes": 412
      }
    }
  ],
  "status": {
    "code": "OK"
  }
}

By writing this schema to a central log sink (such as ClickHouse, Elasticsearch, or an OpenTelemetry Collector) on every iteration, a temporal audit trail is created. This allows developers to retrospectively reproduce why the model at timestamp T believed that order #4921 needed to be retrieved and which parameters were passed along.

Automatically detecting common failure patterns

Observability is only truly valuable when the system can detect anomalous patterns in real time without requiring an engineer to manually comb through thousands of log lines. In autonomous loops, specific failure modes emerge that can be caught using deterministic rules:

When an agent gets stuck in one of these patterns, systematic analysis of the state transitions is essential; read the in-depth article on debugging agentic loops to learn methods for purposefully breaking cyclic errors and infinite repetitions.

Evaluation in production: From traces to quality metrics

Collecting traces is the foundation; the next step is quantifying the agent's effectiveness across thousands of sessions. This requires specific key performance indicators (KPIs) that diverge from traditional software metrics:

1. Task Success Rate (TSR): The percentage of traces in which the agent achieved the user's objective completely autonomously without human intervention or runtime crashes.
2. Step Efficiency Ratio: The number of required iterations relative to the theoretical minimum number of steps for a specific task class. An increasing ratio indicates inefficient reasoning paths or redundant search queries.
3. Tool Error Rate: The ratio of failed tool calls (validation errors, HTTP 500s, timeouts) to successful executions.
4. Cost per Resolved Task: The total token and compute costs expressed per successfully completed task, including all intermediate reasoning sessions.

In addition to automated metrics, collected production traces can serve as a representative test dataset for systematic optimization. To verify whether a modified prompt or instruction set reduces the failure rate, it is advisable to the guide on A/B testing prompts consult to validate changes in a controlled manner against historical production data.

For more advanced metrics, such as automatically evaluating intermediate steps with an independent judge model, the specialized article on agent evaluation guidelines for calculating trajectory accuracy and sub-goal success rates.

Data Redaction and Privacy in Agent Traces

Because agent observability comprehensively captures all inputs, internal thoughts, and tool outputs, a significant privacy and security risk arises. Traces frequently contain personally identifiable information (PII), session tokens, passwords, or confidential business data retrieved by tools.

A robust observability pipeline therefore implements data masking before spans are written to permanent storage. This occurs at two levels:

• Static Masking: Regular expressions and pattern recognition filter credit card numbers, social security numbers (BSN), email addresses, and API keys from both prompts and tool payloads.
• Structural Redaction at Schema Level: Sensitive fields in known tool schemas (such as customer.bank_account or auth.bearer_token) are explicitly marked and automatically replaced with a hashed value or fixed placeholder (for example [REDACTED]).

Masking data should not completely destroy the trace's debugging value. By applying consistent deterministic pseudonymization, it remains possible to verify whether the agent used the same entity in step 1 and step 4, without the actual personal data being readable by developers analyzing the logs.

Architectural Considerations: Build vs. Specialized Backends

When setting up observability, engineering teams face a choice: do we build our own logging layer on top of existing infrastructure (such as OpenSearch or Grafana Tempo), or do we integrate a specialized AI observability platform (such as Langfuse, Arize Phoenix, or Helicone)?

For organizations with strict compliance requirements, an OpenTelemetry pipeline routing data to an internal backend is often mandatory. For fast-paced product development, a specialized open-source platform capable of running self-hosted offers the best compromise between functionality and data sovereignty.

Checklist for Production-Grade Agent Observability

Before deploying an autonomous agent system, verify that the monitoring environment meets the following minimum technical requirements:

• Every agent execution generates a unique trace_id that is propagated across all asynchronous tasks and API boundaries.
• Every tool call logs exact input arguments, execution duration, HTTP/status code, and raw response payload.
• The system tracks token usage split into input, output, and potential cache hits per individual model invocation.
• A real-time alerting system is configured for infinite loops, repeated invalid arguments, and unexpected context growth.
• Automated PII and credential masking sanitizes sensitive data prior to persistence in trace databases.
• A clear link connects production traces to offline evaluation datasets, allowing failed real-world sessions to be converted directly into regression tests.

With a robust observability foundation, an unpredictable AI agent transforms into a manageable, auditable, and continuously optimizable software component. Developers no longer need to guess the internal logic of the model, but can verify and refine every decision, tool selection, and state mutation with mathematical precision.