How Should Teams Use OpenTelemetry to Observe AI Agents in 2026?

What Is OpenTelemetry for AI Agents?

OpenTelemetry for AI agents is the application of the OpenTelemetry tracing standard to events produced by models, tools, retrieval systems, and orchestration runtimes. It records an agent run as connected spans rather than leaving each component with a separate log stream. As of September 26, 2026, OpenTelemetry’s generative-AI semantic conventions are the most portable foundation for this work, but they are still evolving and should not be confused with a complete product category. AWS, Databricks, Dynatrace, Arize, New Relic, Oracle, and several smaller vendors have announced agent-observability capabilities, while projects such as Trail focus on signed traces and Flow addresses dynamic task execution.

Also worth reading: Runtime Security Architecture for AI Agents: How Should Teams Control Autonomous Workflows in 2026? · What is an agent governance policy as code, and how do teams implement it for AI agents? · How do I properly configure the OpenTelemetry Tail Sampling Processor for production tracing?

For a multi-agent workflow, telemetry can show which agent received a task, which model and tool it selected, how long each operation took, what token usage was reported, and whether one agent handed work to another. A trace may include separate spans for planning, model inference, retrieval, tool execution, validation, and final response assembly. This matters because a request that takes 18 seconds may contain a 1.2-second model call, a 15-second retrieval operation, and a 1.8-second queue delay. Without linked spans, teams often see a correct final answer but cannot determine which component caused the delay or failure.

OpenTelemetry itself does not provide an AI control plane, an agent sandbox, or automatic proof that an agent behaved correctly. It is an instrumentation and telemetry standard, supported by libraries, collectors, and compatible storage backends. The closest vendor-neutral reference for implementation is OpenTelemetry’s generative-AI semantic-conventions repository. Teams should treat those conventions as an interoperability contract, then add organization-specific attributes such as customer tier, workflow version, delegated authority, and policy decision.

## How Agent Tracing Differs from Ordinary Microservice Tracing

Traditional distributed tracing already models a request moving from a web service to a database, but an agent adds decisions whose paths are selected at runtime. A conventional service might call one database; an agent might call a vector database, a search engine, a code interpreter, a payment tool, and another agent. It may then retry one operation or branch into several candidates. The trace therefore needs to represent both execution structure and decision structure, which is why ordinary span duration alone is insufficient.

The OpenTelemetry data model is well suited to hierarchy. One trace can represent an end-user request, spans can represent orchestration steps, and nested spans can represent model calls or tool executions. Links are useful when an agent is started by a scheduler, background queue, or earlier planning step that should remain part of the trace without becoming the active parent. Span attributes can then describe attributes such as the generative-AI operation type, provider, requested model, response model, token counts, finish reason, and agent identity where instrumentation knows them. Exact attribute names and stability levels should be checked against the current specification because conventions can change between releases.

There is an important distinction between observability and evaluation. Observability tells teams that a model made 4,200 input tokens and 610 output tokens, called a tool three times, and incurred $0.037 in provider charges. It does not by itself establish whether the answer was factually correct, ethically appropriate, or compliant with a policy. Reliable evaluation needs references, expected outcomes, or a reviewer and should be connected to traces for investigation. A dashboard that labels every successful HTTP request as a successful agent run misses this distinction.

A useful design records three layers: business intent, such as “refund order 1842”; execution telemetry, such as planning and tool spans; and outcome evidence, such as an approved refund identifier and policy-version identifier. The business and evidence fields may be custom, but the telemetry transport should remain OpenTelemetry-compatible. This approach gives multi-agent systems enough detail for debugging while preserving portability across tracing backends and orchestration frameworks.

What Teams Should Instrument First

Start with the request boundary and create one root span for each logical workflow, not one root span for every model invocation. Give it a trace ID that can be searched by customer, conversation, job, or workflow-run ID. The root span should represent elapsed time from accepted work to terminal completion and should carry a clear result such as success, user cancellation, timeout, policy denial, or internal error. If a run fans out across 12 agents, those agents should normally remain inside one trace or a controlled set of linked traces.

Next, instrument orchestration decisions. Record the selected agent, workflow version, task, decision source, and handoff relationship. Model spans should include the provider and model actually used, the operation type, temperature or other requested settings when known, reported token usage, latency, finish status, and retry count. Avoid storing complete prompts or completions by default because they may contain regulated data, secrets, source code, or personal information. Hashing or sampling can confirm identity without reproducing sensitive text.

Tool calls deserve equal attention. Record tool name, normalized input and output size, authorization decision, endpoint class, status, timeout, and retry count, while redacting credentials and direct identifiers. Retrieval spans should distinguish indexing time from query time and should report result count, source classes, latency, and evaluation or reranking stages. For agents that write files, execute code, or call external APIs, record the action type, sandbox decision, resource affected, and whether approval was required. AWS AgentCore Observability and comparable products can cover some managed agent stacks, but teams should still confirm exactly which fields are exported and whether a custom OpenTelemetry pipeline is supported.

Finally, define terminal outcomes and service-level indicators before collecting everything. A reasonable initial set includes completion rate, tool-error rate, p50 and p95 end-to-end latency, model fallback rate, trace coverage, and percentage of failed runs linked to a sampled trace. Do not make token cost the primary success metric: minimizing tokens can increase retries and wall-clock time. Measure cost per successful business outcome, such as $0.12 per resolved ticket, rather than cost per call.

A Practical Implementation Process

A staged rollout reduces instrumentation risk. In week one, map a representative workflow from entry to completion, including every model, tool, queue, and external service. Select one low-risk production use case and designate a trace ID, workflow-run ID, and agent ID. In week two, add OpenTelemetry SDKs or auto-instrumentation to the orchestrator, model gateway, and highest-volume tools, then route spans through a collector owned by the team. Configure a consistent environment, service, workflow, agent, and version set of resource attributes before building dashboards.

During weeks three and four, add semantic-convention fields and test failure cases: malformed model output, rejected tool input, authorization failure, provider timeout, partial completion, and retry. Verify that trace context survives asynchronous queues and parallel branches. A test is incomplete if every component logs separately but no collector receives a coherent trace. A practical acceptance target is at least 95% trace coverage for production runs, 99% correlation between terminal workflow records and trace IDs, and less than 2% telemetry export failure during the first month. These are starting thresholds, not universal standards, and should be adjusted for sampling and privacy requirements.

After 30 days, review telemetry with engineers, security personnel, product owners, and the agent owner. Establish redaction rules, access controls, retention periods, and approved storage locations. Build views for latency, reliability, tool use, model usage, cost, and policy events rather than one overloaded dashboard. Only then should a team tune prompts, routing, or concurrency based on the data. The process is iterative because semantic conventions, SDK behavior, and supported backends can change; pin library versions, test upgrades in staging, and maintain a conformance test suite.

Sampling needs special care in agent systems. Head sampling based on incoming request rate can discard rare but expensive failures. Tail sampling in OpenTelemetry Collector can retain errors, timeouts, unusually expensive runs, or a percentage of successful requests, but the collector must wait for all spans in a trace to arrive. Sensitive payloads should be removed before export where possible, because collector-based redaction may see data after application code has already placed it in memory. Organizations subject to HIPAA, GDPR, PCI DSS, or contractual restrictions should involve compliance and security teams before enabling prompt capture.

OpenTelemetry Versus Agent Observability Platforms

The main choice is not simply “open source versus commercial.” It is between a portable telemetry layer and a broader operating environment. OpenTelemetry is attractive when an organization already uses multiple model providers, clouds, and observability systems, or when contractual requirements prohibit lock-in. It costs engineering time to instrument, map identities, configure collectors, and build queries, but the trace data can generally move to Grafana, commercial tracing products, log platforms, or data warehouses that support OpenTelemetry.

Agent-focused products can provide faster time to visibility. Their dashboards may already understand agent sessions, evaluation runs, prompt versions, model comparisons, guardrails, and workflow branching. A managed tracing service may also reduce collector operations and support authentication, retention, alerts, and incident integrations. The tradeoff is that fields, pricing, and export behavior can differ by vendor. New Relic announced AI-agent and OpenTelemetry capabilities, while Arize describes a broader self-improving-agent platform; these may suit teams wanting prebuilt analytics rather than merely transport and SDK standards.

No product removes the need to define what the agent was supposed to do. Vendor dashboards can make it easy to compare tokens and latencies while leaving unanswered whether a successful support resolution required a prohibited action. Teams should run a small proof of concept using their hardest workflow, perhaps a research agent with 6 tools and 3 delegated subtasks. Import at least 1,000 representative or synthetic runs, test error sampling, export a vendor-neutral record, and measure how long an engineer needs to identify the cause of one failed run. A target of under 10 minutes is more meaningful than a generic feature checklist.

A hybrid design is often practical: use OpenTelemetry as the system-wide collection contract, a managed backend for storage and alerting, and a separate evaluation or orchestration platform for workflow controls. The key is to avoid making one proprietary UI the only place where causal relationships exist. OpenTelemetry complements platforms such as AWS AgentCore, Databricks, Oracle database observability, and other vendor offerings; it does not automatically replace their managed security, governance, or execution features.

Common Mistakes and Measurement Traps

One common error is tracing only the language-model call. That produces many short spans and little evidence about the workflow. Another is using one trace for an entire open-ended conversation, which can create huge traces that are expensive to store and difficult for backends to handle. Better boundaries are usually a user request, scheduled job, or independently retryable workflow. Long-running agents can emit checkpoint events and linked child traces when their total duration makes a single trace impractical.

Teams also over-record prompts and tool outputs. Debugging convenience does not justify retaining secrets or regulated content. A safer pattern records a content hash, content length, source reference, and policy classification, with a separately governed “debug envelope” available under strict access. Another mistake is assuming that a tool’s HTTP 200 response means it succeeded. Validate business status, output schema, side effects, and idempotency separately. For payment or account-changing tools, include an immutable decision record or transaction reference.

Mean latency is another trap. A few long retrieval calls can make the average acceptable while users abandon slower requests; use p50, p90, p95, and p99 latency by workflow. Token averages can be similarly misleading, as total cost may be dominated by retries and repeated context. Do not compare model prices without normalizing provider billing units, cached input, reasoning tokens, tool fees, and retry behavior. Vendor model names can also change, so record both the requested and actual model where the gateway can provide them.

Finally, avoid an instrument-everything strategy. Excessive span volume can increase cost and obscure the failure signal. Begin with 20 to 30 high-value attributes, 5 to 10 useful metrics, and the major workflow transitions. Measure collector CPU, memory, dropped spans, ingestion cost, and sampling rates. Observability that cannot be afforded or trusted at production volume is not complete observability.

When to Act, and What It May Cost

Act now when an agent has crossed from experimentation into production, uses multiple tools or models, delegates work to other agents, or has crossed a risk or reliability threshold. Examples include more than 500 production runs per day, a p95 response target, a stated tool-error budget, or a requirement to reconstruct a high-impact action after an incident. Teams do not need to instrument every internal function first; they need a defensible path from user intent through delegated work to evidence of completion.

The software itself may be free. OpenTelemetry SDKs and the Collector are open source, and many backends offer free tiers, but the total cost includes engineering setup, telemetry storage, computing for tail sampling, dashboards, alerting, redaction, and ongoing maintenance. A small team can often start with existing infrastructure, but a high-volume commercial deployment may need dedicated ingestion capacity. Budget by retained spans rather than only requests, then estimate the monthly data using actual span counts and average bytes per span. Revalidate the calculation after 30 and 90 days because traces are rarely uniform in size.

Do not delay solely because standards are still developing; instrument around a versioned semantic-convention layer and keep business attributes in a controlled namespace. Conversely, do not claim full OpenTelemetry AI compliance from generic HTTP spans. Verify that model and tool operations carry agreed attributes, trace context is propagated, sensitive fields are controlled, and analytics distinguish observations from evaluations. The technology is mature enough for production foundations, but organizational semantics and outcome evaluation remain team responsibilities.

The best time to adopt OpenTelemetry for AI agents is before a multi-agent system becomes difficult to explain. Begin with one workflow, establish a 95% trace-coverage target, retain errors and representative successes, and review results weekly. Expand only after teams can answer three questions reliably: which component failed, which agent handoff caused the behavior, and what did the run cost. That standard is more useful than collecting hundreds of metrics or promising automatic explanation before the evidence is complete.