AI Agent Tracing With OpenTelemetry

AI agent tracing should connect a user request to model calls, tool effects, retries, and parallel work without copying sensitive prompts into telemetry. This guide defines a redacted OpenTelemetry span contract and a causal fixture that makes debugging useful without creating a second secrets database.

The intended reader owns an agent runtime or its observability pipeline. You will leave with stable span boundaries, correlation rules, redaction tests, and a trace review that distinguishes orchestration from side effects.

The implementation connects OpenTelemetry GenAI, tool-call spans, agent observability, and trace correlation while keeping content capture optional and policy-bound.

1. Request
2. Model span
3. Tool effect
4. Outcome

AI agent tracing follows causal work

AI agent tracing begins with creating spans around operations that consume time, change state, or explain a decision rather than every line of orchestration code. This is the narrow boundary for this section; everything outside it belongs in a separate capacity, policy, or product decision. The OpenTelemetry generative observability guidance frames generative operations as ordinary distributed telemetry connected through shared context and consistent semantics.

Work through four explicit moves:

• Start one span at the user boundary
• Create children for model and tool operations
• Link asynchronous work to its cause
• Close only after outcome is known

In a working review, I would put the first move beside the input fixture, use the second to expose the decision boundary, and make the third observable before polishing the interface. The fourth move is the release check. This order matters because a convincing happy path can still conceal incompatible state, unfair scheduling, inaccessible fallback, or ownership ambiguity. Keeping each move named also lets another engineer reproduce the result without inheriting private context.

The named failure mode is turning logs into a flat chronological transcript. Its consequence is parallel work and retries become impossible to attribute.

Mitigate it with parent-child context plus explicit span links. The release receipt is one trace that reconstructs the request without timestamps alone. Those fields connect the implementation to the article's single question and make a later update comparable instead of anecdotal.

Use semantic attributes conservatively

A useful AI agent tracing decision depends on adopting stable operation, provider, model, token, error, and finish fields while isolating experimental conventions. This is the narrow boundary for this section; everything outside it belongs in a separate capacity, policy, or product decision. The OpenTelemetry generative attribute registry defines current attribute names and stability notes that an implementation should version rather than reinvent.

Work through four explicit moves:

• Pin the convention version
• Keep names low-cardinality
• Record units and absence explicitly
• Namespace local fields away from standards

In a working review, I would put the first move beside the input fixture, use the second to expose the decision boundary, and make the third observable before polishing the interface. The fourth move is the release check. This order matters because a convincing happy path can still conceal incompatible state, unfair scheduling, inaccessible fallback, or ownership ambiguity. Keeping each move named also lets another engineer reproduce the result without inheriting private context.

The named failure mode is inventing high-cardinality tags for every prompt and tool. Its consequence is cost, query reliability, and privacy degrade together.

Mitigate it with an allowlist reviewed as an API contract. The release receipt is a schema file with owner, stability, and example value. Those fields connect the implementation to the article's single question and make a later update comparable instead of anecdotal.

Build the retry and parallel fixture

The worked AI agent tracing fixture makes representing one failed model attempt, two parallel reads, one retried call, and one verified effect as separate causal nodes. This is the narrow boundary for this section; everything outside it belongs in a separate capacity, policy, or product decision.

Work through four explicit moves:

• Give each attempt a unique span ID
• Reuse the bounded trace context
• Link fan-out children to their join
• Assert the effect appears once

In a working review, I would put the first move beside the input fixture, use the second to expose the decision boundary, and make the third observable before polishing the interface. The fourth move is the release check. This order matters because a convincing happy path can still conceal incompatible state, unfair scheduling, inaccessible fallback, or ownership ambiguity. Keeping each move named also lets another engineer reproduce the result without inheriting private context.

The named failure mode is overwriting a failed attempt with the successful retry. Its consequence is latency and duplicate-effect risk disappear from analysis.

Mitigate it with immutable attempt spans and effect identity. The release receipt is a deterministic trace tree with expected parent and link sets. Those fields connect the implementation to the article's single question and make a later update comparable instead of anecdotal.

Signal	Decision	Proof
Operation name	Record	Stable low-cardinality value
Prompt body	Omit by default	Policy denies content capture
Tool effect ID	Hash or tokenize	Correlates without raw secret

Redact before export

AI agent tracing needs an explicit rule for classifying attributes at instrumentation time so prompts, responses, credentials, URLs, and user identifiers never depend on a downstream cleanup job. This is the narrow boundary for this section; everything outside it belongs in a separate capacity, policy, or product decision.

Work through four explicit moves:

• Default every content field to absent
• Hash only identifiers needed for joins
• Block secrets by key and value pattern
• Test the exported representation

In a working review, I would put the first move beside the input fixture, use the second to expose the decision boundary, and make the third observable before polishing the interface. The fourth move is the release check. This order matters because a convincing happy path can still conceal incompatible state, unfair scheduling, inaccessible fallback, or ownership ambiguity. Keeping each move named also lets another engineer reproduce the result without inheriting private context.

The named failure mode is sending raw content to a collector and redacting later. Its consequence is sensitive material reaches queues, retries, and backups.

Mitigate it with source-side allowlisting and deny-by-default exporters. The release receipt is a test that searches exported spans for forbidden values. Those fields connect the implementation to the article's single question and make a later update comparable instead of anecdotal.

Runnable artifact. Save this as opentelemetry-ai-agent-tracing.test.mjs and run node --test opentelemetry-ai-agent-tracing.test.mjs. Expected result: PASS: causal trace exported without prompt content. The checked-in copy lives with this batch's evidence.

import assert from "node:assert/strict";
import test from "node:test";

const secret = "customer@example.com";
const spans = [
  { id: "turn", parent: null, name: "agent.turn" },
  { id: "read-a", parent: "turn", name: "tool.read", prompt: secret },
  { id: "read-b", parent: "turn", name: "tool.read" },
  { id: "retry", parent: "turn", name: "model.call", attempt: 2 },
];
const exported = spans.map(({ prompt, ...allowed }) => allowed);

test("keeps causality and removes content", () => {
  assert.equal(exported.filter((span) => span.parent === "turn").length, 3);
  assert.equal(JSON.stringify(exported).includes(secret), false);
  assert.equal(exported.find((span) => span.id === "retry").attempt, 2);
  console.log("PASS: causal trace exported without prompt content");
});

Separate orchestration from effects

In production, AI agent tracing turns on distinguishing a tool proposal, authorization, network attempt, external effect, and verification read. This is the narrow boundary for this section; everything outside it belongs in a separate capacity, policy, or product decision.

Work through four explicit moves:

• Name intent before the call
• Attach the policy decision
• Record each transport attempt
• Verify outside state independently

In a working review, I would put the first move beside the input fixture, use the second to expose the decision boundary, and make the third observable before polishing the interface. The fourth move is the release check. This order matters because a convincing happy path can still conceal incompatible state, unfair scheduling, inaccessible fallback, or ownership ambiguity. Keeping each move named also lets another engineer reproduce the result without inheriting private context.

The named failure mode is labeling a successful HTTP response as completed business work. Its consequence is the trace claims an effect that may not exist or may have duplicated.

Mitigate it with effect receipts and reconciliation spans. The release receipt is an outcome chain from proposal through verified state. Those fields connect the implementation to the article's single question and make a later update comparable instead of anecdotal.

Control sampling with risk

Safe AI agent tracing requires retaining errors, high-latency traces, and consequential effects at higher rates without making content collection automatic. This is the narrow boundary for this section; everything outside it belongs in a separate capacity, policy, or product decision.

Work through four explicit moves:

• Tail-sample on declared outcomes
• Preserve rare policy denials
• Budget normal successful traffic
• Keep content policy independent

In a working review, I would put the first move beside the input fixture, use the second to expose the decision boundary, and make the third observable before polishing the interface. The fourth move is the release check. This order matters because a convincing happy path can still conceal incompatible state, unfair scheduling, inaccessible fallback, or ownership ambiguity. Keeping each move named also lets another engineer reproduce the result without inheriting private context.

The named failure mode is sampling only fast successes. Its consequence is the trace store cannot explain the incidents it was built for.

Mitigate it with risk-aware tail rules and audited overrides. The release receipt is sampled-versus-observed counts by outcome class. Those fields connect the implementation to the article's single question and make a later update comparable instead of anecdotal.

1. PlanPlan

   Create one orchestration span for the bounded turn.

2. ForkFork

   Link parallel model or tool children to the same cause.

3. RetryRetry

   Create a new attempt span with the prior outcome.

4. JoinJoin

   Summarize verified effects and final status.

Operate retention and access

A AI agent tracing rollout should preserve assigning span classes different lifetimes and making trace search an authorized production capability. This is the narrow boundary for this section; everything outside it belongs in a separate capacity, policy, or product decision.

Work through four explicit moves:

• Set retention by field sensitivity
• Restrict content-bearing debug sessions
• Audit reads and exports
• Delete derived stores on schedule

In a working review, I would put the first move beside the input fixture, use the second to expose the decision boundary, and make the third observable before polishing the interface. The fourth move is the release check. This order matters because a convincing happy path can still conceal incompatible state, unfair scheduling, inaccessible fallback, or ownership ambiguity. Keeping each move named also lets another engineer reproduce the result without inheriting private context.

The named failure mode is treating telemetry as harmless engineering exhaust. Its consequence is a long-lived behavioral dataset escapes product privacy controls.

Mitigate it with data inventory, access review, and enforceable expiry. The release receipt is retention evidence and access-log samples. Those fields connect the implementation to the article's single question and make a later update comparable instead of anecdotal.

Review traces as product evidence

The evidence for AI agent tracing is strongest when asking whether the trace explains delay, choice, effect, recovery, and user-visible outcome without exposing unnecessary content. This is the narrow boundary for this section; everything outside it belongs in a separate capacity, policy, or product decision.

Work through four explicit moves:

• Choose one real incident class
• Reconstruct it from spans alone
• Identify missing or excessive fields
• Update the contract before dashboards

In a working review, I would put the first move beside the input fixture, use the second to expose the decision boundary, and make the third observable before polishing the interface. The fourth move is the release check. This order matters because a convincing happy path can still conceal incompatible state, unfair scheduling, inaccessible fallback, or ownership ambiguity. Keeping each move named also lets another engineer reproduce the result without inheriting private context.

The named failure mode is optimizing dashboard aesthetics before causal completeness. Its consequence is operators see attractive charts but cannot answer what happened.

Mitigate it with trace-based incident rehearsals and schema review. The release receipt is a completed reconstruction checklist with gaps assigned. Those fields connect the implementation to the article's single question and make a later update comparable instead of anecdotal.

Put the decision into practice

AI agent tracing works when causal structure and data minimization reinforce each other. Stable spans connect model, tool, retry, policy, and effect evidence; a source-side allowlist prevents that usefulness from turning prompts and secrets into permanent telemetry.

Begin with the runnable four-span fixture, then model one real failure from the service. If an operator can reconstruct the cause and verified effect while the exporter contains no forbidden content, the contract is ready for a measured rollout.

The method connects to four existing Journal notes: AI traces need redaction contracts, multi-agent causal testing, agent activity logs, logs with user-journey context. Each link covers an adjacent boundary while this article stays focused on one outcome. Keep the fixture, visual evidence, command output, and release receipt together so the next review can test the claim against the same starting conditions.