Agent observability: the traces you open after an incident

Agent observability gets graded on one morning: the one after an agent did something confident and wrong. You open one trace. You need what the model saw, what it chose, what the tool returned, and which prompt version was live. A trace that holds only timings answers none of those.

Reading that trace is a postmortem skill, and Postmortems for something nobody typed covers it. The trace itself has to exist first, and someone instrumented it weeks before anything went wrong. That work is cheap to get right and expensive to retrofit. It has a schema that belongs to no vendor, a short list of attributes that answer the postmortem's questions, and a bill in three parts: storing content, sampling, and cardinality.

The shape of an agent trace

OpenTelemetry has a set of semantic conventions for generative AI, and they cover agents. Guangya Liu (IBM) and Sujay Solomon (Google) stated the goal on the OpenTelemetry blog in March 2025. It was a standard shape for the telemetry agent apps emit, "to avoid lock-in." Use them for that reason: your spans then outlive whichever LLM observability backend you pay for this year.

Two facts before you build on them. The conventions now live in their own repository, and the old pages on opentelemetry.io point there. And every spec document in it carries the status Development, as does every gen_ai.* attribute. Names can still change. Treat an upgrade of your instrumentation as a schema migration, not a dependency bump.

An agent run needs three span types from the agent spans and model spans documents, with the chat span defined in client inference:

• invoke_agent {gen_ai.agent.name}, kind INTERNAL for an agent in your own process, one per run.
• {gen_ai.operation.name} {gen_ai.request.model}, kind CLIENT, one per model call. For a chat model that reads chat gpt-4.1.
• execute_tool {gen_ai.tool.name}, kind INTERNAL, one per tool call.

The spec also defines invoke_workflow for a process made of several agents and plan for a planning phase. Start with the three.

Suppose a support team runs refund-agent. It reads a ticket, calls lookup_order, and calls issue_refund when a refund is due. One turn of that run, instrumented by hand:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor
from opentelemetry.trace import SpanKind
 
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("refund-agent")
 
with tracer.start_as_current_span(
    "invoke_agent refund-agent",
    kind=SpanKind.INTERNAL,
    attributes={
        "gen_ai.operation.name": "invoke_agent",
        "gen_ai.agent.name": "refund-agent",
        "gen_ai.conversation.id": "conv_7c41e9d2",
    },
):
    with tracer.start_as_current_span(
        "chat gpt-4.1",
        kind=SpanKind.CLIENT,
        attributes={
            "gen_ai.operation.name": "chat",
            "gen_ai.provider.name": "openai",
            "gen_ai.request.model": "gpt-4.1",
            "gen_ai.prompt.name": "refund-triage",
            "gen_ai.prompt.version": "v14",
        },
    ) as chat:
        # Set after the response arrives.
        chat.set_attributes({
            "gen_ai.response.id": "chatcmpl-9f2c71ab",
            "gen_ai.response.finish_reasons": ["tool_calls"],
            "gen_ai.usage.input_tokens": 6120,
            "gen_ai.usage.cache_read.input_tokens": 5400,
            "gen_ai.usage.output_tokens": 88,
            # Not in the spec: a pointer to the stored prompt and completion.
            "app.content.ref": "s3://agent-content/2026/10/14/chatcmpl-9f2c71ab.json",
        })
 
    with tracer.start_as_current_span(
        "execute_tool lookup_order",
        kind=SpanKind.INTERNAL,
        attributes={
            "gen_ai.operation.name": "execute_tool",
            "gen_ai.tool.name": "lookup_order",
            "gen_ai.tool.type": "function",
            "gen_ai.tool.call.id": "call_mszuSIzq0rd3r",
        },
    ):
        pass  # the real tool runs here

On Python 3.12 with opentelemetry-sdk 1.45.1, the console exporter prints one JSON object per span. Children print first, because they end first. Trimmed to names and parent links:

"name": "chat gpt-4.1",               "parent_id": "0x063aeea8f5110639"
"name": "execute_tool lookup_order",  "parent_id": "0x063aeea8f5110639"
"name": "invoke_agent refund-agent",  "parent_id": null

Every span in that output ends with status UNSET. That detail decides what a sampler keeps.

Sponsored:

Agent observability attributes that pay for themselves

A postmortem asks a fixed set of questions about a run. Each one maps to an attribute, and most of those attributes are small: a string, an integer, a short array. Read the table as a checklist against your own instrumentation.

Prompt version is the join between a change in behavior and a change in the system. The inference span makes gen_ai.prompt.version conditionally required whenever gen_ai.prompt.name is set and a version exists. The spec also says the value should match the identifier in your prompt management system, if you have one. Without it, "the agent started doing this on Thursday" has no join to "someone edited the prompt on Thursday."

The decision is visible without any content. A chat span with finish_reasons of ["tool_calls"] says the model chose to act. The execute_tool span that follows, under the same invoke_agent parent, records which tool ran and its gen_ai.tool.call.id. The chat span carries no such id: on the model's side it lives inside gen_ai.output.messages, which is content. Without content capture, the decision and its execution join only by parent and order, and the arguments are invisible. That is the next section's bill.

Token counts need one rule from the spec. gen_ai.usage.input_tokens should include every kind of input token, cached ones too. In the example, 6,120 input tokens with 5,400 read from cache means 720 uncached. Reasoning tokens, where a model reports them, sit inside gen_ai.usage.output_tokens. Get these wrong and the cost line in your postmortem double-counts. Where those tokens go over a whole run is its own piece on agent token spend.

Compaction is a positive-only flag. Instrumentation sets gen_ai.conversation.compacted to true when it knows compaction happened, and leaves it unset otherwise. An unset value means "unknown," not "no." That matters, because a constraint lost to compaction looks like a model ignoring an instruction.

Retries are hidden by design. The spec says a GenAI span covers the whole logical operation, every automatic retry included. A 40-second chat span may be three rate-limited attempts and one success. The HTTP conventions, which are stable, go the other way: one client span per attempt, and every resent attempt carries http.request.resend_count. If your HTTP client is instrumented, the retries sit one level down. If it is not, they are gone.

Attributes that cost more than they tell

Some of what the spec permits is weight you carry on every call and never read.

gen_ai.tool.definitions is the full list of tools, with schemas, available to the model. It is opt-in, and it is the same on every call made by the same agent build. Record it once per release, next to the prompt version, and point at it.

gen_ai.tool.description repeats per call too. The spec flags it as possibly sensitive. That is a reason to keep it out of a store every on-call engineer can read.

gen_ai.prompt.variable.<key> records each template variable as its own attribute. It is opt-in and flagged as sensitive. A template with many variables also eats into the SDK's default limit of 128 attributes per span (OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT). The attributes in the table share those slots.

A made-up conversation id is worse than none. The spec says that when no real identifier exists, instrumentation should not invent one: no new UUID, no trace id, no hash of the request. A fake id joins nothing to anything, and it looks like it should.

The bill for prompts and completions

The conventions name the content attributes gen_ai.system_instructions, gen_ai.input.messages and gen_ai.output.messages. On tool spans the matching pair is gen_ai.tool.call.arguments and gen_ai.tool.call.result. All of them are opt-in. The spec says instrumentations should not capture them by default, and it gives three patterns: record nothing, record content on the spans, or store it elsewhere and put a reference on the span. It calls the second best suited to manageable volume where privacy rules don't apply or the storage complies with them, pre-production for example. It recommends the third for production when volume or sensitive data is a concern.

Take the third.

Size. The spec warns that content can exceed a backend's limit on attribute values or on a whole telemetry payload. The SDK's default attribute length is "no limit." Nothing stops a span from carrying a long transcript until the backend rejects it. Agent runs make this worse, because every chat call resends the history. Suppose refund-agent makes 12 chat calls in a run. Its context grows evenly from 2,000 tokens on the first call to 10,000 on the twelfth. Recording gen_ai.input.messages on every span stores about 72,000 tokens of input for a conversation whose final context is 10,000. That is more than seven copies of one conversation.

Personal data. The spec marks tool arguments and results as possibly sensitive. For refund-agent, the result of lookup_order is a customer's name, address and order history. Your trace backend is readable by everyone on call, because on-call needs that. Customer data needs the opposite. The spec makes the same point: external storage "enables separate access controls."

Retention. Operational telemetry and customer content have different owners and different deletion rules. A separate content store keyed by gen_ai.response.id can honor a deletion request without rewriting a trace. It can keep content for as long as your data owner allows, while traces live on whatever schedule the backend sells.

The spec does not yet say how to record the reference. Its text on this reads "TODO: document a common approach." The app.content.ref attribute in the example is our own name, outside the gen_ai namespace, because a convention may still arrive.

Whatever share of content you keep, keep all of one kind. For tools that change the world, such as issue_refund, capture the arguments on every call and write them to the content store. They are short, and in the postmortem they are the one field that tells you which order the agent refunded. The cost is a second copy of personal data. Set its retention in the same review that turns it on.

Sampling cannot drop the run that went wrong

The OpenTelemetry SDK's default sampler is parentbased_always_on, which keeps every trace unless an upstream service has already sampled it out. Sampling is a choice you add on top of that, and the choice decides what survives.

Head sampling decides at the start of a trace. OpenTelemetry's own documentation names its weakness: with head sampling alone, you cannot make sure every trace with an error is kept. Tail sampling decides after it has seen most of a trace. Its example policies include keeping every trace with an error and sampling on the value of an attribute. By the same documentation's account it is also hard to run. The components are stateful and must hold a large amount of data; at high volume they can need dozens or hundreds of nodes.

Now the agent problem. Suppose lookup_order returns two orders for the customer, and the model refunds the older one. issue_refund succeeds. The last chat call ends with finish_reasons of ["stop"]. Every span ends without an error, as in the console output. A policy that keeps errors and slow traces drops this run, because by every signal the sampler can see, the run succeeded.

An agent incident does not need an exception, so a sampler that looks for one is the wrong filter. Two changes close the gap:

1. Take content off the spans, then keep every structural span you can afford. Without prompts and completions they hold names, ids, counts and versions, and they are the index you search when a customer complains.
2. Where you still sample, add attribute policies for the tools that change things. If a trace contains an execute_tool span whose gen_ai.tool.name is issue_refund, keep it, whatever its status.

Content can follow its own rule. Where instrumentation supports a content-upload hook, the spec says to call it "regardless of the span sampling decision." The content store then holds content for runs whose traces you dropped. That helps only if you can find the content without the trace. Key it by something a support ticket carries: the conversation id, or the order number.

Cardinality belongs on spans, not on metrics

gen_ai.conversation.id, gen_ai.response.id and gen_ai.tool.call.id take a new value on every run or call, which is what spans are built to carry. Put any of them on a metric, and the Prometheus documentation's warning applies: "every unique combination of key-value label pairs represents a new time series." Its examples of what not to use as labels are user IDs, email addresses, and "other unbounded sets of values."

The GenAI conventions follow the same line. The fetch_response span leaves the response id out of its name because the id is high cardinality. gen_ai.workflow.name must have low cardinality, and the spec tells instrumentation not to capture it by default when no meaningful low-cardinality name exists. The agent metrics, gen_ai.invoke_agent.inference_calls and gen_ai.invoke_agent.tool_calls, are histograms with buckets from 1 to 128, and the spec lists one attribute for them: gen_ai.agent.name.

We would add one more: the prompt version. It is bounded if you retire old versions. Tool calls per run, split by prompt version, is the chart where a bad prompt edit shows up before anyone writes a postmortem. That is our recommendation, not the spec's. It costs one time series per live version per agent.

Monday

Open the trace of the last agent run someone complained about. Go down the table in the second section, row by row, and write down each question you cannot answer from the trace alone. Each one is an attribute to add, starting with gen_ai.prompt.version.

Then read your sampler configuration. Would it keep a run that refunded the wrong order and raised no error? If not, take content off the spans, keep the structural spans for every run, and add a tail policy for the tools that change things. Decide where prompts and completions live, and who can read them, before an incident decides for you.