Own Your Agent's Traces: Vendor-Neutral Observability with OpenTelemetry

Open a trace of your agent and read what's in it. Not the shape, the content.

For a typical tool-using agent, one trace holds the user's message, the full system prompt, every argument the model passed to every tool, and every result those tools sent back. If the agent reads a database, the rows it read are in the trace. If a tool fails, the error message is in the trace, and error messages love to quote the value that broke them.

For Cameron, the personal finance agent I build in public, that means a single trace of "how much did I spend on groceries last month?" contains a slice of the owner's bank ledger. Cameron's whole design rests on one rule: all financial data stays on infrastructure the owner controls. And for months it shipped with tracing that, pointed at the default hosted backend, sent that ledger somewhere else.

Nobody did anything wrong. Cameron inherited the setup from the starter template it grew out of, the same setup I walked through in Observability for AI Agents with Langfuse. It's a good setup. It just answered a different question: how do I see what my agent did? This guide answers the next one: how do you trace an agent without giving the traces away?

The answer has three parts, and none of them is specific to finance or to Cameron:

1. Emit standard spans (OpenTelemetry's GenAI semantic conventions), so the backend is a choice and not a dependency.
2. Treat content as a separate opt-in, off by default, rather than something you try to scrub after the fact.
3. Run the backend yourself, next to the agent, and make swapping it a one-variable change.

Cameron is the worked example throughout, because it has the hardest version of the constraint. The code is LangChain/LangGraph in TypeScript, but the design carries to any stack.

What a trace actually contains

Here is what an instrumented agent run typically records, and where the sensitive data hides:

The top two rows are what you need to answer most operational questions: which step was slow, which model call was expensive, which tool failed, how often the approval gate fired. Everything below them is what you need to answer why the model decided what it decided, which is a debugging question, not a monitoring one.

That split is the whole design. Metadata is cheap, safe and always useful. Content is expensive, sensitive and occasionally essential. They deserve different defaults.

Here is what that looked like in Cameron's inherited Langfuse setup. One question about a subscription category, and the observation list already carries the user's message, the SQL the model wrote against the ledger, and the answer with its amounts:

Open the trace and the full answer is there, month by month:

One question, one slice of the ledger, stored on whatever server the endpoint points to. (The ledger in these screenshots is Cameron's test data.)

Why "just redact it" doesn't hold

The first instinct is to keep capturing everything and add a redaction step: a span processor that strips amounts, account numbers and merchant names before export.

It's worth being precise about why that's the weaker design.

• Redaction is a denylist. You have to enumerate every shape sensitive data can take, in every tool, forever. A new tool that returns a free-text memo field is a leak until someone remembers to add a rule.
• Content shows up in unexpected attributes. Error messages, exception events, finish reasons on a refused generation, a tool's string-serialized return value. The redactor has to know about all of them.
• It fails open. A bug in a regex sends the data. A bug in "don't attach content" sends nothing.

The stronger default is the inverse: an allowlist. The instrumentation sets a small, named set of metadata attributes, and content attributes are only ever attached behind one flag. When the flag is off, there is nothing to redact because the data never entered the span.

OpenTelemetry already agrees with this. Its GenAI conventions treat message and tool content as opt-in, and the instrumentations that follow the spec gate it behind an environment variable, OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT. Cameron reuses that exact variable, so the switch means the same thing in its code as it does in any other OTel-instrumented service.

The standard: OpenTelemetry GenAI conventions in five minutes

If you've used OpenTelemetry for services, the model is unchanged: a trace is a tree of spans, each span has a name, a kind, a start and end, and a bag of attributes. What the GenAI semantic conventions add is an agreed vocabulary for what goes in an agent's spans.

For an agent, three operations cover almost everything:

And a small set of attributes carries the metadata:

• gen_ai.operation.name, gen_ai.agent.name, gen_ai.conversation.id
• gen_ai.provider.name, gen_ai.request.model, gen_ai.response.model
• gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.response.finish_reasons
• gen_ai.tool.name, gen_ai.tool.call.id, gen_ai.tool.type

The content attributes are separate and opt-in: gen_ai.system_instructions, gen_ai.input.messages, gen_ai.output.messages, gen_ai.tool.call.arguments, gen_ai.tool.call.result.

Put together, one agent turn looks like this:

Why bother with a standard at all, rather than whatever your favorite backend's SDK emits? Because the backends converged on it. Phoenix converts gen_ai.* attributes to its own format at ingest (since 15.10), Langfuse and Datadog accept them over OTLP, and generic tracers like Jaeger show them as plain attributes. Emit the standard once and the backend becomes a deployment decision.

One honest caveat. The GenAI conventions are still at Development stability. In mid-2026 they moved out of the main semantic-conventions repository into their own, and attribute names can still change. Build on them, but keep every attribute name in one file so a rename is a one-file change.

Instrumenting the agent

Cameron is a LangGraph agent in TypeScript, but its tracing comes down to three pieces you'd build the same way on any stack: two switches read from the environment, the OpenTelemetry SDK with a plain OTLP exporter, and a handler that turns an agent run into spans. feat(tracing): own the traces with OpenTelemetry instead of Langfuse

Two switches, both off by default

type Env = Record<string, string | undefined>

/** Tracing is on only when the owner names an OTLP endpoint; there is no default destination. */
export function tracingEnabled(env: Env = process.env): boolean {
  if (env.OTEL_SDK_DISABLED === 'true') return false
  return Boolean(env.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT || env.OTEL_EXPORTER_OTLP_ENDPOINT)
}

/** Prompts, messages, tool arguments and results leave the process only when this is "true". */
export function captureContent(env: Env = process.env): boolean {
  return env.OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT === 'true'
}

There is no default endpoint in the code. A vendor SDK usually ships one, which is exactly how traces end up on someone else's servers without anyone deciding they should. Here, tracing does nothing until you name a destination, and naming a destination still sends metadata only.

Every variable is a standard OpenTelemetry one, read the same way by the OTel SDKs in every language. That's also why starting the SDK needs no code worth showing: Cameron's only change from the Langfuse version was swapping LangfuseSpanProcessor for a plain OTLPTraceExporter, which reads its endpoint and headers from those same variables. The destination moved from code into configuration.

Turning a run into spans

The SDK gives you HTTP spans for free. It knows nothing about agents. For that you hook whatever your framework exposes for observing a run (callbacks, middleware, an event stream) and map it onto the three GenAI operations: the run becomes invoke_agent, each model call chat, each tool call execute_tool. In Cameron that hook is a LangChain callback handler, attached to every run while tracing is on.

What matters is not the mapping but where content enters it. Here is the tool span, stripped of the framework plumbing:

function toolSpanAttributes(tool: ToolCall, capture: boolean): Attributes {
  return {
    [GEN_AI.OPERATION_NAME]: 'execute_tool',
    [GEN_AI.TOOL_NAME]: tool.name,
    [GEN_AI.TOOL_CALL_ID]: tool.id,
    [CAMERON.TOOL_MUTATING]: isMutatingTool(tool.name),
    ...(capture ? { [GEN_AI.TOOL_CALL_ARGUMENTS]: tool.args } : {}),
  }
}

The arguments are attached in exactly one place, behind exactly one flag. The same pattern repeats for messages on chat spans and results on execute_tool spans. Reviewing the handler for leaks means reading every capture branch, which is a short list.

Errors get the same treatment, because they're the leak people forget:

function fail(span: Span, err: Error, capture: boolean) {
  span.setAttribute('error.type', err.name || 'Error')
  // An error message can quote a tool's arguments, so it is content too.
  span.setStatus({ code: SpanStatusCode.ERROR, ...(capture ? { message: err.message } : {}) })
  if (capture) span.recordException(err)
}

With content off, a failed span still says that it failed and what kind of error it was. It doesn't say which amount broke it.

Domain attributes go in your own namespace

The GenAI conventions don't know your agent's semantics, and they shouldn't. Cameron's most important behavior is its approval gate: any tool that mutates financial data pauses the run until the owner approves. That's exactly what you want to see in a trace, so Cameron adds a few attributes under its own cameron.* prefix:

• cameron.tool.mutating on every tool span, so you can filter for every attempt to change data.
• A cameron.approval.requested event on the agent span when the gate pauses a run, listing the gated tool names (never their arguments).
• cameron.run.resumed plus approved and rejected counts when the owner's decision resumes the run.

Two design notes worth stealing:

• A pause is not an error. If your framework implements a human-in-the-loop pause by throwing, as LangGraph does, a naive handler records every approval request as a failed span and your error rate fills with noise. Catch the pause and record an event instead.
• Keep your names in the same file as the standard ones. Cameron's semconv.ts holds both the gen_ai.* and the cameron.* names. One place to audit what leaves the process, and one place to absorb a rename.

The implementation notes in Cameron's OBSERVABILITY.md cover the LangChain.js and Next.js specifics this section leaves out.

Your framework may already emit these spans

You may not need to write a handler at all. A growing number of agent frameworks emit GenAI spans themselves: Strands Agents and Pydantic AI produce invoke_agent, chat and execute_tool spans out of the box, and the Vercel AI SDK is moving the same way. Instrumentation libraries like OpenLLMetry and OpenInference cover many of the rest. LangChain.js, at the time of writing, is not one of them, which is why Cameron has its own.

If you go that route, the design above becomes a checklist rather than code. Check the content default. Pydantic AI, for one, includes message content unless you turn it off, which is the opposite of the rule this guide argues for. Check what an error span carries, and whether a pause or an approval shows up as a failure.

Running the backend yourself

With standard spans going out over OTLP, the backend is a container. Cameron defaults to Arize Phoenix, which understands the GenAI conventions natively and runs as a single image with no external database:

# compose.yaml (dev profile)
phoenix:
  profiles: [tracing]
  image: arizephoenix/phoenix:version-20.23.0-nonroot
  environment:
    PHOENIX_TELEMETRY_ENABLED: 'false'
  ports:
    # Loopback only: Phoenix has no login, and traces describe your finances.
    - '127.0.0.1:${PHOENIX_PORT:-6006}:6006'
  volumes:
    - phoenix_data:/home/nonroot

Three lines in there matter more than they look:

• 127.0.0.1: on the port. Phoenix has no login by default. Publishing its port on all interfaces would put your traces on your network.
• PHOENIX_TELEMETRY_ENABLED: 'false'. Self-hosting is pointless if the tool phones home with usage data. Check this for whatever backend you pick.
• A named volume, separate per environment. Cameron's production stack runs its own phoenix-prod with its own volume, so tearing down the dev stack can't touch production traces. Traces of real data deserve the same care as the real data.

Then point the agent at it with one variable:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:6006

Send a message and open http://localhost:6006. Each conversation thread shows up as a Phoenix session (that's what gen_ai.conversation.id buys you), so an approval pause and its resume sit side by side.

With content capture off, which is the default, a chat span still tells you the model, the provider, the token counts and why the call finished. Its input is just the model name:

Everything you need to see that it worked, nothing about what it read. Turn capture on and the same span carries the whole conversation: the system prompt, the user's question, and every tool result the model saw.

One quirk worth knowing: if your agent runs behind an HTTP route, the trace list shows that request span on top, not the agent. In Phoenix, filter by span_kind == 'AGENT' to list agent runs alone. Agent and HTTP spans sharing one trace is deliberate: a slow response is usually something you want to correlate across both.

Swapping the backend is one variable

This is the payoff of emitting the standard instead of a vendor format. To send the same spans to Jaeger:

docker run -p 127.0.0.1:16686:16686 -p 127.0.0.1:4318:4318 jaegertracing/jaeger:2.11.0
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318

To a self-hosted Langfuse:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:3000/api/public/otel
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic%20<base64 of pk-lf-…:sk-lf-…>

No code changes, no new dependency. Jaeger shows the same tree with GenAI attributes as raw key-value pairs; Phoenix and Langfuse render them as LLM-aware views with messages and token counts.

Here is a Cameron run in Jaeger, with no code changed:

Notice the two fetch POST spans next to invoke_agent rather than under the chat spans that made them. The HTTP instrumentation records the model's network calls in the request's context, and a framework hook can open spans but can't make them the parent of code it doesn't call. They're still in the same trace, just one level up.

A hosted backend works the same way, and that's fine for plenty of agents. Just make it a decision rather than a default, and if the agent touches sensitive data, keep content capture off when the destination isn't yours.

The eval trade-off

Many backends can run evals on your traces: an LLM judge scores each production run as it arrives (online evals), or a dataset built from past traces is replayed against a new prompt or model (offline evals). Both need the content. A metadata-only trace tells the judge that query_transactions ran and took 120ms; it can't say whether the answer was right, because the answer was never sent.

That's the real cost of keeping content out, and the two kinds of eval pay it differently:

• Online evals have no workaround. If the judge runs inside the backend, the backend needs the prompt and the response. You either turn content capture on, for everything or a sample, or you run online evals somewhere else.
• Offline evals can rebuild what the trace left out, as long as the agent stores its conversations somewhere you own. The trace keeps gen_ai.conversation.id, so a failing or slow run in the trace view points straight at the conversation it came from.

Cameron is the second case: the LangGraph checkpointer already keeps every conversation in its own Postgres, its evals run offline, and a one-user agent doesn't need online ones. Nothing is lost.

An agent with many users, or no stored conversations, may need online evals. With a self-hosted backend, turning content on is a fair trade, as long as you treat the trace store as a second copy of the data.

What you give up

Owning the traces isn't free. Being honest about the trade:

• You write and maintain the handler. The Langfuse integration was two lines; Cameron's handler is about 300, plus tests. If your framework emits GenAI spans natively, you skip this. LangChain.js doesn't yet, at least not in the standard vocabulary.
• You lose the vendor's semantic depth. A first-party callback knows things a generic handler doesn't bother with. Cameron deliberately gives graph nodes no span.
• Platform features don't ride on OTLP. User feedback is the clearest example. A thumbs up arrives after the trace has closed, and OpenTelemetry has no way to say "the user rated this run." Langfuse gives you a scores API that ties feedback to the trace and the message; Phoenix has span and trace annotations. With plain OTLP you either build that API yourself, keyed on the trace id, or use your backend's and accept the coupling. Prompt management and eval UIs are the same story.
• The conventions will move. Development stability means renames. One file absorbs them, but someone has to notice.
• You run another service. Backups, upgrades and retention are yours now.

Own what earns its keep

Owning your traces is not the goal. Traces are worth what you do with them, and a trace store you own but can't learn from is just a liability you host yourself. The real question is which trade-offs a given agent can live with, and there are three honest answers:

• Standard spans, content off. Cameron. The data is someone's finances, and no feature is worth sending it out.
• A self-hosted platform, features included. The data stays yours, and you accept coupling to one backend for things like feedback and online evals. The middle ground for most agents.
• A hosted vendor. The Agentailor agent on this blog runs on Langfuse Cloud. It answers questions about public articles, so its data isn't sensitive, feedback works out of the box, and the vendor's retention policy covers the rest.

Two questions decide it: what happens if this data lives on someone else's servers, and which features would you otherwise have to build?

Even Cameron's answer isn't absolute. If it needed feedback tomorrow, I'd use Phoenix's annotations before writing my own API. Coupling to a self-hosted backend is a choice you can undo later. Data that already left your infrastructure can't be pulled back.

Key Takeaways

• A trace is a copy of your agent's data. Prompts, tool arguments, tool results and error messages are all content, and tool results are usually the largest share.
• Allowlist, don't redact. Attach content in one place behind one flag (OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT) and leave it off by default. Redaction is a fallback, not a design.
• No default destination. Tracing should do nothing until someone names an endpoint, and naming one should still send metadata only.
• Emit the standard. invoke_agent, chat and execute_tool with gen_ai.* attributes make the backend a one-variable swap. Keep the names in one file, because the conventions are still moving.
• Put domain semantics in your own namespace. Side effects, approvals and pauses are what you actually debug. A pause is an event, not an error.
• Self-host carefully. Loopback-only ports, vendor telemetry off, separate volumes per environment.
• Owning isn't the goal. Pick the trade-offs your agent can live with. For most agents, a self-hosted platform is the middle ground.

What to Read Next

• Observability for AI Agents: Why Tracing Matters and How to Do It with Langfuse — where Cameron's tracing started, and why semantic traces matter in the first place.
• Building an Agent Eval Harness: The Complete Guide — traces show what happened; evals tell you whether it was right.
• Meet Cameron — the agent behind the examples, built in public.

Resources

• OpenTelemetry GenAI semantic conventions
• GenAI agent and framework spans
• Phoenix: OTel GenAI semconv conversion (release note)
• Cameron's OBSERVABILITY.md