16  Observability with OpenTelemetry

Part IV · Portability in production

Implemented reference revision ce47fe2 · OpenTelemetry API 1.9.0 · SDK/context 2.12.0 · deterministic evidence SHA-256 2fbaf4d…0119

Tool progress explains bounded activity to a user. Telemetry explains system behavior to operators. The implemented profile traces the Northstar execution path without exporting prompts, documents, image bytes, generated observations, Python code, tool values, credentials, or raw provider errors.

A trace connects causally related operations. A span records one bounded operation under that trace. Neither is an authorization record, an effect receipt, a factual-support judgment, or permission to retain agent content.

16.1 Why agent telemetry needs its own contract

An agent request is not one provider call. The HTTP boundary admits files, a coordinator can make several model rounds, each round can propose tools, policy can reject or approve a proposal, execution can wait in a queue, and a server-owned tool can call another provider. Cancellation can occur at every boundary.

Without correlation, operators see unrelated durations and error counters. With unrestricted correlation, telemetry becomes a second copy of the most sensitive data in the system. The design therefore needs both a topology and a data-minimization contract.

OpenTelemetry supplies vendor-neutral APIs and SDK concepts for traces, metrics, logs, context, processors, and exporters (“OpenTelemetry Documentation,” n.d.). Its existence does not decide which spans an agent needs or which attributes are safe. Emerging generative-AI semantic conventions can improve vocabulary consistency, but they do not require prompts or completions to be exported and do not replace application-owned policy (“OpenTelemetry Generative AI Semantic Conventions, Revision 06ec68e,” n.d.).

16.2 Implemented trace topology

The reference emits fixed span names:

agent.http.request
├── agent.ingestion
└── agent.model.invocation
    ├── agent.model.round
    ├── agent.tool.proposal
    ├── agent.tool.policy
    └── agent.tool.execution
        ├── agent.provider.nested
        ├── agent.worker.queue
        └── agent.worker.execution

Actual sibling order depends on the run. A search or image operation creates agent.provider.nested; local Python creates queue and worker spans. A rejected proposal can stop at policy. An input failure can stop after ingestion. Span absence is meaningful only under the named instrumentation revision—it is not proof that no remote work happened outside the observed boundary.

16.2.1 HTTP invocation

agent.http.request begins after the handler receives the request and remains open through JSON completion or streamed SSE completion/cancellation. It records a fixed operation category, reviewed provider/model label when admitted, method, loopback server class, bounded status, and terminal outcome.

The span excludes request/response IDs and body content. For SSE, it ends in the stream’s terminal cleanup, not when the server first returns a Response object. A consumer cancellation ends it as cancelled. A server deadline ends it as timeout. Local observation cannot prove provider rollback.

16.2.2 Ingestion

agent.ingestion surrounds ordered input admission and shared document/image parsing. It records the operation class only. Filenames, paths, source IDs, media bodies, extracted text, digests, OCR output, and parser errors are excluded.

A completed span means the admission function returned. It does not certify document truth or parser fidelity. Format-specific limits and extraction evidence remain Chapter 4’s responsibility.

16.2.3 Coordinator and model rounds

agent.model.invocation covers the provider-neutral coordinator operation. agent.model.round uses the AI SDK’s model-call start/end hooks so each provider request is separated from later local tool execution.

Attributes retain reviewed provider and model labels plus a bounded chat operation. Prompts, standardized messages, tool schemas, provider request/response IDs, completions, reasoning content, usage payloads, and raw errors are not exported by the profile.

A multi-tool run can have several rounds under one invocation. This distinguishes “the coordinator was slow” from “the tool queue was slow” without copying the conversation into the trace.

16.2.4 Tool proposal, policy, and execution

The policy runtime creates three spans:

• agent.tool.proposal records that a named reviewed tool was proposed;
• agent.tool.policy covers schema validation, current authorization, approval, and cancellation checkpoints; and
• agent.tool.execution covers the compiled executor.

Only the reviewed tool name and bounded phase/outcome category are retained. Call IDs, input arguments, output values, approval text, principal IDs, tenant IDs, source IDs, artifact IDs, and error messages are excluded.

This separation prevents a common operational error: a policy denial must not be graphed as provider failure, and an invalid tool result must not be called a model refusal. It also preserves Chapter 8’s authority rule—an observer cannot approve, retry, mutate, or fail the operation.

16.2.5 Queue and worker boundaries

The Pyodide path emits agent.worker.queue while waiting for the process-wide slot and agent.worker.execution around worker construction/termination. The host creates these spans. The WebAssembly worker receives no tracer, exporter, prompt, credential, or trace context.

That boundary records queueing and local execution separately while avoiding an instrumentation bridge into generated code. Cancellation while queued ends the queue span without constructing a worker. Deadline expiry terminates the worker and records an error/timeout at the surrounding policy boundary.

16.2.6 Nested providers

Hosted search and image understanding emit agent.provider.nested around their fixed OpenAI recipients. The parent request can have an Anthropic coordinator. The trace therefore makes mixed-recipient execution visible without storing the search query, image, focused question, generated note, or observation.

A nested span is not a provider billing receipt or complete census of provider internals. It shows that the application invoked its reviewed bridge and observed a terminal local outcome.

16.3 Explicit context propagation

The implementation uses exact dependencies:

@opentelemetry/api                 1.9.0
@opentelemetry/context-async-hooks 2.12.0
@opentelemetry/sdk-trace-base      2.12.0

A trusted AgentTraceContext contains the telemetry implementation and OpenTelemetry context. It is created by the server, propagated through the model request and policy runtime, and never populated from HTTP body fields. The policy runtime creates child spans explicitly. Queue and worker spans inherit the trusted parent on the host side.

Async-local context is enabled for deterministic execution, but important boundaries also pass parent context explicitly. This prevents a callback scheduled outside an active lexical scope from silently becoming an unrelated root.

No W3C trace header is accepted from callers in the current loopback profile. Cross-service propagation would require authenticated ingress, baggage policy, trust-domain decisions, and header size/cardinality limits. Blindly accepting external parent IDs could let an untrusted caller join or fragment another trace.

16.4 Closed attribute policy

The implementation admits only a fixed key allowlist. String values must match a small token profile and at most 128 characters. Unknown keys and unsafe values are dropped.

Admitted dimensions include:

• operation category;
• terminal outcome;
• reviewed provider/model labels;
• reviewed tool name;
• tool phase;
• cancellation source category;
• worker kind and queue state;
• bounded HTTP status; and
• selected generative-AI operation/provider/model labels.

Excluded by default:

• prompts, instructions, messages, and completions;
• source/document bodies, quotes, URLs, filenames, and paths;
• image bytes, questions, and generated observations;
• tool arguments and results;
• Python code, stdout, stderr, and expression results;
• credentials, headers, cookies, and endpoint query data;
• raw provider errors and response bodies;
• request, response, source, call, artifact, principal, and tenant identifiers.

This is stronger than “do not log secrets.” Many excluded values are sensitive or high-cardinality even when not credentials. Digests and opaque IDs can still reveal membership or permit correlation, so the default profile omits them.

16.5 Errors, cancellation, and unknown effects

Spans use bounded outcomes: ok, error, cancelled, timeout, and unknown. Error attributes contain a normalized type/category, never Error.message, stack, raw abort reason, provider body, or generated tool output.

Cancellation records local control flow. It does not say:

• the provider did no work;
• a remote tool had no effect;
• billing did not occur;
• an artifact was not created; or
• retry is safe.

Chapter 15’s effect knowledge remains authoritative. Telemetry can point operators to a reconciliation path; it cannot replace the operation ledger or receipt.

Provider failures end model and HTTP spans with safe error categories. Tool failures preserve whether policy rejected before execution or whether execution began. A timeout after execution starts remains compatible with outcome-unknown at the recovery layer.

16.6 Concurrency

Concurrent HTTP invocations receive distinct root traces. Tests start two model/tool runs together and verify that child spans remain under the correct root. They do not place request IDs or prompts in attributes to achieve that isolation; OpenTelemetry span context performs correlation.

The test also checks controlled cardinality. Tool names come from the static registry, providers/models from reviewed routes, and outcomes/phases from finite enums. A future dynamic MCP or A2A destination must not become an unrestricted span value merely because it is useful for debugging.

16.7 Deterministic exporter and evidence

The checked-in profile uses BasicTracerProvider, SimpleSpanProcessor, and InMemorySpanExporter. This is suitable for deterministic tests, not production throughput.

The evidence command runs under the repository’s network-denial preload:

bun test test/telemetry.test.ts
bun run evidence:telemetry

The generated artifact:

evidence/observability/2026-10-23-opentelemetry-sanitized-trace.json
SHA-256 2fbaf4d669a66e74c7680d449bb1b0e58c5155b74e9d627ad05e75f027cf0119

normalizes trace/span IDs to local aliases and removes timestamps and durations. Two consecutive generations produce identical bytes. It records dependency versions, topology, redaction defaults, test coverage, and limitations.

The integrated tests cover:

• successful HTTP/ingestion/model/round/tool/nested hierarchy;
• concurrent trace separation;
• cancellation before execution;
• whole-tool timeout;
• Pyodide queueing and non-cooperative worker termination;
• hosted-search and image-recipient nested calls;
• provider failure; and
• absence of prompts, search terms, source bodies, image observations, Python code/output, credentials, and answer text.

The artifact is sanitized deterministic evidence, not a live provider trace or collector interoperability result.

16.8 Production exporter gate

The reference intentionally does not configure OTLP. Exporting traces creates another data recipient and failure path. A production profile requires review of:

• collector endpoint and authentication;
• TLS and network egress;
• sampling before expensive attribute construction;
• batching, queue bounds, backpressure, and drop behavior;
• exporter timeout and shutdown;
• tenant separation and access control;
• retention, deletion, backup, and legal policy;
• field-level redaction tests at the collector and backend;
• cardinality/cost budgets; and
• incident handling when telemetry is unavailable.

Telemetry failure must not silently authorize work. Whether it should fail a production request is a product policy decision; the educational profile treats observation as best effort and keeps operation semantics independent.

16.9 Metrics and logs

This implementation supplies traces only. Metrics and logs remain proposed.

A future low-cardinality metric set can count admitted requests, model rounds, policy outcomes, tool execution categories, queue delay buckets, cancellation sources, and unknown-effect reconciliation. It must not label by raw source URL, prompt, filename, call ID, trace ID, user text, or exception message.

Structured logs should use the same safe category vocabulary and access controls. Copying complete spans into logs defeats the trace redaction policy. Trace/log correlation is useful only when identifiers and retention remain bounded and authorized.

16.10 Progress is not telemetry

Pi progress cards show a user what the current operation is doing. They can contain approved query previews or source links that are explicitly excluded from operator telemetry. OpenTelemetry spans can outlive a UI connection and can be exported to another backend.

Therefore:

• progress never authorizes or executes tools;
• telemetry never becomes model context;
• a progress card is not a span receipt;
• a successful span is not a user-visible completion guarantee; and
• redaction rules differ because recipients and purposes differ.

16.11 Failure modes

Prompt-as-attribute. Convenient debugging turns the collector into a shadow conversation database. Export categories, not content.

Every ID as correlation. Request/source/artifact IDs create high cardinality and membership leakage. Use trace context for causal correlation and retain business IDs only in separately protected records.

Span success as task success. A provider call completes but the answer is unsupported. Keep semantic evaluation outside telemetry.

Cancellation as rollback. A cancelled span triggers automatic retry. Consult effect knowledge and reconciliation first.

Exporter authority. An exporter error changes tool policy or approval. Observation must not grant authority.

Untrusted parent context. Caller headers join internal traces across tenants. Authenticate and constrain propagation before accepting it.

16.12 Pattern summary

Problem: Agent runs cross model, tool, worker, and provider boundaries, while ordinary tracing can leak the content it is meant to diagnose.

Pattern: Propagate trusted OpenTelemetry context across fixed operation spans, admit only closed low-cardinality attributes, normalize terminal categories, and test redaction with an in-memory exporter.

Result: Operators can distinguish ingestion, model, policy, execution, queue, worker, and secondary-provider behavior without receiving agent payloads.

Cost: Production exporters, metrics, logs, cross-process propagation, retention, sampling, and collector security still require separate implementation and evidence.

16.13 Exercises

1. Add authenticated inbound trace propagation. Define when a caller’s parent is accepted and how tenant isolation is preserved.
2. Design an OTLP batching policy whose full queue cannot block or authorize a tool call.
3. Propose three metrics for queue pressure without using request, model-output, or source identifiers as labels.
4. Construct a cancellation trace with an unknown remote effect. Explain which fact telemetry cannot supply.
5. Add a redaction regression for a secret split across two generated chunks without storing either chunk.

16.14 Chapter contract

• Scope: trace context, model/tool/ingestion/worker/nested-provider spans, terminal classification, redaction, and cardinality.
• Canonical claims: implemented tracing explains bounded execution topology but provides neither authority, semantic correctness, effect receipts, nor production exporter certification.
• Migration inputs: former telemetry taxonomy, current tool-progress contrast, reference instrumentation, and deterministic trace artifact.
• Evidence gates: exact dependency pins, in-memory exporter, hierarchy/concurrency/cancellation/error tests, redaction tests, and byte-stable sanitized evidence.
• Demonstration slots: deterministic sanitized trace only; no live production collector demonstration is claimed.
• Source-claim ledger: OpenTelemetry documentation informs terminology; implementation claims are tied to the exact dependency and reference revisions above.
• Exclusions: user-visible progress belongs to Chapter 8; recovery/effect knowledge belongs to Chapter 15; production security gates belong to Chapter 17.

“OpenTelemetry Documentation.” n.d. Accessed October 7, 2026. https://opentelemetry.io/docs/.

“OpenTelemetry Generative AI Semantic Conventions, Revision 06ec68e.” n.d. Accessed October 7, 2026. https://github.com/open-telemetry/semantic-conventions-genai/tree/06ec68e722c45a7218e23ea1bc1339fe4e21ecae/docs/gen-ai.