No Trace. No Truth.

The dashboard says the model was healthy.

The user says the task failed.

Both can be true.

An agent can receive a fast 200 response from every model call, produce valid JSON, invoke every tool without an HTTP error, and still change the wrong repository. Component health is not outcome truth.

The missing object is an outcome trace: one causal account of what the system believed, observed, proposed, authorized, changed, checked, and finally proved.

Logs, metrics, and traces answer different questions

A log records an event. A metric aggregates measurements. A trace connects operations that belong to one causal path.

For a software agent:

Signal	Useful question	Blind spot
Log	What did this component report?	Which task and decision caused it?
Metric	Are cost, errors, or latency changing?	What happened in one bad run?
Trace	Which operations formed this run?	What did a model decision mean?
Durable history	Which verified state transitions occurred?	May be too detailed for fleet analysis
Evaluation	Was the outcome correct and valuable?	Often lacks production causality

Dapper describes a tracing infrastructure designed for low overhead and broad deployment across large distributed services. X-Trace similarly argues for reconstructing behavior across layers and administrative components. Agent systems inherit that distributed-systems problem, then add probabilistic decisions, retrieved evidence, mutable context, and external action.

A model span alone is equivalent to tracing a database call but ignoring the transaction.

The trace root is the task, not the chat turn

Long-running automation outlives HTTP requests and model sessions. One task can create many traces as workers stop and resume. One chat turn can start several durable tasks. Choose stable identities deliberately:

task_id       business unit of work
run_id        one end-to-end attempt or resume generation
trace_id      one propagated causal execution graph
span_id       one operation
transition_id one logical state mutation
artifact_id   immutable output or evidence object

The W3C Trace Context Recommendation standardizes HTTP headers for propagating trace identity across services. Use that substrate where it fits. Do not force a days-long workflow into one open in-memory span. Link resumed traces to the same task and prior transition.

The task record answers “what eventually happened?” The trace graph answers “how did this execution get there?” They should link, not compete.

Give semantic steps their own spans

Generic names such as llm.call and tool.call are insufficient. Two model calls may serve radically different roles. One plans. One extracts typed arguments. One judges evidence. One compresses history. Their latency and tokens are comparable; their failure semantics are not.

A useful agent trace has spans such as:

task.run
  state.load
  context.assemble
    retrieval.query
    retrieval.rerank
    policy.tools_eligible
  inference.decide
  action.validate
    schema.check
    authorization.check
    budget.check
  tool.execute
  effect.reconcile
  state.reduce
  verifier.run
  outcome.commit

Each span should carry bounded attributes:

• task, tenant, workflow, prompt, model, tool, policy, and verifier versions;
• input and output token counts using the provider's actual accounting when available;
• cache class and hit status;
• queue, time-to-first-token, decode, tool, and verification latency;
• transition source and destination states;
• authority decision and reason code;
• artifact hashes and storage references;
• retry, cancellation, and timeout classification;
• terminal outcome and evidence class.

The OpenTelemetry specification provides vendor-neutral concepts for traces, metrics, and logs, and its semantic conventions define shared naming patterns. Generative-AI conventions are evolving. Pin the schema version you emit and preserve your own domain attributes for workflow state and verified outcomes.

Do not put secrets into observability

The easiest tracing implementation stores every prompt, completion, retrieved document, tool argument, and result. It is also a fast way to build a second ungoverned data lake containing source code, credentials, personal data, contracts, and model-generated secrets.

Separate searchable metadata from protected payloads:

trace attribute: artifact.hash = sha256:...
trace attribute: artifact.class = repository_patch
trace attribute: data.policy = confidential
trace attribute: payload.capture = reference_only
artifact store: encrypted object with tenant policy and retention

Apply allowlists, not denylists. Redact before export. Hashes prove identity but do not make low-entropy secrets safe. A hash of “yes” or a four-digit PIN is easy to guess. Use opaque artifact IDs for sensitive small values.

Record the context assembly manifest rather than necessarily the entire context:

{
  "prompt_version": "planner.v17",
  "state_version": 82,
  "evidence": [
    {"artifact": "doc_91", "hash": "...", "range": "120-188"},
    {"artifact": "log_33", "hash": "...", "range": "1-74"}
  ],
  "tools": ["repo.read@4", "patch.apply@3", "check.run@8"]
}

This makes the input reconstructable by an authorized investigator without broadcasting it to every telemetry consumer.

Cost attribution needs a tree, not one counter

An agent's cost is distributed across decisions and consequences:

model inference
context retrieval and reranking
tool compute
sandbox time
artifact storage
verification
retries and abandoned branches
human exception handling
failure loss

Attribute cost to the task and the span that caused it. If an early retrieval miss triggers three wrong patches, the cost is not merely “three model calls.” The trace should expose the miss, the downstream retries, and the verifier failures.

For every run calculate:

useful cost = cost on the terminal accepted path
exploration cost = rejected but informative branches
waste cost = duplicated, cancelled too late, or repeated equivalent work
recovery cost = reconciliation, replay, and compensation

These labels require judgment. Store a reason code so teams can revise classification without rewriting raw measurements.

Sampling by request hides the failures you need

Uniform head sampling decides whether to keep a trace before the outcome is known. A 1 percent sample is attractive for cost, but a rare 0.1 percent policy failure then appears in only about one sampled trace per 100,000 runs in expectation:

100,000 runs × 0.1% failure × 1% sample = 1 retained failure trace

This is an illustrative probability calculation, not a traffic report. It shows why agent telemetry needs tail decisions after terminal attributes are available.

Retain traces when any of these is true:

• outcome failed or escalated;
• policy denied or a high-consequence tool was proposed;
• cost or latency crossed a slice-specific threshold;
• verifier disagreement occurred;
• recovery, compensation, or duplicate delivery occurred;
• the run represents a stratified success sample;
• the workflow, model, prompt, or tool version is newly deployed.

Keep a small unbiased sample too. Outcome-biased retention is excellent for debugging but cannot estimate fleet rates without sampling weights.

A trace is not automatically an explanation

Tracing records what the instrumented system exposed. It does not prove why a model chose an action. A chain-of-thought string is not a causal explanation and may be incomplete or misleading.

Prefer observable decision inputs and outputs:

• exact context manifest;
• available tool set;
• typed proposed action;
• candidate alternatives if generated;
• verifier scores and rule results;
• policy checks;
• model and sampling configuration;
• resulting environment observation.

Then run counterfactual replays. Hold the workflow constant and vary one factor: retrieved evidence, model, prompt version, tool availability, or verifier. If the outcome changes repeatedly, you have stronger attribution than a persuasive reflection.

Cardinality is an architecture limit

Telemetry systems index attributes so operators can filter and aggregate them. An unbounded value such as raw prompt text, user query, repository path, task ID, commit hash, or tool arguments can create a distinct time series or index entry for almost every request. Cost and query latency then grow for little analytic value.

Classify attributes by their intended operation:

Attribute class	Examples	Storage treatment
Fleet dimension	workflow, model, tool, outcome code	Indexed, bounded vocabulary
Investigation key	task ID, trace ID, artifact ID	Searchable lookup, not a metric label
High-cardinality fact	commit hash, document ID, user ID	Trace or event field with retention policy
Payload	prompt, completion, source, tool output	Governed artifact, normally not indexed
Secret	token, credential, private key	Never capture

A useful test is: “Will we group a population by this value?” If yes, define a bounded semantic convention. If the value only locates one incident, keep it on the trace. If it reconstructs the decision, store it as a protected artifact.

Version attribute names and enumerations. Changing outcome=success to status=verified without a migration splits dashboards and silently corrupts comparisons. Treat telemetry schema as an API consumed by alerts, evaluators, cost reports, and research jobs.

Build a replay bundle, not just a pretty waterfall

A trace UI is useful for scanning time. A failure investigation also needs a portable, immutable bundle:

manifest.json
  task, run, trace, workflow and policy versions
context/
  prompt template, state projection, evidence manifests
decisions/
  model configuration, typed proposals, selection results
effects/
  tool requests, receipts, idempotency identities
artifacts/
  patch, logs, tests, verifier evidence by content hash
outcome.json
  terminal state, reason, cost, latency, proof

The bundle should be sufficient for an authorized engineer to replay pure steps and simulate effecting steps against a sandbox. It should not contain live credentials or rely on mutable “latest” resources.

Replay has levels:

1. Structural replay: can the workflow reduce the history to the same typed state?
2. Input replay: can the exact context and tool eligibility be reconstructed?
3. Model replay: does the same model configuration reproduce or statistically resemble the decision?
4. Environment replay: do sandboxed tools observe equivalent state?
5. Outcome replay: does the verifier reach the same terminal judgment?

Bit-for-bit model output is often unavailable because serving kernels, model aliases, and sampling can change. That does not excuse unversioned evidence or tools. Record what is controllable and label what is not.

Define service objectives around terminal work

An ordinary API may target request availability and latency. An agent task needs outcome objectives with explicit clocks:

acceptance latency: accepted_at - submitted_at
active work latency: sum of intervals with a valid worker lease
external wait: CI + approval + rate-limit + dependency wait
time to terminal outcome: terminal_at - submitted_at
verification delay: verified_at - final_effect_at

Suppose 1,000 tasks have a 95 percent verified-success objective and a 30-minute terminal deadline. If 970 succeed but 70 complete after the deadline, success quality is 97 percent while on-time verified success is only 90 percent:

verified success rate = 970 / 1,000 = 97%
on-time verified success = (970 - 70) / 1,000 = 90%

The numbers are illustrative. The point is that a quality SLO without a deadline rewards eventually correct systems that users cannot depend on. A latency SLO without verification rewards fast wrong work.

Budget error by task slice and consequence. A documentation edit and a production credential rotation should not consume the same reliability budget. Report both the denominator and the exclusion rules so paused, cancelled, denied, and infeasible tasks do not disappear from the metric.

Worked example: one green fleet, one broken task

Assume an agent changes a dependency version. Every component metric is inside its service objective:

Span	Status	What happened
retrieval.query	OK	Returned an archived migration guide
inference.decide	OK	Proposed the old configuration key
schema.check	OK	Action JSON matched schema
authorization.check	OK	Repository write was permitted
patch.apply	OK	Patch committed in sandbox
check.run	OK	Unit suite passed
verifier.integration	FAIL	Staging rejected removed key

The model latency dashboard is green. The tool error dashboard is green. The schema-validity dashboard is green. The outcome failed because evidence freshness and verification coverage were wrong.

The trace supports concrete automation:

1. identify every failed task whose selected evidence version predates the target dependency;
2. replay them with current documentation;
3. add a freshness rule to context assembly;
4. add the staging configuration check before mutation;
5. compare cost per verified outcome before and after.

Without the outcome trace, the likely response is to change the model. With it, the team can fix the actual control boundary.

What to alert on

Do not page on every model error. Many are retried and never affect an outcome. Alert on customer or safety consequence:

• verified-success rate below the slice baseline;
• p95 time to terminal outcome above deadline;
• cost per verified outcome above budget;
• unknown side effects older than reconciliation SLO;
• high-consequence mutation without expected receipt;
• verifier disagreement or missing proof;
• repeated equivalent state or runaway branch growth;
• compensation backlog;
• trace continuity loss between authorized mutation and outcome.

Use component alerts to route diagnosis after the outcome signal fires.

Builder playbook

1. Define task_id, run_id, transition_id, and artifact_id before adding spans.
2. Propagate W3C trace context through synchronous boundaries and explicit links through queues and resumes.
3. Instrument model, context, policy, tool, state, verifier, and outcome operations separately.
4. Adopt versioned semantic conventions and test emitted schemas in CI.
5. Store large or sensitive payloads in a governed artifact store; trace references and hashes.
6. Record tokens, time, cost, cache, retry, cancellation, and result at the span that owns them.
7. Tail-sample failures, expensive runs, policy events, recovery, and new versions while keeping a weighted unbiased sample.
8. Make a trace replayable enough to reconstruct inputs and compare one controlled variation.
9. Join traces to offline evaluations and terminal business outcomes.
10. Run a monthly “green components, failed task” review to find missing semantic spans.

The test of observability is not whether a dashboard looks complete. It is whether an engineer can answer, with evidence:

What failed?
Where did the wrong state enter?
Which later costs did it cause?
Why did verification not stop it sooner?
Which change prevents recurrence?
Did that change improve verified outcomes?

If the trace cannot answer those questions, it recorded activity, not truth.

References

• Sigelman et al., “Dapper, a Large-Scale Distributed Systems Tracing Infrastructure,” Google, 2010. Primary design report on low-overhead distributed tracing at large scale.
• Fonseca et al., “X-Trace: A Pervasive Network Tracing Framework,” NSDI 2007. A cross-layer approach to reconstructing service behavior.
• W3C Trace Context Recommendation, 2021. The standard propagation format for distributed trace context over HTTP.
• OpenTelemetry Specification. Vendor-neutral telemetry concepts and APIs; implementations should pin the convention versions they emit.
• OpenTelemetry General Trace Semantic Conventions. Common attribution rules for trace spans across implementations.
• Yang et al., “SWE-agent,” 2024 preprint. Primary evidence that the agent-computer interface is a meaningful part of automated software behavior and therefore needs tracing.