# Observability

Trace AML evaluations, Agent turns, ACP updates, and process lifecycle without capturing content by default.
Canonical: https://agent-markup-language.com/docs/observability/
Documentation index: https://agent-markup-language.com/docs/
Complete documentation: https://agent-markup-language.com/docs/llms.txt

AML exposes two observability layers:

1. Runtime lifecycle events: `start` and `finish`, for request-level status and run identity.
2. Trace events: immutable spans and point events for the evaluation tree, Agent sessions and turns, ACP activity, capabilities, processes, and cleanup.

This page defines the portable trace contract. For listener timing and evaluation lifecycle decisions, use the [runtime reference](https://agent-markup-language.com/docs/reference/runtime/#errors-and-lifecycle-events).

## Start tracing

**SDK**

```tsx
import { AmlRuntime, createConsoleTracer } from "@aml-jsx/sdk"

const runtime = new AmlRuntime({
  trace: createConsoleTracer({ captureContent: false }),
  onTraceError: (error, event) => {
    console.error("AML trace sink failed", { error, sequence: event.sequence })
  },
})
```

The tracer writes a compact tree to `console.log`. Supply `write` to send complete lines to another destination:

```tsx
const trace = createConsoleTracer({
  captureContent: false,
  write: line => process.stderr.write(`${line}\n`),
})
```

**CLI**

```sh
aml run ./workflow.tsx --trace
```

The [experimental CLI](https://agent-markup-language.com/docs/cli/) writes metadata-only traces to standard error and keeps the workflow result on standard output.

```sh
# Explicitly includes supported sensitive content.
aml run ./workflow.tsx --trace --capture-content
```

`--capture-content` is an AML CLI option. It is not part of ACP and is not sent to the Agent.

Trace sinks are captured when the runtime is constructed. AML does not await a sink's returned promise. Synchronous throws and rejected promises go to `onTraceError` when configured; telemetry failure does not change the workflow result.

## Application-owned spans

Automatic component spans cover both a function component call and resolution of its returned subtree. They cannot
isolate a narrower deterministic phase such as validation, retrieval, or persistence. Measure that work with
`withTraceSpan()` inside an active function component:

```tsx
async function Review({ findings }: { findings: readonly Finding[] }) {
  return await withTraceSpan("review.validate", async () => await validateFindings(findings))
}
```

The runtime allocates an `application` span beneath the active component or enclosing application span. It closes the
span on success, thrown failure, or cancellation without changing the callback result. Calls outside an active
component, including detached calls after component settlement, fail. Nested calls remain correctly parented across
`evaluate()`, `<Parallel>`, and concurrent async work.

Applications supply only the span name and callback. They never supply trace identity, parent fields, metadata, or
content.

## Run summaries

`createTraceSummaryCollector()` derives content-free summaries from the same public trace stream:

```tsx
const summaries = createTraceSummaryCollector()
const runtime = new AmlRuntime({
  onTraceError: (error, event) => reportTraceFailure(error, event),
  trace: summaries.trace,
})

function summaryFor(runId: string) {
  const summary = summaries.forRun(runId)
  summaries.deleteRun(runId)
  return summary
}
```

`forRun()` requires an explicit evaluation `runId`; there is no latest-run API that becomes ambiguous when a runtime
handles overlapping evaluations. Capture that identity from the public `start` event using the request-correlation
pattern in the cookbook below. Completed summaries report evaluation status and wall duration, Agent session and turn
timing, Tool and resource timing, named application spans, ACP tool-call counts, provider-reported usage entries, and
Agent cleanup outcomes. `acpToolCalls.byName` lets an application check whether a supplied provider capability was
invoked without persisting raw ACP events. It counts only initial `tool_call` updates and retains only the exact public
provider-reported name and count—not arguments, results, later updates, prompts, or model text. Provider capability names
do not portably guarantee which backend or MCP server handled a call.

The separate `tools` timing aggregate measures declarative AML `<Tool>` executions. A provider call routed to an AML Tool
may correctly appear in both aggregates; AML does not deduplicate these different boundaries. Timing aggregates contain
`count`, summed `totalDurationMs`, and `slowestMs` when at least one span was observed. The collector does not rewrite
evaluation status. Trace-consumer failures remain separate on the runtime's existing `onTraceError` channel.

`providerUsage` is empty when the provider reports no usage. When ACP supplies usage, AML retains the serialized JSON
string rather than promising token fields that may not exist. A turn is one provider/ACP prompt request, not one model
call. Summaries do not infer provider retries, cache behavior, costs, or billing data. See [Correlate application spans and run
summaries](/docs/cookbook/application-observability/) for a concurrent production pattern.

## What an Agent trace represents

AML traces the lifecycle it owns and the events that cross its ACP connection. One ACP `session/prompt` request is one **Agent turn**. It is not necessarily one model API call.

```text
agent Agent
└─ agent.session                    one provider session
   ├─ sandbox.process               process lifecycle facts
   ├─ acp.session.created           ACP session established
   ├─ agent.turn kind=initial       first session/prompt request
   │  ├─ acp.session.prompt.submitted
   │  ├─ acp.session.update         zero or more streamed ACP updates
   │  └─ acp.session.prompt.completed
   ├─ agent.turn kind=follow-up     next authored <FollowUp />
   │  └─ ...
   └─ agent.cleanup                 session and process cleanup
```

Turn spans start immediately before provider execution and end before the next turn begins. A two-turn Agent therefore never reports turn two as started while turn one is still running. ACP updates are emitted in the order AML consumes them from `ActiveSession.nextUpdate()`.

An illustrative console trace looks like this:

```text
▶ agent Agent
  ▶ agent.session provider="opencode"
  • sandbox.process state="spawn_requested"
  • sandbox.process execution.id="remote-or-local-id" state="started"
  • acp.session.created sessionId="session-1"
    ▶ agent.turn index=1 kind="initial"
    • acp.session.prompt.submitted sessionId="session-1"
    • acp.session.prompt.completed sessionId="session-1" stopReason="end_turn"
    ✓ agent.turn 1.2s sessionId="session-1" stopReason="end_turn"
    ▶ agent.turn index=2 kind="follow-up"
    • acp.session.prompt.submitted sessionId="session-1"
    • acp.session.update sessionId="session-1" sessionUpdate="tool_call" toolName="shell"
    • acp.session.prompt.completed sessionId="session-1" stopReason="end_turn"
    ✓ agent.turn 2.4s sessionId="session-1" stopReason="end_turn"
    ▶ agent.cleanup
    • sandbox.process execution.id="remote-or-local-id" state="kill_requested"
    • sandbox.process execution.id="remote-or-local-id" state="kill_completed"
    ✓ agent.cleanup 8.1ms
  ✓ agent.session 3.6s
✓ agent Agent 3.6s
```

Exact updates depend on the Agent implementation. A provider that emits no ACP updates still produces the prompt-submitted event, the turn result or failure, cleanup, and enclosing span completion.

The console tracer omits `agent_message_chunk`, `agent_thought_chunk`, and `tool_call_update` events to keep the interactive tree compact, even when content capture is enabled. These events remain in the trace stream for custom sinks. The initial `tool_call` remains visible and includes `toolName` when the ACP Agent supplies its optional programmatic name.

## The ACP boundary

Every ACP session notification becomes one `acp.session.update` point event. AML deliberately does not translate each ACP update variant into a second AML schema.

Metadata-only tracing includes:

```ts
{
  name: "acp.session.update",
  attributes: {
    sessionId: "session-1",
    sessionUpdate: "tool_call",
    toolName: "shell",
  },
  type: "event",
}
```

`sessionId` and `sessionUpdate` come directly from the ACP notification. An initial `tool_call` also includes `toolName` when ACP supplies its optional programmatic name. Message chunks, thought chunks, the rest of the Tool lifecycle, plans, usage, configuration, session information, and future ACP variants otherwise use this same event shape when the Agent emits them.

With content capture enabled, AML adds an `update` attribute containing the serialized ACP update object unchanged. Consumers that need `toolCallId`, Tool status, plan entries, usage details, or another variant-specific field can parse that captured ACP object according to the ACP version they support. Apart from the optional programmatic Tool name, AML does not copy those fields into a parallel portable schema.

**Note — Portable ACP telemetry, not model-call telemetry**

AML can report what crosses ACP. It cannot universally observe model retries, fallbacks, provider queue time, internal
sub-agents, dynamically selected model names, vendor cache or billing metrics, or Tool work the Agent does not report.
A provider may add its own telemetry for those details, but it should label them as provider telemetry rather than an
AML Agent turn or portable ACP event.

## Agent lifecycle events

### Spans

| Span name       | Important metadata                                                                          | Boundary                                                                 |
| --------------- | ------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ |
| `agent.session` | `provider`, optional configured `model`                                                     | Provider session setup through completed cleanup                         |
| `agent.turn`    | `index`, `kind`; on success, ACP `sessionId`, `stopReason`, and optional serialized `usage` | One actual `runTurn()` / ACP prompt request                              |
| `agent.cleanup` | normal span status and duration                                                             | Provider session close, process termination, and session-owned resources |

Failed span ends include `error.type`. `error.message` is sensitive and appears only for content-capturing sinks.

### Point events

| Event                          | Metadata-only attributes                                     | Meaning                                                        |
| ------------------------------ | ------------------------------------------------------------ | -------------------------------------------------------------- |
| `acp.session.created`          | `sessionId`                                                  | ACP session creation completed                                 |
| `acp.session.prompt.submitted` | `sessionId`                                                  | AML submitted one turn                                         |
| `acp.session.update`           | `sessionId`, `sessionUpdate`                                 | One unchanged ACP update crossed the connection                |
| `acp.session.prompt.completed` | `sessionId`, `stopReason`, optional serialized `usage`       | ACP returned the final prompt response                         |
| `acp.session.cancel`           | `sessionId`                                                  | AML sent ACP session cancellation                              |
| `agent.session`                | `state="cancellation_requested"`                             | Evaluation cancellation reached the provider session           |
| `agent.output`                 | submission `call`, `status`                                  | Structured output was `accepted`, `ignored`, or `invalid`      |
| `sandbox.process`              | `state`, optional opaque `execution.id`, optional `exitCode` | Process spawn, termination request, observation, or completion |
| `capability.tool` / `.mcp`     | capability metadata                                          | AML granted a Tool or MCP capability                           |
| `loop.transition`              | loop transition metadata                                     | AML advanced a Loop                                            |

Process states currently emitted by the shared ACP path are:

| State             | What AML knows                                                                 |
| ----------------- | ------------------------------------------------------------------------------ |
| `spawn_requested` | AML asked the selected local or Sandbox runtime to start the command           |
| `started`         | The runtime returned an opaque process handle                                  |
| `kill_requested`  | AML called the process handle's termination boundary                           |
| `kill_completed`  | That termination request resolved                                              |
| `exited`          | `wait()` resolved with an exit code                                            |
| `wait_failed`     | `wait()` rejected before AML requested termination; process exit is not proven |

`execution.id` is opaque. A local provider may use a PID-backed value; Docker or a remote Sandbox may use a lease, session, or provider execution identifier. Consumers must not parse it as a Unix PID. `kill_completed` does not mean `exited`, and AML does not synthesize an exit merely because cancellation was requested.

## Content capture and redaction

Metadata-only tracing is the default. It retains lifecycle, correlation, update discriminants, optional programmatic Tool names, stop reasons, token usage supplied by ACP, durations, provider/model configuration, and opaque process identity.

Content capture can additionally expose:

- initial and follow-up prompt text;
- Agent message and thought content inside serialized ACP updates;
- raw ACP Tool input, output, titles, and progress;
- plans and repository information carried by ACP;
- structured-result submissions;
- executable commands and failure messages;
- Tool input/output and other runtime fields explicitly marked sensitive by AML.

The console tracer still suppresses `agent_message_chunk` and `tool_call_update` lines. Attach a custom content-capturing sink when those streamed payloads are required.

```tsx
// Diagnostic-only choice. Approve the destination and retention policy first.
runtime.on("trace", createConsoleTracer({ captureContent: true }))
```

Each trace sink chooses its own `captureContent` policy. Treat captured content as customer data. The option is consent to receive supported fields, not a general-purpose redactor and not a guarantee that every external provider action becomes observable.

**Caution — Thoughts require explicit content capture**

AML never includes Agent thought or reasoning content in metadata-only traces. When capture is enabled, thought
updates are present only if the Agent emitted them through ACP. Review whether storing reasoning is permitted before
enabling capture in any shared or production destination.

## The discriminated event union

Every event contains immutable `runId`, `spanId`, optional `parentSpanId`, `sequence`, `timestamp`, and scalar `attributes`.

| `type`       | Shape-specific fields                  | Meaning                      |
| ------------ | -------------------------------------- | ---------------------------- |
| `span.start` | `kind`, `name`                         | Opens an execution boundary  |
| `span.end`   | `kind`, `name`, `status`, `durationMs` | Closes the matching span     |
| `event`      | `name`                                 | Records a point-in-time fact |

TypeScript narrowing is the safest way to adapt the stream to OpenTelemetry, PostHog, LangChain telemetry, or an application logger:

```ts
runtime.on("trace", event => {
  if (event.type === "span.end") {
    telemetry.recordSpan({
      durationMs: event.durationMs,
      name: event.name,
      parentSpanId: event.parentSpanId,
      runId: event.runId,
      spanId: event.spanId,
      status: event.status,
    })
    return
  }

  if (event.type === "event") {
    telemetry.recordEvent(event.name, event.attributes)
  }
})
```

AML attributes are scalars (`boolean`, `number`, `string`, or `readonly string[]`) so observers do not receive live workflow, provider, Tool, or resource objects. Serialized ACP `usage` and content-captured `update` values are strings for the same reason.

## Correlation and ordering

Use:

- `runId` for one root evaluation;
- `spanId` and `parentSpanId` to reconstruct nesting;
- `sequence` for the authoritative event order within a run;
- `timestamp` for wall-clock correlation;
- `durationMs` on `span.end` for elapsed time.

Do not infer ordering from timestamps when `sequence` is available. Do not assume a point event has `status` or `durationMs`.

## What traces can answer

| Question                                     | Evidence                                                                 |
| -------------------------------------------- | ------------------------------------------------------------------------ |
| Did turn two start before turn one ended?    | ordered `agent.turn` span starts/ends and `sequence`                     |
| Is a slow Agent still producing progress?    | `acp.session.update` events inside the active turn                       |
| Did ACP report Tool or plan activity?        | `sessionUpdate` on `acp.session.update`; captured `update` when approved |
| Why did the turn stop?                       | `stopReason` on prompt completion and the successful turn end            |
| Did ACP supply token usage?                  | serialized `usage` on prompt completion and the successful turn end      |
| Was structured output accepted?              | `agent.output` `call` and `status`                                       |
| Was process termination requested or proven? | distinct `sandbox.process` states                                        |
| Did cleanup finish?                          | `agent.cleanup` span end and enclosing `agent.session` end               |

Trace metadata is execution telemetry, not a business audit log. External effects, provider billing, and internal model calls require their own authoritative records.