Observability

AML exposes two observability layers:

1. Runtime lifecycle events: start and finish, for request-level status and run identity.
2. Trace events: immutable spans and point events for the evaluation tree, Agent sessions and turns, ACP activity, capabilities, processes, and cleanup.

This page defines the portable trace contract. For listener timing and evaluation lifecycle decisions, use the runtime reference.

Start tracing

SDK

import { AmlRuntime, createConsoleTracer } from "@aml-jsx/sdk"

const runtime = new AmlRuntime({
  trace: createConsoleTracer({ captureContent: false }),
  onTraceError: (error, event) => {
    console.error("AML trace sink failed", { error, sequence: event.sequence })
  },
})

The tracer writes a compact tree to console.log. Supply write to send complete lines to another destination:

const trace = createConsoleTracer({
  captureContent: false,
  write: line => process.stderr.write(`${line}\n`),
})

CLI

aml run ./workflow.tsx --trace

The experimental CLI writes metadata-only traces to standard error and keeps the workflow result on standard output.

# Explicitly includes supported sensitive content.
aml run ./workflow.tsx --trace --capture-content

--capture-content is an AML CLI option. It is not part of ACP and is not sent to the Agent.

Trace sinks are captured when the runtime is constructed. AML does not await a sink’s returned promise. Synchronous throws and rejected promises go to onTraceError when configured; telemetry failure does not change the workflow result.

Application-owned spans

Automatic component spans cover both a function component call and resolution of its returned subtree. They cannot isolate a narrower deterministic phase such as validation, retrieval, or persistence. Measure that work with withTraceSpan() inside an active function component:

async function Review({ findings }: { findings: readonly Finding[] }) {
  return await withTraceSpan("review.validate", async () => await validateFindings(findings))
}

The runtime allocates an application span beneath the active component or enclosing application span. It closes the span on success, thrown failure, or cancellation without changing the callback result. Calls outside an active component, including detached calls after component settlement, fail. Nested calls remain correctly parented across evaluate(), <Parallel>, and concurrent async work.

Applications supply only the span name and callback. They never supply trace identity, parent fields, metadata, or content.

Run summaries

createTraceSummaryCollector() derives content-free summaries from the same public trace stream:

const summaries = createTraceSummaryCollector()
const runtime = new AmlRuntime({
  onTraceError: (error, event) => reportTraceFailure(error, event),
  trace: summaries.trace,
})

function summaryFor(runId: string) {
  const summary = summaries.forRun(runId)
  summaries.deleteRun(runId)
  return summary
}

forRun() requires an explicit evaluation runId; there is no latest-run API that becomes ambiguous when a runtime handles overlapping evaluations. Capture that identity from the public start event using the request-correlation pattern in the cookbook below. Completed summaries report evaluation status and wall duration, Agent session and turn timing, Tool and resource timing, named application spans, ACP tool-call counts, provider-reported usage entries, and Agent cleanup outcomes. acpToolCalls.byName lets an application check whether a supplied provider capability was invoked without persisting raw ACP events. It counts only initial tool_call updates and retains only the exact public provider-reported name and count—not arguments, results, later updates, prompts, or model text. Provider capability names do not portably guarantee which backend or MCP server handled a call.

The separate tools timing aggregate measures declarative AML <Tool> executions. A provider call routed to an AML Tool may correctly appear in both aggregates; AML does not deduplicate these different boundaries. Timing aggregates contain count, summed totalDurationMs, and slowestMs when at least one span was observed. The collector does not rewrite evaluation status. Trace-consumer failures remain separate on the runtime’s existing onTraceError channel.

providerUsage is empty when the provider reports no usage. When ACP supplies usage, AML retains the serialized JSON string rather than promising token fields that may not exist. A turn is one provider/ACP prompt request, not one model call. Summaries do not infer provider retries, cache behavior, costs, or billing data. See Correlate application spans and run summaries for a concurrent production pattern.

What an Agent trace represents

AML traces the lifecycle it owns and the events that cross its ACP connection. One ACP session/prompt request is one Agent turn. It is not necessarily one model API call.

agent Agent
└─ agent.session                    one provider session
   ├─ sandbox.process               process lifecycle facts
   ├─ acp.session.created           ACP session established
   ├─ agent.turn kind=initial       first session/prompt request
   │  ├─ acp.session.prompt.submitted
   │  ├─ acp.session.update         zero or more streamed ACP updates
   │  └─ acp.session.prompt.completed
   ├─ agent.turn kind=follow-up     next authored <FollowUp />
   │  └─ ...
   └─ agent.cleanup                 session and process cleanup

Turn spans start immediately before provider execution and end before the next turn begins. A two-turn Agent therefore never reports turn two as started while turn one is still running. ACP updates are emitted in the order AML consumes them from ActiveSession.nextUpdate().

An illustrative console trace looks like this:

▶ agent Agent
  ▶ agent.session provider="opencode"
  • sandbox.process state="spawn_requested"
  • sandbox.process execution.id="remote-or-local-id" state="started"
  • acp.session.created sessionId="session-1"
    ▶ agent.turn index=1 kind="initial"
    • acp.session.prompt.submitted sessionId="session-1"
    • acp.session.prompt.completed sessionId="session-1" stopReason="end_turn"
    ✓ agent.turn 1.2s sessionId="session-1" stopReason="end_turn"
    ▶ agent.turn index=2 kind="follow-up"
    • acp.session.prompt.submitted sessionId="session-1"
    • acp.session.update sessionId="session-1" sessionUpdate="tool_call" toolName="shell"
    • acp.session.prompt.completed sessionId="session-1" stopReason="end_turn"
    ✓ agent.turn 2.4s sessionId="session-1" stopReason="end_turn"
    ▶ agent.cleanup
    • sandbox.process execution.id="remote-or-local-id" state="kill_requested"
    • sandbox.process execution.id="remote-or-local-id" state="kill_completed"
    ✓ agent.cleanup 8.1ms
  ✓ agent.session 3.6s
✓ agent Agent 3.6s

Exact updates depend on the Agent implementation. A provider that emits no ACP updates still produces the prompt-submitted event, the turn result or failure, cleanup, and enclosing span completion.

The console tracer omits agent_message_chunk, agent_thought_chunk, and tool_call_update events to keep the interactive tree compact, even when content capture is enabled. These events remain in the trace stream for custom sinks. The initial tool_call remains visible and includes toolName when the ACP Agent supplies its optional programmatic name.

The ACP boundary

Every ACP session notification becomes one acp.session.update point event. AML deliberately does not translate each ACP update variant into a second AML schema.

Metadata-only tracing includes:

{
  name: "acp.session.update",
  attributes: {
    sessionId: "session-1",
    sessionUpdate: "tool_call",
    toolName: "shell",
  },
  type: "event",
}

sessionId and sessionUpdate come directly from the ACP notification. An initial tool_call also includes toolName when ACP supplies its optional programmatic name. Message chunks, thought chunks, the rest of the Tool lifecycle, plans, usage, configuration, session information, and future ACP variants otherwise use this same event shape when the Agent emits them.

With content capture enabled, AML adds an update attribute containing the serialized ACP update object unchanged. Consumers that need toolCallId, Tool status, plan entries, usage details, or another variant-specific field can parse that captured ACP object according to the ACP version they support. Apart from the optional programmatic Tool name, AML does not copy those fields into a parallel portable schema.

Portable ACP telemetry, not model-call telemetry

AML can report what crosses ACP. It cannot universally observe model retries, fallbacks, provider queue time, internal sub-agents, dynamically selected model names, vendor cache or billing metrics, or Tool work the Agent does not report. A provider may add its own telemetry for those details, but it should label them as provider telemetry rather than an AML Agent turn or portable ACP event.

Agent lifecycle events

Spans

Span name	Important metadata	Boundary
agent.session	provider, optional configured model	Provider session setup through completed cleanup
agent.turn	index, kind; on success, ACP sessionId, stopReason, and optional serialized usage	One actual runTurn() / ACP prompt request
agent.cleanup	normal span status and duration	Provider session close, process termination, and session-owned resources

Failed span ends include error.type. error.message is sensitive and appears only for content-capturing sinks.

Point events

Event	Metadata-only attributes	Meaning
acp.session.created	sessionId	ACP session creation completed
acp.session.prompt.submitted	sessionId	AML submitted one turn
acp.session.update	sessionId, sessionUpdate	One unchanged ACP update crossed the connection
acp.session.prompt.completed	sessionId, stopReason, optional serialized usage	ACP returned the final prompt response
acp.session.cancel	sessionId	AML sent ACP session cancellation
agent.session	state="cancellation_requested"	Evaluation cancellation reached the provider session
agent.output	submission call, status	Structured output was accepted, ignored, or invalid
sandbox.process	state, optional opaque execution.id, optional exitCode	Process spawn, termination request, observation, or completion
capability.tool / .mcp	capability metadata	AML granted a Tool or MCP capability
loop.transition	loop transition metadata	AML advanced a Loop

Process states currently emitted by the shared ACP path are:

State	What AML knows
spawn_requested	AML asked the selected local or Sandbox runtime to start the command
started	The runtime returned an opaque process handle
kill_requested	AML called the process handle’s termination boundary
kill_completed	That termination request resolved
exited	wait() resolved with an exit code
wait_failed	wait() rejected before AML requested termination; process exit is not proven

execution.id is opaque. A local provider may use a PID-backed value; Docker or a remote Sandbox may use a lease, session, or provider execution identifier. Consumers must not parse it as a Unix PID. kill_completed does not mean exited, and AML does not synthesize an exit merely because cancellation was requested.

Content capture and redaction

Metadata-only tracing is the default. It retains lifecycle, correlation, update discriminants, optional programmatic Tool names, stop reasons, token usage supplied by ACP, durations, provider/model configuration, and opaque process identity.

Content capture can additionally expose:

• initial and follow-up prompt text;
• Agent message and thought content inside serialized ACP updates;
• raw ACP Tool input, output, titles, and progress;
• plans and repository information carried by ACP;
• structured-result submissions;
• executable commands and failure messages;
• Tool input/output and other runtime fields explicitly marked sensitive by AML.

The console tracer still suppresses agent_message_chunk and tool_call_update lines. Attach a custom content-capturing sink when those streamed payloads are required.

// Diagnostic-only choice. Approve the destination and retention policy first.
runtime.on("trace", createConsoleTracer({ captureContent: true }))

Each trace sink chooses its own captureContent policy. Treat captured content as customer data. The option is consent to receive supported fields, not a general-purpose redactor and not a guarantee that every external provider action becomes observable.

Thoughts require explicit content capture

AML never includes Agent thought or reasoning content in metadata-only traces. When capture is enabled, thought updates are present only if the Agent emitted them through ACP. Review whether storing reasoning is permitted before enabling capture in any shared or production destination.

The discriminated event union

Every event contains immutable runId, spanId, optional parentSpanId, sequence, timestamp, and scalar attributes.

type	Shape-specific fields	Meaning
span.start	kind, name	Opens an execution boundary
span.end	kind, name, status, durationMs	Closes the matching span
event	name	Records a point-in-time fact

TypeScript narrowing is the safest way to adapt the stream to OpenTelemetry, PostHog, LangChain telemetry, or an application logger:

runtime.on("trace", event => {
  if (event.type === "span.end") {
    telemetry.recordSpan({
      durationMs: event.durationMs,
      name: event.name,
      parentSpanId: event.parentSpanId,
      runId: event.runId,
      spanId: event.spanId,
      status: event.status,
    })
    return
  }

  if (event.type === "event") {
    telemetry.recordEvent(event.name, event.attributes)
  }
})

AML attributes are scalars (boolean, number, string, or readonly string[]) so observers do not receive live workflow, provider, Tool, or resource objects. Serialized ACP usage and content-captured update values are strings for the same reason.

Correlation and ordering

Use:

• runId for one root evaluation;
• spanId and parentSpanId to reconstruct nesting;
• sequence for the authoritative event order within a run;
• timestamp for wall-clock correlation;
• durationMs on span.end for elapsed time.

Do not infer ordering from timestamps when sequence is available. Do not assume a point event has status or durationMs.

What traces can answer

Question	Evidence
Did turn two start before turn one ended?	ordered agent.turn span starts/ends and sequence
Is a slow Agent still producing progress?	acp.session.update events inside the active turn
Did ACP report Tool or plan activity?	sessionUpdate on acp.session.update; captured update when approved
Why did the turn stop?	stopReason on prompt completion and the successful turn end
Did ACP supply token usage?	serialized usage on prompt completion and the successful turn end
Was structured output accepted?	agent.output call and status
Was process termination requested or proven?	distinct sandbox.process states
Did cleanup finish?	agent.cleanup span end and enclosing agent.session end

Trace metadata is execution telemetry, not a business audit log. External effects, provider billing, and internal model calls require their own authoritative records.