# Testing AML workflows

Test AML trees with deterministic providers, recorded requests, lifecycle assertions, and Vitest spies.
Canonical: https://agent-markup-language.com/docs/cookbook/testing/
Documentation index: https://agent-markup-language.com/docs/
Complete documentation: https://agent-markup-language.com/docs/llms.txt

AML workflows can be tested without model credentials or real infrastructure. Use deterministic providers to exercise tree evaluation and normalized contracts, then reserve credentialed smoke tests for the provider combinations your deployment actually runs.

## What each test layer proves

| Layer                         | Proves                                                                 | Does not prove                                                        |
| ----------------------------- | ---------------------------------------------------------------------- | --------------------------------------------------------------------- |
| Workflow test                 | Authored order, prompts, capabilities, results, limits, and placement. | Live model quality, executable installation, or remote compatibility. |
| Deterministic resource test   | Acquisition, command, save, release, conflict, and failure handling.   | Container or remote-platform isolation and filesystem semantics.      |
| Provider conformance test     | Provider-neutral lifecycle requirements checked by AML.                | Credentials, vendor availability, or every provider-specific feature. |
| Credentialed integration test | The selected executable, credentials, image, and service work now.     | Every model, environment, or future vendor release.                   |

## Run a complete workflow with testing defaults

Keep provider selection at the runtime boundary when you want to run the same AML tree in production and tests. An [`AmlRuntime`](https://agent-markup-language.com/docs/reference/runtime/) supplies defaults to every [`<Agent />`](https://agent-markup-language.com/docs/reference/primitives/agent/), [`<Sandbox />`](https://agent-markup-language.com/docs/reference/primitives/sandbox/), and [`<Workspace />`](https://agent-markup-language.com/docs/reference/primitives/workspace/) that does not set its own `provider` prop.

This workflow contains no test-specific branches and no component-level providers:

```tsx title="src/review-workflow.tsx"
import { Agent, Sandbox, Workspace } from "@aml-jsx/sdk"

export function ReviewWorkflow() {
  return (
    <Workspace id="release-review" save>
      <Sandbox access="read-only">
        <Agent>Review the release candidate.</Agent>
      </Sandbox>
    </Workspace>
  )
}
```

The test creates another runtime whose defaults come entirely from `@aml-jsx/sdk/testing`:

```tsx title="src/review-workflow.test.tsx"
import { expect, it } from "vitest"
import { AmlRuntime } from "@aml-jsx/sdk"
import {
  DeterministicAgentProvider,
  DeterministicSandboxProvider,
  DeterministicWorkspaceProvider,
} from "@aml-jsx/sdk/testing"
import { ReviewWorkflow } from "./review-workflow.js"

it("runs the complete review without live resources", async () => {
  const agentProvider = new DeterministicAgentProvider({
    respond: request => ({ text: `fixture:${request.prompt}` }),
    supportsSandbox: () => true,
  })
  const sandboxProvider = new DeterministicSandboxProvider()
  const workspaceProvider = new DeterministicWorkspaceProvider()
  const runtime = new AmlRuntime({
    agentProvider,
    sandboxProvider,
    workspaceProvider,
  })

  await expect(runtime.evaluate(<ReviewWorkflow />)).resolves.toBe("fixture:Review the release candidate.")
  expect(agentProvider.calls).toHaveLength(1)
  expect(sandboxProvider.acquisitions).toHaveLength(1)
  expect(sandboxProvider.releases).toHaveLength(1)
  expect(workspaceProvider.saves).toHaveLength(1)
  expect(workspaceProvider.releases).toHaveLength(1)
})
```

[`DeterministicAgentProvider`](https://agent-markup-language.com/docs/reference/testing/#deterministicagentprovider) records normalized Agent requests and returns the response selected by `respond`. Because this `<Agent />` runs inside a `<Sandbox />`, `supportsSandbox` explicitly tells AML that the fixture accepts the effective Sandbox session. [`DeterministicSandboxProvider`](https://agent-markup-language.com/docs/reference/testing/#deterministicsandboxprovider) records acquisition and release and exposes controllable `exec` and `spawn` hooks. [`DeterministicWorkspaceProvider`](https://agent-markup-language.com/docs/reference/testing/#deterministicworkspaceprovider) records acquisition, save, and release while enforcing its in-memory writer-conflict rules.

Runtime defaults only replace omitted providers. If the workflow writes `<Agent provider={liveProvider}>`, that local provider still wins. Keeping environment-specific providers out of reusable workflow components makes whole-workflow substitution predictable.

## Select testing providers with `NODE_ENV`

Applications that own a shared runtime factory can select the testing entrypoint at their composition root. Make the production options lazy so a test run does not construct live providers before the environment check:

```ts title="src/runtime.ts"
import { AmlRuntime, type AmlRuntimeOptions } from "@aml-jsx/sdk"

export async function createApplicationRuntime(production: () => AmlRuntimeOptions): Promise<AmlRuntime> {
  if (process.env.NODE_ENV !== "test") {
    return new AmlRuntime(production())
  }

  const { DeterministicAgentProvider, DeterministicSandboxProvider, DeterministicWorkspaceProvider } =
    await import("@aml-jsx/sdk/testing")

  return new AmlRuntime({
    agentProvider: new DeterministicAgentProvider({
      supportsSandbox: () => true,
    }),
    sandboxProvider: new DeterministicSandboxProvider(),
    workspaceProvider: new DeterministicWorkspaceProvider(),
  })
}
```

```tsx title="src/review-workflow.test.tsx"
import { expect, it } from "vitest"
import { createApplicationRuntime } from "./runtime.js"
import { ReviewWorkflow } from "./review-workflow.js"

it("uses testing providers in the test environment", async () => {
  const runtime = await createApplicationRuntime(() => {
    throw new Error("production providers must not be constructed in this test")
  })

  await expect(runtime.evaluate(<ReviewWorkflow />)).resolves.toBe("Review the release candidate.")
})
```

`NODE_ENV` is an application-level selection policy, not hidden AML behavior: `AmlRuntime` uses exactly the providers passed to its constructor. Set `NODE_ENV=test` in the test command or Vitest environment rather than mutating it inside individual tests. Prefer the explicit runtime from the first example when a test needs custom responses or direct access to recorded calls; use the shared factory when the application should exercise its normal runtime wiring with deterministic defaults.

These patterns follow AML's maintained tests for [runtime Agent defaults](https://github.com/we-are-singular/aml/blob/main/sdk/tests/agent-runtime.test.tsx), [Sandbox defaults and lifecycle](https://github.com/we-are-singular/aml/blob/main/sdk/tests/sandbox-runtime.test.tsx), [Workspace defaults and lifecycle](https://github.com/we-are-singular/aml/blob/main/sdk/tests/workspace-runtime.test.tsx), and [combined Agent, Sandbox, Workspace, and Script behavior](https://github.com/we-are-singular/aml/blob/main/sdk/tests/script-runtime.test.tsx).

## Assert normalized Agent requests

The deterministic Agent provider records every request in execution order. This makes workflow tests assert behavior at AML's provider boundary rather than snapshotting internal JSX nodes.

```tsx
import { describe, expect, it } from "vitest"
import { Agent, AmlRuntime, System } from "@aml-jsx/sdk"
import { DeterministicAgentProvider } from "@aml-jsx/sdk/testing"

describe("release summary", () => {
  it("sends the resolved prompt and system instruction", async () => {
    const provider = new DeterministicAgentProvider({
      respond: request => ({ text: `Reviewed: ${request.prompt}` }),
    })
    const runtime = new AmlRuntime({ agentProvider: provider })

    await expect(
      runtime.evaluate(
        <Agent>
          <System>Return one sentence.</System>
          Review release 42.
        </Agent>
      )
    ).resolves.toBe("Reviewed: Review release 42.")

    expect(provider.calls).toHaveLength(1)
    expect(provider.calls[0]?.request).toMatchObject({
      prompt: "Review release 42.",
      system: "Return one sentence.",
    })
  })
})
```

Prefer focused assertions on the normalized request and result. Avoid broad snapshots of trace IDs or other intentionally dynamic metadata.

## Spy on application-owned Tool behavior

Vitest spies are useful at dependencies your application owns. This example deliberately calls `.execute(input, context)` because the deterministic provider receives AML's low-level provider-facing port. Application components should call the `defineTool()` result directly.

```tsx
import { expect, it, vi } from "vitest"
import { Agent, AmlRuntime, Tool, defineTool } from "@aml-jsx/sdk"
import { DeterministicAgentProvider } from "@aml-jsx/sdk/testing"
import { z } from "zod"

it("calls the granted repository operation once", async () => {
  const repository = {
    async findOrder(id: string) {
      return { id, status: "ready" }
    },
  }
  const findOrder = vi.spyOn(repository, "findOrder")
  const tool = defineTool({
    name: "find_order",
    description: "Find one order by ID.",
    input: z.object({ id: z.string() }),
    execute: async ({ id }) => await repository.findOrder(id),
  })
  const provider = new DeterministicAgentProvider({
    async respond(request, context) {
      const granted = request.tools.find(candidate => candidate.name === "find_order")
      if (granted?.kind !== "javascript") throw new Error("find_order was not granted")
      const order = await granted.execute({ id: "order-17" }, { signal: context.signal, trace: context.trace })
      return { text: JSON.stringify(order) }
    },
  })

  await new AmlRuntime({ agentProvider: provider }).evaluate(
    <Agent>
      <Tool use={tool} />
      Inspect order-17.
    </Agent>
  )

  expect(findOrder).toHaveBeenCalledOnce()
  expect(findOrder).toHaveBeenCalledWith("order-17")
})
```

**Danger — A deterministic provider can execute granted Tools**

The provider itself is credential-free, but a Tool is application code. Use test doubles for databases, network
clients, process launchers, and other side effects rather than pointing a workflow test at production dependencies.

## Assert Sandbox and Workspace lifecycle

Constructor hooks pair naturally with `vi.fn()` when the order or presence of cleanup matters:

```tsx
import { expect, it, vi } from "vitest"
import { AmlRuntime, Workspace } from "@aml-jsx/sdk"
import { DeterministicWorkspaceProvider } from "@aml-jsx/sdk/testing"

it("saves successful Workspace output before release", async () => {
  const save = vi.fn(async () => {})
  const release = vi.fn(async () => {})
  const workspaceProvider = new DeterministicWorkspaceProvider({ save, release })

  await new AmlRuntime({ workspaceProvider }).evaluate(
    <Workspace id="review" load={false} save={{ on: "always" }}>
      ready
    </Workspace>
  )

  expect(save).toHaveBeenCalledOnce()
  expect(release).toHaveBeenCalledOnce()
  expect(workspaceProvider.saves).toHaveLength(1)
  expect(workspaceProvider.releases).toHaveLength(1)
})
```

Use `DeterministicSandboxProvider` the same way to control `exec`, `spawn`, and release behavior without starting a host process or remote Sandbox.

## Test failure and non-execution paths

1. Make a fixture hook throw the failure the application must classify.
2. Assert the evaluation rejects with the expected class, identity, or cause.
3. Assert later provider calls, Tools, saves, or other effects did not run.
4. Assert owned cleanup still ran when acquisition had completed.

Tests for validation failures should usually assert `provider.calls` remains empty. Tests for Tool validation should spy on the executor and prove invalid model input never reached application code.

## Test a custom provider contract

Provider authors can run AML's test-runner-independent conformance helper inside Vitest:

```ts
import { expect, it } from "vitest"
import { agentProviderConformance } from "@aml-jsx/sdk/testing"

it("implements the AML Agent provider contract", async () => {
  await expect(agentProviderConformance(provider)).resolves.toBeUndefined()
})
```

Run the matching Sandbox or Workspace conformance helper for those provider types. Add provider-specific tests for configuration, process transport, storage behavior, and cleanup beyond the neutral contract.

## Choose the next test

- [Testing API reference](https://agent-markup-language.com/docs/reference/testing/) — Review every deterministic fixture, recorded call surface, storage spy, and conformance helper.

- [Compatibility](https://agent-markup-language.com/docs/compatibility/) — Understand what depends on executables, images, credentials, and provider-specific runtime behavior.

- [Examples](https://agent-markup-language.com/docs/examples/) — Run maintained deterministic and resource-backed workflows before adapting them into tests.