Testing AML workflows

AML workflows can be tested without model credentials or real infrastructure. Use deterministic providers to exercise tree evaluation and normalized contracts, then reserve credentialed smoke tests for the provider combinations your deployment actually runs.

What each test layer proves

Layer	Proves	Does not prove
Workflow test	Authored order, prompts, capabilities, results, limits, and placement.	Live model quality, executable installation, or remote compatibility.
Deterministic resource test	Acquisition, command, save, release, conflict, and failure handling.	Container or remote-platform isolation and filesystem semantics.
Provider conformance test	Provider-neutral lifecycle requirements checked by AML.	Credentials, vendor availability, or every provider-specific feature.
Credentialed integration test	The selected executable, credentials, image, and service work now.	Every model, environment, or future vendor release.

Run a complete workflow with testing defaults

Keep provider selection at the runtime boundary when you want to run the same AML tree in production and tests. An AmlRuntime supplies defaults to every <Agent />, <Sandbox />, and <Workspace /> that does not set its own provider prop.

This workflow contains no test-specific branches and no component-level providers:

src/review-workflow.tsx

import { Agent, Sandbox, Workspace } from "@aml-jsx/sdk"

export function ReviewWorkflow() {
  return (
    <Workspace id="release-review" save>
      <Sandbox access="read-only">
        <Agent>Review the release candidate.</Agent>
      </Sandbox>
    </Workspace>
  )
}

The test creates another runtime whose defaults come entirely from @aml-jsx/sdk/testing:

src/review-workflow.test.tsx

import { expect, it } from "vitest"
import { AmlRuntime } from "@aml-jsx/sdk"
import {
  DeterministicAgentProvider,
  DeterministicSandboxProvider,
  DeterministicWorkspaceProvider,
} from "@aml-jsx/sdk/testing"
import { ReviewWorkflow } from "./review-workflow.js"

it("runs the complete review without live resources", async () => {
  const agentProvider = new DeterministicAgentProvider({
    respond: request => ({ text: `fixture:${request.prompt}` }),
    supportsSandbox: () => true,
  })
  const sandboxProvider = new DeterministicSandboxProvider()
  const workspaceProvider = new DeterministicWorkspaceProvider()
  const runtime = new AmlRuntime({
    agentProvider,
    sandboxProvider,
    workspaceProvider,
  })

  await expect(runtime.evaluate(<ReviewWorkflow />)).resolves.toBe("fixture:Review the release candidate.")
  expect(agentProvider.calls).toHaveLength(1)
  expect(sandboxProvider.acquisitions).toHaveLength(1)
  expect(sandboxProvider.releases).toHaveLength(1)
  expect(workspaceProvider.saves).toHaveLength(1)
  expect(workspaceProvider.releases).toHaveLength(1)
})

DeterministicAgentProvider records normalized Agent requests and returns the response selected by respond. Because this <Agent /> runs inside a <Sandbox />, supportsSandbox explicitly tells AML that the fixture accepts the effective Sandbox session. DeterministicSandboxProvider records acquisition and release and exposes controllable exec and spawn hooks. DeterministicWorkspaceProvider records acquisition, save, and release while enforcing its in-memory writer-conflict rules.

Runtime defaults only replace omitted providers. If the workflow writes <Agent provider={liveProvider}>, that local provider still wins. Keeping environment-specific providers out of reusable workflow components makes whole-workflow substitution predictable.

Select testing providers with NODE_ENV

Applications that own a shared runtime factory can select the testing entrypoint at their composition root. Make the production options lazy so a test run does not construct live providers before the environment check:

src/runtime.ts

import { AmlRuntime, type AmlRuntimeOptions } from "@aml-jsx/sdk"

export async function createApplicationRuntime(production: () => AmlRuntimeOptions): Promise<AmlRuntime> {
  if (process.env.NODE_ENV !== "test") {
    return new AmlRuntime(production())
  }

  const { DeterministicAgentProvider, DeterministicSandboxProvider, DeterministicWorkspaceProvider } =
    await import("@aml-jsx/sdk/testing")

  return new AmlRuntime({
    agentProvider: new DeterministicAgentProvider({
      supportsSandbox: () => true,
    }),
    sandboxProvider: new DeterministicSandboxProvider(),
    workspaceProvider: new DeterministicWorkspaceProvider(),
  })
}

src/review-workflow.test.tsx

import { expect, it } from "vitest"
import { createApplicationRuntime } from "./runtime.js"
import { ReviewWorkflow } from "./review-workflow.js"

it("uses testing providers in the test environment", async () => {
  const runtime = await createApplicationRuntime(() => {
    throw new Error("production providers must not be constructed in this test")
  })

  await expect(runtime.evaluate(<ReviewWorkflow />)).resolves.toBe("Review the release candidate.")
})

NODE_ENV is an application-level selection policy, not hidden AML behavior: AmlRuntime uses exactly the providers passed to its constructor. Set NODE_ENV=test in the test command or Vitest environment rather than mutating it inside individual tests. Prefer the explicit runtime from the first example when a test needs custom responses or direct access to recorded calls; use the shared factory when the application should exercise its normal runtime wiring with deterministic defaults.

These patterns follow AML’s maintained tests for runtime Agent defaults, Sandbox defaults and lifecycle, Workspace defaults and lifecycle, and combined Agent, Sandbox, Workspace, and Script behavior.

Assert normalized Agent requests

The deterministic Agent provider records every request in execution order. This makes workflow tests assert behavior at AML’s provider boundary rather than snapshotting internal JSX nodes.

import { describe, expect, it } from "vitest"
import { Agent, AmlRuntime, System } from "@aml-jsx/sdk"
import { DeterministicAgentProvider } from "@aml-jsx/sdk/testing"

describe("release summary", () => {
  it("sends the resolved prompt and system instruction", async () => {
    const provider = new DeterministicAgentProvider({
      respond: request => ({ text: `Reviewed: ${request.prompt}` }),
    })
    const runtime = new AmlRuntime({ agentProvider: provider })

    await expect(
      runtime.evaluate(
        <Agent>
          <System>Return one sentence.</System>
          Review release 42.
        </Agent>
      )
    ).resolves.toBe("Reviewed: Review release 42.")

    expect(provider.calls).toHaveLength(1)
    expect(provider.calls[0]?.request).toMatchObject({
      prompt: "Review release 42.",
      system: "Return one sentence.",
    })
  })
})

Prefer focused assertions on the normalized request and result. Avoid broad snapshots of trace IDs or other intentionally dynamic metadata.

Spy on application-owned Tool behavior

Vitest spies are useful at dependencies your application owns. This example deliberately calls .execute(input, context) because the deterministic provider receives AML’s low-level provider-facing port. Application components should call the defineTool() result directly.

import { expect, it, vi } from "vitest"
import { Agent, AmlRuntime, Tool, defineTool } from "@aml-jsx/sdk"
import { DeterministicAgentProvider } from "@aml-jsx/sdk/testing"
import { z } from "zod"

it("calls the granted repository operation once", async () => {
  const repository = {
    async findOrder(id: string) {
      return { id, status: "ready" }
    },
  }
  const findOrder = vi.spyOn(repository, "findOrder")
  const tool = defineTool({
    name: "find_order",
    description: "Find one order by ID.",
    input: z.object({ id: z.string() }),
    execute: async ({ id }) => await repository.findOrder(id),
  })
  const provider = new DeterministicAgentProvider({
    async respond(request, context) {
      const granted = request.tools.find(candidate => candidate.name === "find_order")
      if (granted?.kind !== "javascript") throw new Error("find_order was not granted")
      const order = await granted.execute({ id: "order-17" }, { signal: context.signal, trace: context.trace })
      return { text: JSON.stringify(order) }
    },
  })

  await new AmlRuntime({ agentProvider: provider }).evaluate(
    <Agent>
      <Tool use={tool} />
      Inspect order-17.
    </Agent>
  )

  expect(findOrder).toHaveBeenCalledOnce()
  expect(findOrder).toHaveBeenCalledWith("order-17")
})

A deterministic provider can execute granted Tools

The provider itself is credential-free, but a Tool is application code. Use test doubles for databases, network clients, process launchers, and other side effects rather than pointing a workflow test at production dependencies.

Assert Sandbox and Workspace lifecycle

Constructor hooks pair naturally with vi.fn() when the order or presence of cleanup matters:

import { expect, it, vi } from "vitest"
import { AmlRuntime, Workspace } from "@aml-jsx/sdk"
import { DeterministicWorkspaceProvider } from "@aml-jsx/sdk/testing"

it("saves successful Workspace output before release", async () => {
  const save = vi.fn(async () => {})
  const release = vi.fn(async () => {})
  const workspaceProvider = new DeterministicWorkspaceProvider({ save, release })

  await new AmlRuntime({ workspaceProvider }).evaluate(
    <Workspace id="review" load={false} save={{ on: "always" }}>
      ready
    </Workspace>
  )

  expect(save).toHaveBeenCalledOnce()
  expect(release).toHaveBeenCalledOnce()
  expect(workspaceProvider.saves).toHaveLength(1)
  expect(workspaceProvider.releases).toHaveLength(1)
})

Use DeterministicSandboxProvider the same way to control exec, spawn, and release behavior without starting a host process or remote Sandbox.

Test failure and non-execution paths

1. Make a fixture hook throw the failure the application must classify.
2. Assert the evaluation rejects with the expected class, identity, or cause.
3. Assert later provider calls, Tools, saves, or other effects did not run.
4. Assert owned cleanup still ran when acquisition had completed.

Tests for validation failures should usually assert provider.calls remains empty. Tests for Tool validation should spy on the executor and prove invalid model input never reached application code.

Test a custom provider contract

Provider authors can run AML’s test-runner-independent conformance helper inside Vitest:

import { expect, it } from "vitest"
import { agentProviderConformance } from "@aml-jsx/sdk/testing"

it("implements the AML Agent provider contract", async () => {
  await expect(agentProviderConformance(provider)).resolves.toBeUndefined()
})

Run the matching Sandbox or Workspace conformance helper for those provider types. Add provider-specific tests for configuration, process transport, storage behavior, and cleanup beyond the neutral contract.

Choose the next test

Testing API referenceReview every deterministic fixture, recorded call surface, storage spy, and conformance helper.

CompatibilityUnderstand what depends on executables, images, credentials, and provider-specific runtime behavior.

ExamplesRun maintained deterministic and resource-backed workflows before adapting them into tests.