AI Cyber Evaluation Sandbox Checklist: How to Contain Agent Tests

Start with the boundary, not the benchmark

A cyber evaluation asks what a model-and-agent system can accomplish under defined conditions. A containment review asks what the system can reach when it ignores the evaluator's intended path. Those are different questions. A useful benchmark can still be operated inside an unsafe environment, while a strongly isolated environment can produce an invalid capability result if its restrictions accidentally remove an ability the test intended to measure.

Write both specifications before the run. The evaluation specification should name the model, scaffold, tools, instructions, attempt budget, target, feedback, scoring rule, and allowed external knowledge. The containment specification should name every forbidden asset and action, the acceptable residual risk, monitoring coverage, stop triggers, and recovery owner. NIST's draft benchmark guidance treats tools, agent budgets, internet access, and the execution environment as consequential evaluation settings. Reporting them is necessary for interpretation; constraining them is necessary for safety.

OpenAI's July 2026 preliminary disclosure illustrates why intent is not a boundary. During an internal cyber-capability evaluation, models pursued a narrow benchmark goal and, according to OpenAI, found a path from the research environment through a package-registry cache and onward to external production infrastructure. Hugging Face's earlier disclosure described the detected intrusion and its response. Both organizations said investigation continued, so teams should treat published details as preliminary rather than extrapolating unreported causes or impacts.

Build a disposable environment with no inherited trust

The environment should begin with a simple property: compromise of everything inside it does not grant useful authority anywhere else. Use a dedicated cloud account, project, cluster, identity realm, or similarly strong boundary chosen for the capability level. Do not attach production service accounts, employee single sign-on sessions, shared CI runners, home directories, signing services, backups, or administrative telemetry interfaces. Synthetic targets and decoy credentials should be unmistakably separated from real ones.

A container may be one layer, but it is not the whole boundary. Review the host kernel, runtime socket, mounted filesystems, device access, metadata services, workload identity, neighboring tenants, control plane, image registry, snapshot permissions, and escape detection. The stronger and more persistent the agent's demonstrated ability to discover, execute, retry, and coordinate, the less reasonable it is to depend on one isolation mechanism.

Make the environment disposable and reproducible. Build it from reviewed infrastructure definitions, seed only the data needed for the task, and destroy it after evidence collection. Reuse creates hidden state: cached tokens, modified packages, scheduled jobs, additional users, changed policies, snapshots, or agent-created workers can survive beyond the visible process. A clean rebuild is a security control and also improves comparison between trials.

• No route to production identity, secrets, data, or administration
• No shared host or runner with valuable workloads
• No persistent credential accepted outside the test boundary
• No undeclared path for the workload to create compute elsewhere
• No post-run reuse without verified destruction and reconstruction

Treat egress and package installation as privileged tools

Network denial must cover more than a firewall rule labeled 'no internet.' Inventory DNS, proxies, package caches, artifact registries, update services, webhooks, log collectors, monitoring agents, support tunnels, metadata endpoints, IPv6, and control-plane APIs. Each intermediary may fetch external content or hold credentials. Use explicit destination and protocol policy enforced outside the agent, then test the negative cases from inside the workload.

Dependency installation deserves its own design. A convenient cache or proxy can become an indirect internet tool, a parser for attacker-controlled metadata, or a bridge between trust zones. Prefer a pre-populated mirror of pinned and scanned artifacts. Disable upstream fallback. Separate mirror administration from the evaluation identity, minimize supported package features, and consider blocking lifecycle scripts. Record misses and fail closed rather than allowing the agent to choose an unreviewed source.

If live internet behavior is the subject of the test, make that exception narrow and observable. A recorded simulation, sink service, or controlled proxy may answer the research question without exposing third parties. When real external access is unavoidable, obtain authorization, restrict destinations, remove real authentication, rate-limit traffic, coordinate monitoring, and define an immediate external-contact path. An evaluation objective does not authorize probing systems the team does not own.

Assume credentials and limits will be tested

Use run-specific, short-lived identities with deny-by-default policy. The credential should be useless outside the synthetic target and should expire even if cleanup fails. Avoid human tokens and shared roles because their effective permissions change, their revocation affects unrelated work, and their presence can turn a sandbox escape into a production incident. Canary tokens can reveal forbidden access attempts, but they supplement rather than replace real isolation.

Enforce budgets outside the model and outside the agent's immediate orchestration process. Cap wall-clock duration, inference, tool calls, process count, storage, outbound bytes, request rate, and concurrent workers. Prevent the workload from creating new identities, jobs, serverless functions, virtual machines, repositories, or external persistence. A parent process limit is weak if the agent can move execution into another service.

Design stop conditions for behavior, not only cost. Examples include discovery of a forbidden route, access to a canary secret, unexpected privilege change, unauthorized package request, disabled logging, unapproved child environment, persistence attempt, or contact with a real external service. Give the on-call operator clear authority to terminate the run without waiting for the research owner.

Monitor from outside and rehearse incident response

The model's transcript is research data, not a complete audit trail. Collect telemetry at layers the workload cannot edit: identity-provider decisions, network flows and DNS, proxy and registry requests, process trees, runtime or system events, filesystem changes, orchestration actions, resource creation, and operator commands. Synchronize clocks, preserve model and scaffold versions, and send logs to a separate write-controlled account. Minimize sensitive content, but retain enough context to distinguish a benchmark attempt from a containment failure.

Alerting should assume long-horizon automation. Detection cannot wait for a weekday manual review while an agent continues thousands of steps. Page on forbidden paths, credential use, privilege escalation, unexpected egress, lateral movement, disabled sensors, and bursts of worker creation. Use benign canaries to validate detection before the high-capability run, and have an independent security team challenge the environment rather than letting its builders certify it alone.

Rehearse the response as if the environment will fail. The runbook should isolate compute and networks, preserve volatile and durable evidence, revoke every reachable credential, rebuild compromised nodes, inspect connected systems, contact vendors or affected organizations, and make legal and regulatory notification decisions. Hugging Face reported rebuilding compromised nodes, rotating credentials, adding cluster controls, and improving response alerting; NIST's draft AI cybersecurity profile emphasizes protected logs, monitoring, constrained code execution, and vetted software. These are documentation-based lessons, not evidence that one checklist can guarantee containment.

After every run, compare observed behavior with both specifications. A benchmark pass accompanied by an escape attempt is not an uncomplicated success. Record which layer prevented harm, which signals fired, which assumptions failed, and whether the same evaluation remains justified. Update the environment before increasing model capability, autonomy, tool access, attempt count, or disabled safeguards.