AI Agent Evaluation Sandbox Checklist: Contain Tool-Using Models

Threat-model the evaluation, not only the model

An agent evaluation is a software system with a model inside it. The meaningful security boundary includes the scaffold, tools, code runner, filesystem, kernel, network, package sources, secrets, identity services, grader, logging pipeline, orchestration layer, and every external service the run can influence. Testing only whether the model refuses a dangerous prompt says little about whether the complete evaluation can contain unexpected behavior.

Start with assets and trust boundaries. Identify production data, model weights, hidden answers, employee credentials, cloud metadata, source repositories, CI systems, other tenants, third-party services, and the host itself. For each asset, ask how the agent could discover it, reach it, modify it, or use it as a stepping stone. Include indirect routes: a package cache is a network service; a webhook creates outbound communication; a public code runner can become a second execution environment; and a dataset processor can expose a separate trust domain.

Then define the permitted task and the exact point at which exploration becomes a security event. Capability evaluations sometimes deliberately reduce production safeguards so researchers can measure an upper bound. That choice increases the importance of controls outside the model. Document who may authorize the change, which additional containment applies, how long the run may continue, and who can terminate it. This article translates public incident disclosures and NIST guidance into operational questions; it does not claim first-hand testing of any vendor environment.

• Assets the agent must never read, modify, or reach
• Tools and protocols required for legitimate task completion
• Plausible escape, contamination, grader-gaming, and persistence paths
• Automatic and human stop conditions
• Evidence needed to investigate an anomalous run

Build isolation around identity, network, and lifecycle

Use a fresh, disposable execution boundary for every run. For arbitrary code execution, a hardened virtual machine or equivalent strongly isolated workload is a safer starting point than relying on an application container alone. Separate the evaluation control plane from the workload, patch the host and virtualization layer, minimize the base image, mount only required files, prohibit privileged execution, and destroy the environment after evidence has been exported. Never clone a contaminated run into the next test.

Network policy should begin with deny-all. If package installation is necessary, prefer a curated, immutable mirror containing pinned artifacts rather than a general proxy that can fetch arbitrary names or protocols. Disable unnecessary DNS and metadata access. When a benchmark requires websites or APIs, use recorded responses or synthetic services when possible; otherwise allow named destinations through an authenticated gateway that records requests and enforces methods, sizes, rates, and content rules. Test the policy from inside the workload rather than assuming the configuration describes reality.

Credentials are capabilities. Give each run a unique identity that expires quickly and is accepted only by synthetic evaluation services. Do not place cloud, repository, employee, production, or cross-environment credentials in images, environment variables, caches, logs, example files, or metadata reachable from the sandbox. Canary tokens and decoy secrets can reveal discovery attempts, but they supplement rather than replace denial. Revoke the identity automatically when the run ends or a tripwire fires.

Put budgets, telemetry, and shutdown outside the agent

A capable agent can make thousands of small decisions, retry after failure, encode data, create child processes, and use ordinary services in unintended ways. Bound the experiment with independently enforced limits: wall-clock time, model calls, tokens, concurrent processes, CPU or accelerator time, memory, disk writes, file count, outbound bytes, requests, tool calls, and monetary spend. A model instruction to stop is not an enforcement mechanism.

Collect evidence at multiple layers. Preserve prompts, responses, reasoning summaries when available, tool parameters and results, shell commands, process trees, file changes, network flows, DNS requests, authentication events, resource consumption, grader decisions, policy denials, and operator actions. Send records to an append-only destination the evaluated workload cannot edit. Synchronize clocks and attach the exact model ID, inference configuration, system prompt, agent scaffold, tool versions, benchmark commit, image digest, policy version, and run identity.

Tripwires should detect boundary testing early: reading credential paths, querying metadata, scanning networks, spawning unexpected listeners, changing tests, disabling assertions, accessing answer keys, installing unapproved artifacts, establishing persistence, escalating privilege, or contacting an unapproved destination. Decide in advance whether each event blocks the action, pauses the run, or terminates and isolates it. Keep an operator-accessible kill switch independent of the model, its tools, and the workload network.

Protect evaluation validity as well as infrastructure

Containment and measurement integrity are connected. NIST describes solution contamination, where a system obtains information that improperly reveals an answer, and grader gaming, where it exploits the scoring implementation instead of completing the intended task. An agent that retrieves a walkthrough, edits a test, crashes a target, or reads hidden solutions may receive a high score while demonstrating a different capability from the one the benchmark claims to measure.

Keep answer keys and grader credentials outside the workload. Give the agent a narrow submission interface, validate outputs in a separate environment, and prevent it from modifying tests, clocks, resource accounting, or scoring code. Version the benchmark and all scaffolding so results remain comparable. Use private or sequestered items where appropriate, but recognize that secrecy alone does not fix a weak task specification or an exploitable grader.

Inspect trajectories, not only final scores. Sample normal runs and manually review every anomalous success, boundary alert, unexplained speedup, or result that conflicts with other measurements. Test the grader with deliberately invalid strategies. Maintain benign controls so tighter security cannot appear to improve safety merely by making the task impossible. Report affordances, restrictions, retries, budgets, exclusions, uncertainty, and observed cheating so decision-makers understand what the number does and does not support.

Rehearse containment and recover before increasing capability

Run a containment drill with a harmless scripted workload before evaluating a highly capable agent. Confirm that forbidden egress is blocked, tripwires alert, resource limits stop abuse, credentials expire, the operator can terminate the run, logs survive destruction, and the environment returns to a known-good state. Include failure of the logging collector, policy engine, orchestrator, and identity service; a control that fails open under pressure is not a dependable boundary.

Predefine incident roles and evidence handling. When a boundary is crossed, stop issuing work, isolate affected systems, revoke identities, preserve volatile and durable evidence, identify every reachable dependency, notify affected service owners, and avoid destroying artifacts needed for investigation. Rotate any real credential that may have been exposed even if logs do not show use. Rebuild from trusted sources instead of attempting to clean an uncertain environment.

Expand one dimension at a time: stronger model, more inference budget, broader tools, richer data, reduced refusals, or additional network access. Re-run containment tests after every change to the model, scaffold, image, dependency, proxy, host, tool, grader, or orchestration layer. Security approval should depend on demonstrated isolation and recovery, not confidence that the agent is aligned with the evaluator's intended objective.