Terminal-Bench 2

Terminal-Bench 2 evaluates whether an agent can complete realistic command-line tasks in task-specific containers. AgentCompass uses the Terminal-Bench 2.0 task set with a terminal harness, normally terminus2.

How it works

1. Load tasks. On its first run, AgentCompass shallow-clones the Terminal-Bench 2.0 repository from GitHub into the data directory. Each task supplies its instruction, container definition, and verifier.
2. Run the agent. The task’s container image and resource requirements are applied by the environment recipe. The task instruction is passed to the harness, which operates in the prepared terminal workspace.
3. Verify the result. The benchmark runs the task’s tests/test.sh through the Harbor verifier. A verifier reward of 1 is recorded as correct.

Parameters

Configure Terminal-Bench-specific options with --benchmark-params '{...}'.

Parameter	Type	Default	Description
verifier_timeout_multiplier	float	1.0	Multiplier applied to the verifier timeout.
execute_timeout_multiplier	float	1.0	Multiplier applied to the task-level agent execution timeout.

Run examples

agentcompass run takes the benchmark id, harness id, and model id in that order. The default configuration uses the local docker environment; the recipe applies each task’s image automatically.

Run configuration is split into two JSON blocks: --benchmark-params carries Terminal-Bench configuration (the timeout multipliers above and optional task selection), and --harness-params carries the selected harness’s own configuration. The examples below use terminus2, whose relevant settings include max_turns and timeout. Both blocks can instead be written to benchmark.params and harness.params in --config; command-line values take precedence on shared keys.

Recommended harness

The recommended terminal agent is terminus2.

Smoke test (single task end-to-end)

Verify the end-to-end flow works — sample_ids selects which case to run, with all other parameters using their defaults.

agentcompass run \
  terminal_bench_2 \
  terminus2 \
  "$MODEL_NAME" \
  --env docker \
  --benchmark-params '{"sample_ids":["<task-id>"]}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY"

Custom parameters

Raise the timeout multipliers for a runtime environment with slower model responses.

agentcompass run \
  terminal_bench_2 \
  terminus2 \
  "$MODEL_NAME" \
  --env docker \
  --benchmark-params '{
    "verifier_timeout_multiplier": 8,
    "execute_timeout_multiplier": 16
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY"

AgentCompass recommended config

Run a full evaluation with the AgentCompass recommended configuration. If model service conditions, deployment instance count, task concurrency, or model capability increase task duration, pass verifier_timeout_multiplier and execute_timeout_multiplier through --benchmark-params.

agentcompass run \
  terminal_bench_2 \
  terminus2 \
  "$MODEL_NAME" \
  --env docker \
  --task-concurrency 16 \
  --harness-params '{"max_turns":300,"timeout":14400}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY"

Other optional harnesses

codex and claude_code are two other harness options. Pass --recipe terminalbench2_docker_ac to use the AgentCompass prebuilt image. It includes download dependencies such as Node.js, npm, curl, and wget for Codex, Claude Code, and similar harnesses.

Run with the official image

Omit --recipe to use the official task image. Because it does not include the Node bootstrap dependencies, provide the matching installation command explicitly.

# Codex
agentcompass run terminal_bench_2 codex "$MODEL_NAME" \
  --env docker \
  --task-concurrency 16 \
  --harness-params '{"install_command":"apt-get update && apt-get install -y curl ca-certificates && curl -fsSL https://deb.nodesource.com/setup_20.x | bash - && apt-get install -y nodejs && npm install -g @openai/codex"}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-responses

# Claude Code
agentcompass run terminal_bench_2 claude_code "$MODEL_NAME" \
  --env docker \
  --task-concurrency 16 \
  --harness-params '{"install_command":"apt-get update && apt-get install -y curl ca-certificates && curl -fsSL https://deb.nodesource.com/setup_20.x | bash - && apt-get install -y nodejs && npm install -g @anthropic-ai/claude-code"}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol anthropic

AgentCompass recommended config

Pass --recipe terminalbench2_docker_ac to select the AgentCompass prebuilt image without explicitly providing the corresponding installation command.

# Codex
agentcompass run terminal_bench_2 codex "$MODEL_NAME" \
  --env docker \
  --recipe terminalbench2_docker_ac \
  --task-concurrency 16 \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-responses

# Claude Code
agentcompass run terminal_bench_2 claude_code "$MODEL_NAME" \
  --env docker \
  --recipe terminalbench2_docker_ac \
  --task-concurrency 16 \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol anthropic

Output

A run produces two kinds of results under results/terminal_bench_2/<model>/<run>/: aggregate metrics in summary.md and one JSON record per task in details/.

Aggregate metrics (summary.md)

summary.md contains the run overview (Model, Total, Evaluated, and Error) and its headline metric, accuracy. accuracy is the share of evaluated tasks for which the Harbor verifier returns the full reward (1), so it is the task pass rate for Terminal-Bench.

Per-task details (details/)

Each task JSON records correct, execution status, attempts, the agent trajectory and harness metrics, plus the raw verifier output used to determine the result. See Results.