Terminal-Bench 2.1

Terminal-Bench 2.1 is the AgentCompass entry for the Terminal-Bench 2.1 task collection. It uses the same containerized execution and verification flow as Terminal-Bench 2, normally with the terminus2 harness.

How it works

1. Load tasks. With the default dataset address, AgentCompass downloads Terminal-Bench 2.1 through the Harbor CLI. An explicitly configured regular Git source is shallow-cloned instead.
2. Run the agent. The environment recipe selects the image declared by each task and starts the agent in its prepared terminal workspace.
3. Verify the result. The task’s tests/test.sh is run through the Harbor verifier. A reward of 1 marks the task correct.

Parameters

Configure Terminal-Bench-specific options with --benchmark-params '{...}'.

Parameter	Type	Default	Description

Run examples

Use --benchmark-params for dataset and judge settings, --harness-params for agent and tool settings, and --execution-params for phase timeouts and multipliers. YAML uses benchmark.params, harness.params, and execution; explicit CLI values override YAML values.

Recommended harness

The recommended terminal agent is terminus2.

Smoke test (single task end-to-end)

Verify the end-to-end flow works — sample_ids selects which case to run, with all other parameters using their defaults.

agentcompass run \
  terminal_bench_2_1 \
  terminus2 \
  "$MODEL_NAME" \
  --env docker \
  --benchmark-params '{"sample_ids":["<task-id>"]}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY"

Custom parameters

Raise the timeout multipliers for a runtime environment with slower model responses.

agentcompass run \
  terminal_bench_2_1 \
  terminus2 \
  "$MODEL_NAME" \
  --env docker \
  --execution-params '{"evaluation_timeout_multiplier": 8, "run_timeout_multiplier": 16}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY"

AgentCompass recommended config

Run a full evaluation with the AgentCompass recommended configuration. If model service conditions, deployment instance count, task concurrency, or model capability increase task duration, pass evaluation_timeout_multiplier and run_timeout_multiplier through --execution-params.

agentcompass run \
  terminal_bench_2_1 \
  terminus2 \
  "$MODEL_NAME" \
  --env docker \
  --task-concurrency 16 \
  --harness-params '{"max_turns": 300}' \
  --execution-params '{"run_timeout_seconds": 14400}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY"

Other optional harnesses

codex and claude_code are two other harness options. Pass --recipe terminalbench2_1_docker_ac to use the AgentCompass prebuilt image. It includes download dependencies such as Node.js, npm, curl, and wget for Codex, Claude Code, and similar harnesses.

Run with the official image

Omit --recipe to use the official task image. Because it does not include the Node bootstrap dependencies, provide the matching installation command explicitly.

# Codex
agentcompass run terminal_bench_2_1 codex "$MODEL_NAME" \
  --env docker \
  --task-concurrency 16 \
  --harness-params '{"install_command":"apt-get update && apt-get install -y curl ca-certificates && curl -fsSL https://deb.nodesource.com/setup_20.x | bash - && apt-get install -y nodejs && npm install -g @openai/codex"}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-responses

# Claude Code
agentcompass run terminal_bench_2_1 claude_code "$MODEL_NAME" \
  --env docker \
  --task-concurrency 16 \
  --harness-params '{"install_command":"apt-get update && apt-get install -y curl ca-certificates && curl -fsSL https://deb.nodesource.com/setup_20.x | bash - && apt-get install -y nodejs && npm install -g @anthropic-ai/claude-code"}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol anthropic

AgentCompass recommended config

Pass --recipe terminalbench2_1_docker_ac to select the AgentCompass prebuilt image without explicitly providing the corresponding installation command.

# Codex
agentcompass run terminal_bench_2_1 codex "$MODEL_NAME" \
  --env docker \
  --recipe terminalbench2_1_docker_ac \
  --task-concurrency 16 \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-responses

# Claude Code
agentcompass run terminal_bench_2_1 claude_code "$MODEL_NAME" \
  --env docker \
  --recipe terminalbench2_1_docker_ac \
  --task-concurrency 16 \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol anthropic

Output

A run writes per-task details and the aggregate views summary.md and metrics.json under the run directory.

Aggregate metrics (summary.md)

The primary metric is binary correct: the Harbor verifier’s full reward (1) maps to true. At k=1, correct.native@1 is Terminal-Bench’s pass rate over evaluated observations. At k>1, the generic reducers can emit correct.avg@k and correct.pass@k, each with independent counts.

Per-task details (details/)

Each task JSON stores the binary observation at attempts.<N>.metrics.correct, together with execution status, the agent trajectory, Harness diagnostics, and raw verifier evidence. See Results.