> ## Documentation Index
> Fetch the complete documentation index at: https://agent-compass.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Terminal-Bench 2

Terminal-Bench 2 evaluates whether an agent can complete realistic command-line tasks in task-specific containers. AgentCompass uses the [Terminal-Bench 2.0](https://github.com/harbor-framework/terminal-bench-2) task set with a terminal harness, normally [`terminus2`](/en/user_guide/modules/harnesses/terminus2).

## How it works

1. **Load tasks.** On its first run, AgentCompass shallow-clones the Terminal-Bench 2.0 repository from GitHub into the data directory. Each task supplies its instruction, container definition, and verifier.
2. **Run the agent.** The task's container image and resource requirements are applied by the environment recipe. The task instruction is passed to the harness, which operates in the prepared terminal workspace.
3. **Verify the result.** The benchmark runs the task's `tests/test.sh` through the Harbor verifier. A verifier reward of `1` is recorded as `correct`.

## Parameters

Configure Terminal-Bench-specific options with `--benchmark-params '{...}'`.

| Parameter                     | Type  | Default | Description                                                   |
| ----------------------------- | ----- | ------- | ------------------------------------------------------------- |
| `verifier_timeout_multiplier` | float | `1.0`   | Multiplier applied to the verifier timeout.                   |
| `execute_timeout_multiplier`  | float | `1.0`   | Multiplier applied to the task-level agent execution timeout. |

## Run examples

`agentcompass run` takes the benchmark id, harness id, and model id in that order. The default configuration uses the local `docker` environment; the recipe applies each task's image automatically.

Run configuration is split into two JSON blocks: `--benchmark-params` carries Terminal-Bench configuration (the timeout multipliers above and optional task selection), and `--harness-params` carries the selected harness's own configuration. The examples below use [`terminus2`](/en/user_guide/modules/harnesses/terminus2), whose relevant settings include `max_turns` and `timeout`. Both blocks can instead be written to `benchmark.params` and `harness.params` in `--config`; command-line values take precedence on shared keys.

### Recommended harness

The recommended terminal agent is [`terminus2`](/en/user_guide/modules/harnesses/terminus2).

<Tabs>
  <Tab title="Smoke test (single task end-to-end)">
    Verify the end-to-end flow works — `sample_ids` selects which case to run, with all other parameters using their defaults.

    ```bash theme={"system"}
    agentcompass run \
      terminal_bench_2 \
      terminus2 \
      "$MODEL_NAME" \
      --env docker \
      --benchmark-params '{"sample_ids":["<task-id>"]}' \
      --model-base-url "$MODEL_BASE_URL" \
      --model-api-key "$MODEL_API_KEY"
    ```
  </Tab>

  <Tab title="Custom parameters">
    Raise the timeout multipliers for a runtime environment with slower model responses.

    ```bash theme={"system"}
    agentcompass run \
      terminal_bench_2 \
      terminus2 \
      "$MODEL_NAME" \
      --env docker \
      --benchmark-params '{
        "verifier_timeout_multiplier": 8,
        "execute_timeout_multiplier": 16
      }' \
      --model-base-url "$MODEL_BASE_URL" \
      --model-api-key "$MODEL_API_KEY"
    ```
  </Tab>

  <Tab title="AgentCompass recommended config">
    Run a full evaluation with the AgentCompass recommended configuration. If model service conditions, deployment instance count, task concurrency, or model capability increase task duration, pass `verifier_timeout_multiplier` and `execute_timeout_multiplier` through `--benchmark-params`.

    ```bash theme={"system"}
    agentcompass run \
      terminal_bench_2 \
      terminus2 \
      "$MODEL_NAME" \
      --env docker \
      --task-concurrency 16 \
      --harness-params '{"max_turns":300,"timeout":14400}' \
      --model-base-url "$MODEL_BASE_URL" \
      --model-api-key "$MODEL_API_KEY"
    ```
  </Tab>
</Tabs>

### Other optional harnesses

[`codex`](/en/user_guide/modules/harnesses/codex) and [`claude_code`](/en/user_guide/modules/harnesses/claude_code) are two other harness options. Pass `--recipe terminalbench2_docker_ac` to use the AgentCompass prebuilt image. It includes download dependencies such as Node.js, npm, curl, and wget for Codex, Claude Code, and similar harnesses.

<Tabs>
  <Tab title="Run with the official image">
    Omit `--recipe` to use the official task image. Because it does not include the Node bootstrap dependencies, provide the matching installation command explicitly.

    ```bash theme={"system"}
    # Codex
    agentcompass run terminal_bench_2 codex "$MODEL_NAME" \
      --env docker \
      --task-concurrency 16 \
      --harness-params '{"install_command":"apt-get update && apt-get install -y curl ca-certificates && curl -fsSL https://deb.nodesource.com/setup_20.x | bash - && apt-get install -y nodejs && npm install -g @openai/codex"}' \
      --model-base-url "$MODEL_BASE_URL" \
      --model-api-key "$MODEL_API_KEY" \
      --model-api-protocol openai-responses

    # Claude Code
    agentcompass run terminal_bench_2 claude_code "$MODEL_NAME" \
      --env docker \
      --task-concurrency 16 \
      --harness-params '{"install_command":"apt-get update && apt-get install -y curl ca-certificates && curl -fsSL https://deb.nodesource.com/setup_20.x | bash - && apt-get install -y nodejs && npm install -g @anthropic-ai/claude-code"}' \
      --model-base-url "$MODEL_BASE_URL" \
      --model-api-key "$MODEL_API_KEY" \
      --model-api-protocol anthropic
    ```
  </Tab>

  <Tab title="AgentCompass recommended config">
    Pass `--recipe terminalbench2_docker_ac` to select the AgentCompass prebuilt image without explicitly providing the corresponding installation command.

    ```bash theme={"system"}
    # Codex
    agentcompass run terminal_bench_2 codex "$MODEL_NAME" \
      --env docker \
      --recipe terminalbench2_docker_ac \
      --task-concurrency 16 \
      --model-base-url "$MODEL_BASE_URL" \
      --model-api-key "$MODEL_API_KEY" \
      --model-api-protocol openai-responses

    # Claude Code
    agentcompass run terminal_bench_2 claude_code "$MODEL_NAME" \
      --env docker \
      --recipe terminalbench2_docker_ac \
      --task-concurrency 16 \
      --model-base-url "$MODEL_BASE_URL" \
      --model-api-key "$MODEL_API_KEY" \
      --model-api-protocol anthropic
    ```
  </Tab>
</Tabs>

## Output

A run produces two kinds of results under `results/terminal_bench_2/<model>/<run>/`: aggregate metrics in `summary.md` and one JSON record per task in `details/`.

### Aggregate metrics (summary.md)

`summary.md` contains the run overview (`Model`, `Total`, `Evaluated`, and `Error`) and its headline metric, **`accuracy`**. `accuracy` is the share of evaluated tasks for which the Harbor verifier returns the full reward (`1`), so it is the task pass rate for Terminal-Bench.

### Per-task details (details/)

Each task JSON records `correct`, execution status, attempts, the agent trajectory and harness metrics, plus the raw verifier output used to determine the result. See [Results](/en/user_guide/results).