Claude Code

The claude_code harness runs Claude Code non-interactively in a benchmark-prepared workspace.

​

How it works

• Prepare Claude Code CLI. An optional Runtime Capsule first supplies a generic toolchain; the harness then uses the existing CLI, installs on demand, or uploads an executable through the original strategy.
• Configure and run. The harness writes a session-scoped Anthropic-compatible model configuration and invokes Claude Code headlessly. Recipes may pass an external settings file so credentials cannot enter a generated repository patch.
• Collect results. Claude Code JSON records are normalized into the trajectory, final answer, and execution status in a RunResult.

​

Parameters

Pass harness options with --harness-params '{...}'. The table exposes the installation and agent behavior that normally needs tuning; path and upload details use built-in defaults.

​

Parameter reference

Parameter	Type	Default	Choices / values	Description
install_strategy	string	install_if_missing	preinstalled / install_if_missing / upload	How to prepare Claude Code.
install_command	string	npm install -g @anthropic-ai/claude-code	shell command	Command used when Claude Code is missing; select the Claude Code version here.
setup_capsule_tag	string	unset	node24-v1	Optional predefined toolchain Capsule injected before Claude Code installation.
max_turns	int	400	integer ≥ 1	Maximum agent turns for one task.
max_output_tokens	int / null	unset	integer ≥ 1 or null	Maximum output tokens for one response. When unset, the harness uses max_tokens from model params if present; otherwise Claude Code uses its default.
max_context_tokens	int / null	unset	integer ≥ 1 or null	Context window size that Claude Code assumes for the model.
auto_compact_window	int / null	unset	integer ≥ 1 or null	Token threshold at which Claude Code automatically compacts the conversation.
append_system_prompt	string	unset	any string	Additional system prompt appended for Claude Code.
dangerously_skip_permissions	bool	true	true / false	Whether to pass —dangerously-skip-permissions. Set it to false only when Claude Code permission prompts must be retained.

​

Model API

Claude Code requires an Anthropic-compatible --model-base-url and --model-api-key, with --model-api-protocol anthropic. When the base URL ends in /v1, the harness removes that suffix before writing the Claude Code configuration.

When model params are nonempty, the harness serializes the complete object into CLAUDE_CODE_EXTRA_BODY. This forwards request fields such as temperature and top_p to Claude Code’s API request. If model params contain max_tokens and the harness parameter max_output_tokens is unset, the same positive value also configures CLAUDE_CODE_MAX_OUTPUT_TOKENS; an explicit harness value takes precedence. The max_context_tokens and auto_compact_window harness parameters configure CLAUDE_CODE_MAX_CONTEXT_TOKENS and CLAUDE_CODE_AUTO_COMPACT_WINDOW, respectively.

Planning validates JSON serialization and the positive-integer max_tokens constraint. An explicit Harness max_output_tokens overrides both max_tokens in EXTRA_BODY and the output-limit environment variable; max_tokens=null is omitted from the request body. Full forwarding activates previously ignored fields such as temperature and top_p in existing configurations.

Benchmarks can provide typed stdio, SSE, or streamable-http MCP connections and a skills directory. The Harness registers these servers through a dedicated MCP configuration and copies skills into a task-specific Claude Code config directory without changing official instructions.

​

Run examples

Default

Run with the default installation and agent behavior.

agentcompass run \
  swebench_verified \
  claude_code \
  "$MODEL_NAME" \
  --env docker \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol anthropic

Custom params

Forward model request fields and adjust Claude Code’s turn, context, and compaction limits.

agentcompass run \
  swebench_verified \
  claude_code \
  "$MODEL_NAME" \
  --env docker \
  --harness-params '{
    "max_turns": 80,
    "max_context_tokens": 131072,
    "auto_compact_window": 100000,
    "append_system_prompt": "Prefer minimal diffs."
  }' \
  --model-params '{
    "max_tokens": 16384,
    "temperature": 0,
    "top_p": 0.95
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol anthropic

Runtime Capsule

Use a prebuilt Capsule in an Environment without compatible Node, npm, or glibc.

agentcompass run \
  swebench_verified \
  claude_code \
  "$MODEL_NAME" \
  --env docker \
  --harness-params '{
    "install_strategy": "install_if_missing",
    "install_command": "npm install -g @anthropic-ai/claude-code@<version>",
    "setup_capsule_tag": "node24-v1"
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol anthropic

See Runtime Capsules for build and distribution details.

​

Output

The harness returns a RunResult per task: trajectory, final answer, and execution status. Per-task details and aggregate metrics are written by the benchmark under the run directory (see Results).

High-frequency system/thinking_tokens progress events are filtered from both saved stdout and raw events before checkpointing. Assistant messages, tool results, and the final result event, including its usage statistics, are retained.

Set the task execution deadline through --execution-params with run_timeout_seconds and run_timeout_multiplier. See phase timeouts.