WildClawBench

WildClawBench (arxiv) evaluates an agent on real-world, long-horizon productivity tasks in executable workspaces. AgentCompass uses OpenClaw to perform each task and runs the task’s Automated Checks afterward. If its optional Python dependency is missing, AgentCompass reports the required extra and installation command; see Dependencies.

How it works

1. Prepare the task. AgentCompass downloads and validates the dataset when necessary. The Docker recipe selects the WildClawBench OpenClaw image, prepares the public task data, skills, warm-up commands, and task workspace, while keeping private ground truth outside the inference environment.
2. Run OpenClaw. The prompt and task-specific timeout are passed to the OpenClaw harness, which operates in the prepared workspace. WildClawBench requires a Brave Search credential.
3. Run Automated Checks. After inference, AgentCompass decrypts and uploads only the current task’s ground truth, executes the task’s Automated Checks inside the same environment, and converts overall_score into the task score.

Parameters

Configure WildClawBench-specific options with --benchmark-params '{...}'.

Parameter	Type	Default	Choices / values	Description
category	string / list	”all"	"all”, one category, or a list	Filter tasks by category; a list takes the union.
pass_threshold	float	1.0	numeric score	Minimum Automated Checks score required for correct=true when execution and grading succeed.
grading_timeout_seconds	int	300	integer ≥ 1	Wall-clock timeout for the Automated Checks runner.

Run examples

Run configuration is split into two JSON blocks: --benchmark-params carries WildClawBench filtering and grading settings, while --harness-params carries OpenClaw settings such as the Brave Search key, context window, and timeout. Both can also be written under benchmark.params and harness.params in --config.

Smoke test (single task end-to-end)

Verify the end-to-end flow works — sample_ids selects which case to run, with all other parameters using their defaults.

agentcompass run \
  wildclawbench \
  openclaw \
  "$MODEL_NAME" \
  --env docker \
  --benchmark-params '{"sample_ids":["<task-id>"]}' \
  --harness-params '{"brave_api_key":"${BRAVE_API_KEY}"}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY"

Custom parameters

Filter by category, adjust the pass threshold and grading timeout, and provide explicit OpenClaw context and task limits.

agentcompass run \
  wildclawbench \
  openclaw \
  "$MODEL_NAME" \
  --env docker \
  --benchmark-params '{
    "category": "<category>",
    "pass_threshold": 0.8,
    "grading_timeout_seconds": 600
  }' \
  --harness-params '{
    "brave_api_key": "${BRAVE_API_KEY}",
    "context_window": 262144,
    "timeout": 14400
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY"

AgentCompass recommended config

Run a full evaluation with the AgentCompass recommended configuration. The Docker recipe supplies the task image; provide the required Brave Search credential through --harness-params.

agentcompass run \
  wildclawbench \
  openclaw \
  "$MODEL_NAME" \
  --env docker \
  --harness-params '{
    "brave_api_key": "${BRAVE_API_KEY}",
    "context_window": 262144,
    "timeout": 14400
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --task-concurrency 16

Outputs

A run writes aggregate metrics and per-task details under results/wildclawbench/<model>/<run>/.

Aggregate metrics (summary.md)

summary.md contains the run counts (Total, Evaluated, and Error) and the headline metric mean_score: the arithmetic mean of each task’s Automated Checks score. Category-level mean scores are included when categories are present.

Per-task details (details/)

Each task JSON records score, correct, execution status, trajectory, and harness artifacts. Automated Checks output is stored under attempts[*].extra.scoring, including the normalized score, notes, raw grading payload, and any grading error.