OpenClaw

The openclaw harness runs an OpenClaw agent inside a prepared environment’s container, driving the model-under-test through tasks one at a time — commonly used for OpenClaw-style productivity / agentic-coding benchmarks (such as PinchBench and SkillsBench).

You only need to provide the model’s access credentials; the harness handles the rest: it installs openclaw in the container, onboards your model to OpenClaw as a callable custom model, then runs the tasks one by one and collects the results. Credentials are passed on the command line via --model-base-url / --model-api-key, and both the openai-chat and openai-responses --model-api-protocol protocols are supported.

Parameters

Pass a chunk of JSON via --harness-params '{...}', or write it into the harness.params block of the YAML given to --config; on shared keys the command line wins (deep-merge override). For merge and precedence, see the Harnesses overview.

Parameter reference

Parameter	Type	Default	Choices / values	Description
binary	string	openclaw	—	Name or full path of the openclaw executable; usually no need to change.
install_strategy	string	auto	5 options	How to prepare openclaw in the container; 5 in total (see Install strategy below).
openclaw_version	string	2026.3.22	—	The openclaw version used when auto-installing; only takes effect when install_strategy triggers an install.
install_command	string	""	—	Custom install command. When empty, it is auto-assembled from openclaw_version as npm install -g openclaw@<version>.
upload_src	string	""	—	Only used when install_strategy is the upload strategy: the path to the openclaw executable on the local machine (the one running AgentCompass) to upload; required under that strategy.
upload_dst	string	/usr/local/bin/openclaw	—	Only used when install_strategy is the upload strategy: the destination path inside the container, which is also the openclaw path actually used at runtime.
provider_id	string	vllm	—	The name your model is registered under in OpenClaw; keep the default unless you need to align with a name in existing config.
gateway_port	int	18789	≥ 1	The local port OpenClaw uses to reach your model; change it only if it conflicts with another service.
gateway_bind	string	loopback	—	The gateway’s listen scope; defaults to loopback (localhost only).
openclaw_local	bool	true	true / false	Whether to run OpenClaw in local mode (—local); on by default.
brave_api_key	string	—	Injects a Brave Search API key into OpenClaw; when provided, OpenClaw can use Brave for web_search. When not explicitly provided, web_search may be limited or unavailable. See the OpenClaw Web Search documentation.
max_message_chars	int	131072	≥ 1	Maximum characters in a single message sent to the model; longer messages are automatically split into several sent in sequence.
max_tokens	int	0	≥ 0	Max output tokens for a single model reply; 0 means unset, using the model’s own default. See Context and token limits below.
context_window	int	250000	≥ 0	Tells OpenClaw how long a context your model accepts; 0 means unset. For custom models, set the real value; see Context and token limits below.
timeout	int / null	9600	≥ 1	Total time limit for a single task from start to finish (seconds); aborted on timeout. null means no limit.
provider_timeout_seconds	int	3600	—	How long the model service can stay idle before being reclaimed (seconds).

Install strategy

install_strategy decides how openclaw lands in the container:

• auto — applies an environment default (not a probing auto-install): host_process uses install_if_missing, other environments like docker use preinstalled. If openclaw isn’t preinstalled on the image in docker and similar environments, auto errors out — switch it to install_if_missing or upload manually.
• preinstalled — uses what’s already installed on the image; errors if absent.
• install_if_missing — installs only when openclaw is not present in the container.
• install_always — reinstalls on every run (some benchmarks pin install_if_missing, in which case the benchmark’s setting takes precedence).
• upload — instead of installing from npm, uploads an existing openclaw executable from your local machine (the one running AgentCompass) into the container: it uploads from upload_src to upload_dst, adds the executable bit, and uses that file at runtime (binary automatically points to upload_dst, no need to set it separately). Suited to containers that can’t reach npm, or when you want a self-compiled / pinned binary; upload_src is required in this case.

Setting openclaw_version is enough — install_command is auto-derived as npm install -g openclaw@<version>. Set install_command explicitly only when you need to point at an internal registry or a non-npm source.

Context and token limits

context_window and max_tokens are written into openclaw.json only when set greater than 0, and are independent of each other.

• context_window — the model’s total context length (corresponds to vLLM’s --max-model-len). Setting it explicitly stops OpenClaw from falling back to an over-small default window when it can’t recognize the model-under-test, which would trigger context compaction prematurely. When the value is 0, AgentCompass does not write the field into openclaw.json and the window is left to OpenClaw: if OpenClaw has a built-in context window for the model, it uses that built-in value; if it’s an unrecognized custom model, it falls back to a smaller default window and may compact the context before it has actually exceeded the limit, losing information and hurting long-task performance. Passing the model-under-test’s real context length explicitly during evaluation is therefore recommended.
• max_tokens — the per-turn completion budget; it should be noticeably smaller than context_window so that “input tokens + max_tokens” doesn’t exceed the served context length. When the value is 0 it is likewise not written into openclaw.json, and the model / server’s own default output limit applies.

Run examples

openclaw is passed as the second positional argument to agentcompass run <benchmark> openclaw <model>; harness config is passed via --harness-params.

Default configuration

OpenClaw is already installed on the image — run with the default parameters.

agentcompass run \
  pinchbench \
  openclaw \
  "$MODEL_NAME" \
  --env docker \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat

Install a pinned version

When the container isn’t preinstalled (or you want to switch versions), let the harness install a pinned version and give it a generous wall-clock timeout.

agentcompass run \
  gdpval_ac \
  openclaw \
  "$MODEL_NAME" \
  --env docker \
  --harness-params '{
    "install_strategy": "install_if_missing",
    "openclaw_version": "2026.5.7",
    "timeout": 14400
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat

Custom provider and limits

Write the context window and per-turn token budget for a custom model, and pin the provider id.

agentcompass run \
  wildclawbench \
  openclaw \
  "$MODEL_NAME" \
  --env docker \
  --harness-params '{
    "provider_id": "vllm",
    "context_window": 262144,
    "max_tokens": 32768
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat