OpenHands

The openhands harness runs OpenHands against the repository workspace prepared by a benchmark. It is intended for repository-repair benchmarks such as SWE-bench Verified, SWE-bench Multilingual, SWE-bench Pro, and SWE-bench Pro Verified. It can also serve as a terminal-operation harness for benchmarks such as Terminal-Bench 2.

AgentCompass installs a pinned OpenHands SDK/tools runtime inside the selected environment, passes the issue prompt and model endpoint to OpenHands, forwards terminal actions to the prepared workspace, and converts the OpenHands event history into the standard RunResult trajectory. The model under test is configured by the CLI --model-* flags; both openai-chat and openai-responses are supported.

Set agent setup/run environment variables with --env-params '{"run_env_variables":{"MY_VARIABLE":"value"}}'. The former Harness env field is no longer accepted. These variables apply to installation, agent execution, and its tools; they are removed from subsequent artifact and verifier commands. See Task environment variables.

How it works

1. Prepare an isolated runtime. By default, the harness creates /opt/agentcompass/openhands/runtime through micromamba. With setup_capsule_tag, a predefined Python 3.12 Capsule installs packages under /tmp/agentcompass-capsule-setups/<bundle-sha256> instead. Both paths install openhands-sdk / openhands-tools at openhands_version, probe the imports, and upload the AgentCompass entrypoint.
2. Build the OpenHands conversation. The benchmark-provided prompt and workspace are passed to an OpenHands Conversation. tool_preset selects the terminal/editor tool set, and an optional condenser summarizes older events. max_iterations limits iteration count; conversation_timeout controls one LLM request, command_timeout controls one terminal command, terminal_no_change_timeout_seconds is the soft limit after output stops changing, and terminal_max_output_size truncates terminal output returned to the agent.
3. Condense context. With enable_condenser=true, OpenHands uses an LLM summarizing condenser. condenser_max_size controls the maximum context event count, and condenser_keep_first preserves the earliest events.
4. Run tools in the task workspace. Terminal actions execute through the selected AgentCompass environment. The harness keeps live state under <workspace>/.agentcompass/, so partial history and an active terminal command or model request can still be recovered when a run times out.
5. Collect the submission. SWE-style tasks normally request a patch file such as patch.txt; the first requested file collected successfully becomes final_answer. If no output file was requested, the OpenHands finish message is used instead.

Timeout layers

The timeout fields are independent and the first applicable limit to fire stops that operation:

Layer	Configuration	Default	Scope
LLM request	--model-params.timeout, otherwise conversation_timeout	3600 seconds	One agent or condenser LLM request, including the time spent waiting for that request. An explicit model timeout wins over conversation_timeout.
Terminal no-change	terminal_no_change_timeout_seconds	600 seconds	OpenHands soft limit after a terminal action stops producing changed output.
Terminal command	command_timeout	1800 seconds	Hard limit for one terminal action. null disables this per-command limit.
Agent loop	max_iterations	250 iterations	Maximum OpenHands conversation iterations; this is a count limit, not a duration.
Whole inference task	--execution-params.run_timeout_seconds	null (inherit)	Task/Benchmark budget, falling back to 9600 seconds. null inherits this budget.
Evaluation timeout	--execution-params.evaluation_timeout_seconds	benchmark-specific	Fresh benchmark evaluation after the harness returns a patch. It does not extend or replace any inference timeout above.

For example, if run_timeout_seconds=7200 and one model request has --model-params '{"timeout":9000}', the outer 7200-second task limit can still terminate the run first. When the outer timeout fires, RunResult is a run error, but partial trajectory and timeout diagnostics are retained when available.

Parameters

Pass a JSON object via --harness-params '{...}', or use harness.params in a YAML file given to --config; the CLI wins on shared keys. See the Harnesses overview for merge precedence.

Parameter reference

Parameter	Type	Default	Choices / values	Description
openhands_version	string	1.23.0	OpenHands SDK/tools version	Version installed into the isolated runtime. Keep it pinned for comparable runs.
setup_capsule_tag	string	unset	python312-v1	Optional predefined Python 3.12 Capsule used instead of micromamba runtime bootstrap.
tool_preset	string	default	default / gemini / gpt5 / planning	OpenHands tool preset used by the agent.
max_iterations	int	250	integer ≥ 1	Maximum conversation iterations for one task.
conversation_timeout	int	3600	integer ≥ 1	Default timeout for one LLM request, in seconds.
command_timeout	int / null	1800	integer ≥ 1 or null	Hard timeout for one terminal command, in seconds.
terminal_no_change_timeout_seconds	int	600	integer ≥ 1	Soft timeout after terminal output stops changing.
terminal_max_output_size	int	200000	integer ≥ 1	Maximum terminal-output characters returned to the agent.
enable_condenser	bool	true	true / false	Enable the LLM summarizing condenser.
interleaved_thinking	bool	false	true / false	Whether to include tool use and other reasoning state returned by the previous model request in the next request.
condenser_max_size	int	240	integer ≥ 1	Maximum event count before condenser processing.
condenser_keep_first	int	2	integer ≥ 1	Number of earliest events preserved by the condenser.
skill_dirs	list	[]	directory paths	OpenHands skill directories. Paths must exist inside the execution environment.

Model request parameters

--model-params is passed to the OpenHands SDK LLM constructor for both the main agent and the optional condenser. It is a separate JSON object from --harness-params.

Parameter	AgentCompass default	Scope and precedence
temperature	unset	Sampling temperature. The provider / model default applies when omitted.
max_output_tokens	unset	Maximum output tokens for one OpenHands LLM response. This is not the whole-task token budget.
num_retries	OpenHands SDK default (4)	Maximum SDK request retries; this is not AgentCompass task-attempt k.
retry_min_wait	OpenHands SDK default (5)	Minimum retry wait in seconds.
retry_max_wait	OpenHands SDK default (30)	Maximum retry wait in seconds.
retry_multiplier	OpenHands SDK default (2)	Exponential retry backoff multiplier.
reasoning_effort	unset	OpenHands reasoning level: none, low, medium, high, or xhigh, subject to model/provider support.
reasoning_summary	unset	Optional reasoning summary mode: auto, concise, or detailed, subject to provider support.
extended_thinking_budget	unset	Extended-thinking token budget for compatible providers such as Anthropic.
extra_body	unset	Provider-specific OpenAI-compatible request fields. AgentCompass maps it to OpenHands litellm_extra_body.

A practical request and retry configuration is:

--model-params '{
  "temperature": 0,
  "max_output_tokens": 32768,
  "timeout": 3600,
  "num_retries": 10,
  "retry_min_wait": 8,
  "retry_max_wait": 64,
  "retry_multiplier": 2
}'

Thinking and reasoning

There is no OpenHands harness parameter named thinking; configure reasoning under --model-params. Choose the form expected by the model server:

Reasoning effort

Use OpenHands’ typed reasoning fields for a provider that supports reasoning effort:

--harness-params '{"interleaved_thinking":true}' \
--model-api-protocol openai-chat \
--model-params '{
  "max_output_tokens": 32768,
  "reasoning_effort": "high",
  "reasoning_summary": "auto"
}'

In particular, with openai-chat, interleaved_thinking==true bypasses OpenHands’ model-name allowlist, preserves each assistant response’s reasoning_content, and sends it back to the server.

Responses API

Use reasoning_effort and reasoning_summary to configure Responses API generation:

--harness-params '{"interleaved_thinking":true}' \
--model-api-protocol openai-responses \
--model-params '{
  "max_output_tokens": 32768,
  "reasoning_effort": "high",
  "reasoning_summary": "auto"
}'

In particular, with openai-responses, interleaved_thinking==true replays the reasoning item returned by the server (the Responses output item with type: "reasoning" that carries reasoning state; see the OpenAI documentation). AgentCompass wraps OpenHands’ message-formatting logic here: true serializes prior reasoning items back into input, while false filters them out.

vLLM / Qwen thinking switch

OpenAI-compatible vLLM reasoning endpoints commonly expose the model’s chat-template switch through extra_body. The server must also be launched with the matching reasoning and tool-call parsers.

--model-params '{
  "max_output_tokens": 32768,
  "extra_body": {
    "chat_template_kwargs": {
      "enable_thinking": true
    }
  }
}'

Anthropic extended thinking

For a compatible Anthropic model, use the OpenHands extended-thinking budget:

--model-params '{
  "max_output_tokens": 32768,
  "extended_thinking_budget": 8192
}'

These forms are provider-specific. Do not send all of them together unless the serving backend documents that combination. Thinking tokens also consume the model’s output/context budget, so increase max_output_tokens and the served context window together when necessary.

Run examples

Default configuration

Run one SWE-bench Verified task using the default OpenHands parameters.

agentcompass run \
  swebench_verified \
  openhands \
  "$MODEL_NAME" \
  --env docker \
  --benchmark-params '{"sample_ids":["astropy__astropy-12907"]}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat

Custom parameters

Customize every inference timeout layer, request retries, reasoning, condenser behavior, and optional skills.

agentcompass run \
  swebench_verified \
  openhands \
  "$MODEL_NAME" \
  --env docker \
  --benchmark-params '{"sample_ids":["astropy__astropy-12907"]}' \
  --harness-params '{"max_iterations": 150, "conversation_timeout": 3600, "command_timeout": 1200, "terminal_no_change_timeout_seconds": 600, "enable_condenser": false, "interleaved_thinking": true, "skill_dirs": ["/opt/agent-skills"]}' \
  --execution-params '{"run_timeout_seconds": 7200}' \
  --model-params '{
    "temperature": 0,
    "max_output_tokens": 32768,
    "timeout": 3600,
    "reasoning_effort": "high",
    "num_retries": 10,
    "retry_min_wait": 8,
    "retry_max_wait": 64,
    "retry_multiplier": 2
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat

Runtime Capsule

Use the stable Python 3.12 Capsule and install only the OpenHands packages during setup.

agentcompass run \
  swebench_verified \
  openhands \
  "$MODEL_NAME" \
  --env docker \
  --benchmark-params '{"sample_ids":["astropy__astropy-12907"]}' \
  --harness-params '{
    "openhands_version": "1.23.0",
    "setup_capsule_tag": "python312-v1"
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat

Output

The harness returns one RunResult per task:

• final_answer — the first requested output file, normally the submitted patch for SWE-style benchmarks;
• trajectory — the normalized OpenHands conversation and tool history, including partial history on supported timeout paths;
• artifacts.file — all requested files that were collected;
• artifacts.openhands — raw status, error, finish message, history, and OpenHands metrics;
• metrics — workspace, selected preset and model protocol, requested/resolved output paths, runtime status, and timeout diagnostics.

A non-zero remote exit, wall-clock timeout, OpenHands error, or missing requested output file produces RUN_ERROR. The benchmark then stores the harness result with its evaluation data under the run directory’s details/ subdirectory; see Results.

Set the task execution deadline through --execution-params with run_timeout_seconds and run_timeout_multiplier. See phase timeouts.