Skip to main content
Harnesses define how a prepared task is attempted. They adapt a model, command-line agent, or external agent framework to AgentCompass’ PreparedTask -> RunResult contract. Harnesses should not own benchmark scoring. They run the agent, collect trajectory and output, and return normalized data for the benchmark and analyzers.

​
What Harnesses Own

Compatibility

Declare whether the harness supports the selected environment and model protocol.

Session setup

Start any local process, remote entrypoint, tmux session, or framework session needed for tasks.

Task execution

Consume PreparedTask and return RunResult with prediction, trajectory, errors, and metadata.

Cleanup

Close sessions without deleting benchmark-owned result artifacts.

​
Core Interface

MethodResponsibility
supports(environment, model)Validate environment and model compatibility before tasks run.
build_config(req)Build harness-specific config from merged params.
build_plan(req, environment)Add harness-side execution plan fields.
start_session(env, req, plan)Prepare reusable state for one environment session.
run_task(session, prepared, req, plan)Execute one prepared task and return RunResult.
close_session(session)Release harness-owned resources.

​
Harness Families

FamilyExamplesBest Use when
Direct model callsopenai_chat, qwen3vl_guiThe task is a prompt, messages, or GUI grounding request.
Coding agentsmini_swe_agent, openhands, claude_code, codexThe task provides a repository workspace and issue prompt.
Terminal agentsterminus2, terminus2_skillsThe task requires shell actions in an environment.
Research agentsresearchharness, naive_search_agentThe task requires browsing, search, or long-form research.
Tool-use benchmark harnessesscicode_tool_use, openclawThe benchmark has a specialized tool protocol or framework.

​
Usage: GUI Grounding Harness

agentcompass run \
  screenspot \
  qwen3vl_gui \
  qwen3-vl \
  --env <env-provider> \
  --benchmark-params '{"category":"desktop"}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat
Here screenspot prepares image and instruction data. qwen3vl_gui sends the prepared GUI grounding request to the model endpoint. ScreenSpot still owns scoring.

​
Usage: Coding Agent Harness

agentcompass run \
  swebench_verified \
  mini_swe_agent \
  your-model \
  --env <env-provider> \
  --benchmark-params '{"sample_ids":["astropy__astropy-12907"]}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY"
The benchmark prepares a repository task under /testbed; the Modal recipe selects the task image; mini_swe_agent runs the repair workflow against that workspace.

​
Usage: Terminal Agent Harness

agentcompass run \
  terminal_bench_2 \
  terminus2 \
  your-model \
  --env <env-provider> \
  --benchmark-params '{"sample_ids":["overfull-hbox"]}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY"
Terminal harnesses depend heavily on environment behavior. Use remote sandboxes when the task image, command execution, or workspace isolation should not consume local resources.

​
What Harnesses Return

Good harness output includes:
  • final prediction or generated files;
  • structured trajectory steps;
  • model usage, latency, and tool-call metadata when available;
  • error fields that can be parsed by analyzers;
  • enough workspace or command context for benchmark evaluation.

​
Developer Notes

Add a harness when an agent framework, CLI, model interaction protocol, or terminal control loop is new. If only the task image or workspace differs, add a recipe or environment config instead.