Code Implementation

Implement harness compatibility, lifecycle, installation, limits, secrets, and normalized results.

Implement the smallest adapter that owns the agent loop and communicates only through public runtime contracts.

1. Establish the Upstream Execution Contract

Record the official framework or CLI version, supported model protocols, configuration format, prompt flow, tool and workspace behavior, installation method, timeouts, termination rules, trajectory format, and credential handling.

Pin the upstream version when it affects commands, prompts, output parsing, or score reproducibility. Prefer the public SDK or CLI contract over private internal functions.

2. Define Config and Plan Types

Create the harness under src/agentcompass/harnesses/. A typical integration contains:

• A RuntimeHarnessConfig subclass for user-facing parameters.
• A typed HarnessPlan for resolved runtime state.
• A BaseHarness subclass registered with HARNESSES.
• Focused adapters for config generation, launch, and trajectory parsing when needed.

Public knobs belong in the config class so CLI, Python SDK, config files, and generated component documentation resolve the same fields. Keep secrets out of dataclass representations and persisted plan metadata.

Use the plan for normalized runtime choices such as version, launch mode, installation strategy, step limit, command timeout, and cost behavior. Do not mutate RunRequest or inspect private benchmark fields.

3. Validate Compatibility Early

Implement supports(environment, model) around capabilities rather than benchmark ids. Validate:

Dimension	Questions to answer
Model protocol	Which of openai-chat, openai-responses, and anthropic-messages are supported?
Environment	Does the agent require a shell, files, endpoint forwarding, browser, GUI, or privileged operation?
Workspace	Can it operate in the prepared workspace without assuming a benchmark-specific path?
Credentials	Are model or external-service credentials available at the correct execution location?
Installation	Can the selected launch and install strategy work in the chosen environment?

Reject unsupported combinations before environment startup when possible. Do not silently switch protocols, providers, installation modes, or models.

4. Implement the Lifecycle

The runtime calls the harness in this order:

start_session under setup network policy
  -> run_task under run network policy
  -> close_session after setup policy is restored

start_session() may install a trusted harness, generate its config, upload launch files, start a background process, or construct a client. Keep this setup separate from the untrusted agent rollout so a benchmark can use public setup and a restricted run policy.

run_task() must execute exactly one PreparedTask. Consume prepared.input.prompt, messages, files, media, tools, and workspace; do not reach back into benchmark internals. Return the best available final answer, requested files, ordered trajectory, token usage, timing, artifacts, and accurate TaskStatus.

close_session() must release harness-owned clients, subprocesses, servers, temporary configuration, and background tasks on success, timeout, cancellation, and error. It does not close the environment; the runtime owns that lifecycle.

5. Normalize Results Without Changing Scores

A harness reports execution, not benchmark correctness. A process exit code of zero does not necessarily mean the task succeeded, and a non-empty answer does not mean it is correct.

Preserve:

• The final answer and output artifacts available to the evaluator.
• Accurate COMPLETED or RUN_ERROR status, with timeout, refusal, invalid-output, and termination details preserved.
• Ordered assistant messages, tool calls, command results, and environment observations.
• Model and harness usage, latency, step count, and termination reason.
• Installation, launch, parsing, model API, and runtime error context.

Never convert an evaluator failure into a harness failure or set benchmark correct and score inside the harness.

6. Handle Installation and Dependencies

Support only installation strategies that the implementation can make reproducible. Common choices are a pinned preinstalled task image, controlled installation before restricted execution, or an isolated driver-side package extra.

• Pin agent CLI or package versions when behavior affects results.
• Surface installation return code, stdout, and stderr without leaking credentials.
• Do not assume every task image contains a package manager or compiler.
• Do not broaden the agent run network policy to make setup convenient.
• Prefer compatible prebuilt images when setup is expensive or network-sensitive.
• Keep specialized packages out of the default AgentCompass installation.

Model, judge, search, and provider credentials must be passed through supported environment/config mechanisms and redacted recursively from commands, generated files, logs, trajectories, URLs, and exceptions.

7. Bound Commands, Steps, Cost, and Model Requests

Give each limit one clear owner. A harness command timeout bounds an agent command or process; a step limit bounds the agent loop; model client retries handle request transport; runtime retries repeat a failed task attempt. Do not reuse one parameter name for multiple layers.

Treat unknown model pricing according to the harness cost-tracking contract. Missing cost metadata must not abort a scored run when the user explicitly selects an ignore-errors or disabled cost mode.

8. Register and Inspect the Component

Export the harness from src/agentcompass/harnesses/__init__.py, then verify registry discovery and its config schema:

uv run agentcompass list harness
uv run agentcompass config docs harness <harness-id>

The harness must define a stable id and non-empty user-facing description.