PreparedTask and an EnvironmentSession, owns the execution loop, and returns a normalized RunResult for benchmark
evaluation.
Integration Boundary
The harness owns agent setup, prompt and workspace handoff, model interaction, tool execution, trajectory capture, usage, completion state, and harness cleanup. It must not load benchmark datasets, interpret benchmark correctness, select provider-specific task images, or create its own untracked sandbox lifecycle.Integration Workflow
- Implement the harness contract, including compatibility, session lifecycle, installation, limits, secrets, and result normalization.
- Update public documentation with supported protocols, environments, parameters, and complete commands.
- Validate the integration across real tasks, protocols, environments, failure paths, concurrency, and any official alignment claim.
Completion Gate
- Compatibility follows capabilities and fails before expensive work.
- Setup, run, and cleanup phases have explicit behavior.
- Every claimed protocol, environment, and installation strategy has real evidence.
- Results preserve answer, trajectory, usage, limits, termination, and error meaning.
- Secrets are redacted from every persisted and displayed surface.
- Public documentation matches the implemented schema and runnable combinations.