Validation and Alignment

Validate a harness across protocols, environments, lifecycle failures, concurrency, and official settings.

Focused Validation

Verify import, registry discovery, config parsing, protocol validation, launch command generation, output parsing, redaction, and cleanup behavior before launching a full task.

Test controlled failures for:

• Missing executable or optional dependency.
• Unsupported model protocol or environment capability.
• Invalid credentials or model endpoint response.
• Installation failure.
• Command timeout, step limit, malformed output, and cancellation.
• Cleanup after partial session startup.

End-to-End Matrix

Start with one real benchmark task, task concurrency 1, and retries disabled. Then cover:

Dimension	Minimum evidence
Benchmark	One intended benchmark using a normal PreparedTask and evaluator
Protocol	Every newly claimed model API protocol
Environment	Every provider claimed in the harness documentation
Installation	Every claimed launch/install strategy
Outcome	Successful run plus controlled failure and timeout paths
Concurrency	Small concurrent batch after the single-task path succeeds

The small concurrent batch should expose shared config files, fixed process names, session-id collisions, global mutable state, log mixing, client reuse, and cleanup races.

When the harness supports setup/run network transitions, prove installation or startup under the setup policy and issue a real denied request during restricted execution. Verify that close_session() still runs when policy switching or agent execution fails.

Official Alignment

If the harness claims parity with an official agent or leaderboard configuration, run the complete official benchmark split using the pinned harness version and match its prompt, model protocol, inference settings, step/cost limits, timeouts, resources, network policy, retry policy, and attempts per task.

Report:

• Supported benchmark, protocol, environment, and installation matrix.
• Exact redacted smoke and full-evaluation commands.
• A representative normalized trajectory and termination record.
• Task coverage, classified failures, score, and comparison with the official setting.
• Material prompt, provider, model deployment, or resource differences.

Follow the report structure in Benchmark Validation and Alignment when publishing a score alignment claim.

Repository Checks

uvx pre-commit run --all-files --show-diff-on-failure
cd docs
mint broken-links
mint validate