Validation and Alignment

Validate a benchmark from focused checks through full official-result alignment.

Validation Ladder

Expand validation only after the previous level passes:

1. Import, registry, config parsing, version validation, and task selection.
2. Focused checks for dataset conversion, evaluator shaping, recipes, and secret redaction.
3. One real end-to-end smoke task with concurrency 1 and retries disabled.
4. One representative task for every claimed environment and recommended harness path.
5. Recipe automatic and explicit-override paths across every claimed provider.
6. Enforcement tests for restricted network behavior.
7. The complete official split with an official or recommended configuration.
8. Score and failure alignment against a public official result.
9. Repository lint and documentation validation.

The smoke task must exercise normal assets and the official evaluator:

task loading
  -> environment creation
  -> benchmark preparation
  -> harness execution
  -> artifact collection
  -> evaluation
  -> result persistence

Do not use a synthetic empty task and do not substitute mocked recipe checks for a real sandbox run.

Provider and Precedence Matrix

Verify every sibling recipe, provider, and supported benchmark version:

Inputs	Expected resolution
Provider-native selector plus explicit/task image	Provider-native selector
Explicit image plus task image	Explicit image
Task image only	Task image
No image source	Documented fallback or early actionable error
Explicit and task resources	Explicit values win per field; missing values inherit task defaults
Explicit workspace plus recipe default	Explicit workspace

Confirm both the resolved plan and a real harness startup when the custom image supplies harness prerequisites.

Network and Security Validation

When execution is restricted, issue a real request from inside the sandbox. no-network must fail at the transport layer; agent compliance is not proof. For allowlist, prove one allowed and one denied destination. When a known exploitation case exists, include the destination that previously exposed answer-bearing material.

Confirm that secrets, proxy URLs, tokens, and internal endpoints are redacted and that failure cleanup removes temporary network, proxy, and sandbox resources.

Full Evaluation

Run the complete official split with at least one model and harness setting used by the official repository, paper, technical report, blog, or leaderboard. Match all material dimensions:

• Dataset version, revision, split, category, language and task count.
• Model checkpoint, endpoint protocol, temperature, reasoning mode and request body.
• Harness version, prompt, installation mode, step and cost behavior.
• Environment image, workspace, resources, provider and network policy.
• Task, command and verifier timeouts; retries; attempts per task; aggregation.

Use k=1 when the intent is one attempt per task. Keep framework retry behavior separate from benchmark sampling. Do not call a run aligned when task coverage, failure denominator, model settings, or evaluator version differ without an explicit explanation.

Alignment Report

Present the final authoritative result set as one evaluation unless rerun history materially affects comparability. The report must include:

1. Results summary: complete denominator, normally completed tasks, evaluator passes and failures, infrastructure errors, primary score, and meaningful category or split results.
2. Configuration alignment matrix: official versus AgentCompass benchmark, dataset, harness, model, prompt, inference, timeout, resource, provider, network, k, retry, and aggregation settings.
3. Official result comparison: cited source, official score and interval, run count, valid attempts, observed score, absolute delta, and whether the comparison is point-to-point or single-run-to-distribution.
4. Efficiency and trajectory comparison: duration, steps, tokens, and material differences when official artifacts expose them.
5. Policy enforcement and reliability: resolved network policies, observed blocked access, and classified runtime failures.
6. Conclusion: passed/total, percentage, infrastructure error count, alignment verdict, and remaining differences.
7. Appendix A: an accessible download link to redacted evaluation artifacts.
8. Appendix B: the exact reproduction command, with secrets replaced only by environment-variable references.

Use ✅, ⚠️, and ❌ in the alignment Status column and add this legend immediately below the table:

✅ Aligned · ⚠️ Explained difference expected not to materially affect the result · ❌ Not aligned

Classify every non-successful task from its logs before interpreting the score. Do not hide failed tasks inside an average or compare only normally completed tasks against an official all-task denominator.

Repository Checks

uvx pre-commit run --all-files --show-diff-on-failure
cd docs
mint broken-links
mint validate

Focused local tests are useful when developing a benchmark adapter. Follow the repository’s current test policy and record reproducible commands and outcomes in the pull request.