> ## Documentation Index
> Fetch the complete documentation index at: https://agent-compass.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Validation and Alignment

> Validate a benchmark from focused checks through full official-result alignment.

## Validation Ladder

Expand validation only after the previous level passes:

1. Import, registry, config parsing, version validation, and task selection.
2. Focused checks for dataset conversion, evaluator shaping, recipes, and secret redaction.
3. One real end-to-end smoke task with concurrency `1` and retries disabled.
4. One representative task for every claimed environment and recommended harness path.
5. Recipe automatic and explicit-override paths across every claimed provider.
6. Enforcement tests for restricted network behavior.
7. The complete official split with an official or recommended configuration.
8. Score and failure alignment against a public official result.
9. Repository lint and documentation validation.

The smoke task must exercise normal assets and the official evaluator:

```text theme={"system"}
task loading
  -> environment creation
  -> benchmark preparation
  -> harness execution
  -> artifact collection
  -> evaluation
  -> result persistence
```

Do not use a synthetic empty task and do not substitute mocked recipe checks for a real sandbox run.

## Provider and Precedence Matrix

Verify every sibling recipe, provider, and supported benchmark version:

| Inputs                                            | Expected resolution                                                 |
| ------------------------------------------------- | ------------------------------------------------------------------- |
| Provider-native selector plus explicit/task image | Provider-native selector                                            |
| Explicit image plus task image                    | Explicit image                                                      |
| Task image only                                   | Task image                                                          |
| No image source                                   | Documented fallback or early actionable error                       |
| Explicit and task resources                       | Explicit values win per field; missing values inherit task defaults |
| Explicit workspace plus recipe default            | Explicit workspace                                                  |

Confirm both the resolved plan and a real harness startup when the custom image supplies harness prerequisites.

## Network and Security Validation

When execution is restricted, issue a real request from inside the sandbox. `no-network` must fail at the transport layer;
agent compliance is not proof. For `allowlist`, prove one allowed and one denied destination. When a known exploitation
case exists, include the destination that previously exposed answer-bearing material.

Confirm that secrets, proxy URLs, tokens, and internal endpoints are redacted and that failure cleanup removes temporary
network, proxy, and sandbox resources.

## Full Evaluation

Run the complete official split with at least one model and harness setting used by the official repository, paper,
technical report, blog, or leaderboard. Match all material dimensions:

* Dataset version, revision, split, category, language and task count.
* Model checkpoint, endpoint protocol, temperature, reasoning mode and request body.
* Harness version, prompt, installation mode, step and cost behavior.
* Environment image, workspace, resources, provider and network policy.
* Task, command and verifier timeouts; retries; attempts per task; aggregation.

Use `k=1` when the intent is one attempt per task. Keep framework retry behavior separate from benchmark sampling. Do not
call a run aligned when task coverage, failure denominator, model settings, or evaluator version differ without an
explicit explanation.

## Alignment Report

Present the final authoritative result set as one evaluation unless rerun history materially affects comparability. The
report must include:

1. **Results summary:** complete denominator, normally completed tasks, evaluator passes and failures, infrastructure
   errors, primary score, and meaningful category or split results.
2. **Configuration alignment matrix:** official versus AgentCompass benchmark, dataset, harness, model, prompt,
   inference, timeout, resource, provider, network, `k`, retry, and aggregation settings.
3. **Official result comparison:** cited source, official score and interval, run count, valid attempts, observed score,
   absolute delta, and whether the comparison is point-to-point or single-run-to-distribution.
4. **Efficiency and trajectory comparison:** duration, steps, tokens, and material differences when official artifacts
   expose them.
5. **Policy enforcement and reliability:** resolved network policies, observed blocked access, and classified runtime
   failures.
6. **Conclusion:** passed/total, percentage, infrastructure error count, alignment verdict, and remaining differences.
7. **Appendix A:** an accessible download link to redacted evaluation artifacts.
8. **Appendix B:** the exact reproduction command, with secrets replaced only by environment-variable references.

Use `✅`, `⚠️`, and `❌` in the alignment Status column and add this legend immediately below the table:

> `✅ Aligned` · `⚠️ Explained difference expected not to materially affect the result` · `❌ Not aligned`

Classify every non-successful task from its logs before interpreting the score. Do not hide failed tasks inside an
average or compare only normally completed tasks against an official all-task denominator.

## Repository Checks

```bash theme={"system"}
uvx pre-commit run --all-files --show-diff-on-failure
cd docs
mint broken-links
mint validate
```

Focused local tests are useful when developing a benchmark adapter. Follow the repository's current test policy and
record reproducible commands and outcomes in the pull request.