> ## Documentation Index
> Fetch the complete documentation index at: https://agent-compass.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Validation and Alignment

Validate a Benchmark from focused checks through full official-result alignment.

Use the shared [testing and validation guide](https://agent-compass.mintlify.site/en/developer_guide/contributing/testing.md) to distinguish repository checks, component discovery, manual smoke runs, and evidence that requires external infrastructure.

## Validation Ladder

Expand validation only after the previous level passes:

1. Import, registry, config parsing, version validation, and task selection.
2. Focused checks for dataset conversion, evaluator shaping, recipes, and secret redaction.
3. One real end-to-end smoke task with concurrency `1` and retries disabled.
4. One representative task for every claimed Environment and recommended Harness or Benchmark-owned runner path.
5. Recipe automatic and explicit-override paths across every claimed provider.
6. Enforcement tests for restricted network behavior.
7. The complete official split with an official or recommended configuration.
8. Score and failure alignment against a public official result.
9. Repository lint and documentation validation.

The smoke task must exercise normal assets and the official evaluator:

```text theme={"system"}
task loading
  -> Environment creation
  -> Benchmark preparation
  -> Harness or Benchmark-owned runner execution
  -> artifact collection
  -> evaluation
  -> result persistence
```

Do not use a synthetic empty task and do not substitute mocked Recipe checks for a real sandbox run.

## Provider and Precedence Matrix

Verify each Recipe policy across its supported providers and Benchmark versions:

| Inputs | Expected resolution |
| - | - |
| Request `setup.image` plus task image | Explicit request image |
| Task setup image only | Task image |
| No image source | Documented fallback or early actionable error |
| Explicit and task resources | Explicit values win per field; missing values inherit task defaults |
| Explicit workspace plus Recipe default | Explicit workspace |
| Explicit `evaluation_setup` plus task verifier setup | Explicit evaluation fields override verifier defaults |
| Separate task verifier setup, including an empty baseline | Independent from run setup; only an omitted baseline inherits resolved run setup |

Use the shared setup fields described in [Shared Contracts](https://agent-compass.mintlify.site/en/developer_guide/extensions/benchmark/code_implementation/shared_contracts.md#harbor-tasks-and-execution-requirements).
Provider-native startup selectors and legacy `image`/`evaluation_image` parameters are rejected public inputs.

For image-dependent paths, follow [image preparation requirements](https://agent-compass.mintlify.site/en/developer_guide/extensions/recipe_integration.md#prepare-required-images). Check that the selected Environment can obtain the resolved execution and verifier images, and confirm actual Harness or Benchmark-runner startup and scoring. Plan inspection alone does not establish image readiness.

## Network and Security Validation

When execution is restricted, issue a real request from inside the sandbox. `no-network` must fail at the transport layer;
agent compliance is not proof. For `allowlist`, prove one allowed and one denied destination. When a known exploitation
case exists, include the destination that previously exposed answer-bearing material.

Confirm that secrets, proxy URLs, tokens, and internal endpoints are redacted and that failure cleanup removes temporary
network, proxy, and sandbox resources.

## Full Evaluation

Run the complete official split with at least one Model and Harness or Benchmark-runner setting used by the official repository, paper,
technical report, blog, or leaderboard. Match all material dimensions:

* Dataset version, revision, split, category, language and task count.
* Model checkpoint, endpoint protocol, temperature, reasoning mode and request body.
* Harness or Benchmark-runner version, prompt, installation mode, step and cost behavior; simulator/judge settings where used.
* Environment image, workspace, resources, provider and network policy.
* Task, command and verifier timeouts; retries; attempts per task; aggregation.

Use `k=1` when the intent is one attempt per task. Keep framework retry behavior separate from Benchmark sampling. Do not
call a run aligned when task coverage, failure denominator, Model settings, or evaluator version differ without an
explicit explanation.

## Alignment Report

Add a concise **Benchmark alignment** section directly to the PR description. A separate committed report is not
required. Use the final authoritative result set; mention reruns only when they materially affect comparability.

* **Sources and reproduction:** pinned dataset/evaluator revisions, official result link, and an exact sanitized command
  or launch YAML. Include Model/protocol, Harness or Benchmark-runner/version, judge settings where used, split, attempts/retries,
  Environment/image/resources/network, and time limits. Reference credentials through environment variables.
* **Results:** a compact table with Model/configuration, total tasks, evaluated, invalidated, errors, and primary metric.
  Use the metric report's coverage counts; error and evaluated counts can overlap. Include pass/fail counts for binary
  metrics and category/split rows when useful.
* **Official comparison:** official score and uncertainty/run count when available, observed score and absolute delta,
  material configuration differences, and a verdict of aligned, differs, or pending. When no comparable official result
  exists, state that limitation. A short list of differences is sufficient; a full configuration matrix is optional.
* **Validation and evidence:** smoke-tested combinations, important checks and network-enforcement observations,
  classified failures, checks not run and why, and accessible sanitized artifact links where needed.

When presenting an alignment comparison table, use `✅`, `⚠️`, and `❌` in its `Status` column and add this legend immediately below the table:

> ✅ Aligned · ⚠️ Explained difference expected not to materially affect the result · ❌ Not aligned

For unverified items, use `Pending` and state the missing evidence.

Classify non-successful tasks from their logs before interpreting the score. Do not hide failures inside an average or
compare a completed-only score against an official all-task denominator. Add efficiency or trajectory comparisons only
when they help explain a discrepancy and upstream evidence is available.

If a full run or required provider is unavailable, submit the feasible implementation and evidence with the affected
support/alignment explicitly marked pending and the command needed to complete it. Smoke success alone does not prove
performance alignment; maintainers still assess the missing evidence before merge.

## Repository Checks

```bash theme={"system"}
uvx pre-commit run --all-files --show-diff-on-failure
cd docs
mint broken-links
mint validate
```

Focused local tests are useful when developing a Benchmark adapter. Follow the repository's current test policy and
record reproducible commands and outcomes in the pull request.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.