Testing and Validation

Validate the contract you changed with the smallest reproducible check, then exercise one real task and record evidence a reviewer can inspect.

Run the Basic Checks

The public repository currently has no unified project test suite, no declared pytest test workflow, and no CI test job. The checked-in pull-request workflow runs pre-commit only. Do not cite a generic pytest command as repository validation unless your change adds an actual test target and you run that exact target. This baseline does not lower the correctness requirement: combine the existing static checks with component discovery, configuration inspection, focused deterministic checks where available, and representative single-task execution.

During development, run pre-commit on the files you changed:

uvx pre-commit run --files \
  src/agentcompass/<changed-module>.py \
  docs/en/<changed-page>.mdx \
  docs/zh/<changed-page>.mdx

Before requesting review, run the same full command used by the current CI workflow:

uvx pre-commit run --all-files --show-diff-on-failure

The configured hooks run Flake8, isort, YAPF, whitespace and line-ending checks, YAML validation, requirement sorting, and merge-conflict detection. Some hooks can modify files; inspect the diff and rerun until the command succeeds without unexpected changes.

Documentation changes also require the checks in Documentation Contributions.

Validate the Changed Surface

First inspect components and configuration from the installed revision instead of relying on a remembered component list:

uv run agentcompass list benchmark
uv run agentcompass list harness
uv run agentcompass list env
uv run agentcompass list analyzer

These commands confirm that package imports reach the registration decorator and that the expected ID is unique. They do not verify credentials, dependencies, provider availability, compatibility, or task execution.

For a Benchmark, Harness, or Environment config dataclass, inspect the generated public schema:

uv run agentcompass config docs benchmark <benchmark-id>
uv run agentcompass config docs harness <harness-id>
uv run agentcompass config docs env <environment-id>

Confirm that field names, types, defaults, and descriptions match the implementation. Analyzer conf dictionaries and Recipe classes are not supported by config docs; validate their documented keys against source and runtime output instead.

To inspect merged configuration without starting an evaluation, use:

uv run agentcompass config show \
  --config <config.yaml> \
  --benchmark <benchmark-id> \
  --harness <harness-id> \
  --env <environment-id>

For a multi-request orchestration file, the implemented dry-run path is:

uv run agentcompass launch <orchestration.yaml> --dry-run

agentcompass run has no --dry-run option. An orchestration dry run validates and prints resolved requests but does not load Benchmark tasks, open Environments, execute Harnesses, evaluate results, or prove cleanup. It cannot replace a single-task smoke run.

Next run one known task. Use a stable task ID from a locally available public Benchmark and minimize unrelated variability:

uv run agentcompass run \
  <benchmark-id> \
  <harness-id> \
  "$MODEL_NAME" \
  --env <environment-id> \
  --benchmark-params '{"sample_ids":["<task-id>"]}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --task-concurrency 1 \
  --max-retries 0 \
  --progress plain \
  --log-level DEBUG

Use the exact supported combination and component-specific parameters documented for that Benchmark. Keep --max-retries 0 for initial diagnosis so a retry does not hide the first failure. Redact credentials and private endpoint details before sharing the command.

Inspect the resulting run_info.json, params.json, details/, summary.md, and run log as applicable. A process exit alone is insufficient evidence: verify the task ID, resolved execution plan, status, error category, final answer or artifacts, Benchmark score, cleanup events, and denominator represented by the summary.

Finally, add checks for behavior unique to the changed surface:

Changed surface	Additional focus
Benchmark	Prepared input and ground truth; exact task selection; expected status, score shape, and artifacts for success and a representative evaluator failure
Harness	Supported Model protocol and dependency path; normalized final answer, trajectory, tool calls, and metrics; timeout and error mapping; session cleanup
Environment	Open, execute, transfer, and network switching where supported; timeout and close behavior; no leaked resource after an injected failure
Recipe	Matching and non-matching inputs; expected applied_recipes ID; resolved fallbacks; preservation of compatible explicit user values
Analyzer and results	Dataset and data-requirement eligibility; expected analysis_result payload; skip and error paths; old and new persisted results; declared summary fields
runtime and configuration	Consistent request resolution from CLI, SDK, and configuration inputs; override precedence; retries, cancellation, persistence, cleanup, and affected public Environment combinations

For lifecycle or compatibility changes, also test the relevant boundary: optional versus required input, a valid zero score versus evaluator failure, or timeout and cancellation followed by cleanup. Keep Harness, Environment, and Model failures distinguishable in persisted status and errors instead of normalizing them into one generic exception. Expand to other public Environment combinations only when a shared runtime change affects them.

Before intentionally rewriting a summary, first run:

uv run agentcompass summary <run-dir> --dry-run

Add narrower deterministic checks in the pull request when the change contains pure parsing, merging, matching, plan transformation, aggregation, or serialization logic. If no reusable test harness exists, include the exact invocation or small fixture used instead of describing it as an automated suite.

Record Validation Evidence

For every reported check, include:

• the exact redacted command, tested revision, relevant dependency or provider versions, and selected public Benchmark, task ID, Harness, Environment, and Model protocol;
• the pass/fail outcome and important output fields or artifact paths;
• any checks not run, with the concrete reason and effect on risk assessment;
• any score comparison, with matching task coverage, settings, and denominator.

Do not commit credentials, private endpoints, downloaded datasets, complete result directories, or large trajectories. Use small sanitized excerpts or stable external artifacts when a reviewer needs more than the pull request description can hold.