> ## Documentation Index > Fetch the complete documentation index at: https://agent-compass.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Testing and Validation Validate the contract you changed with the smallest reproducible check, then exercise one real task and record evidence a reviewer can inspect. ## Run the Basic Checks The public repository currently has no unified project test suite, no declared `pytest` test workflow, and no CI test job. The checked-in pull-request workflow runs `pre-commit` only. Do not cite a generic `pytest` command as repository validation unless your change adds an actual test target and you run that exact target. This baseline does not lower the correctness requirement: combine the existing static checks with component discovery, configuration inspection, focused deterministic checks where available, and representative single-task execution. During development, run `pre-commit` on the files you changed: ```bash theme={"system"} uvx pre-commit run --files \ src/agentcompass/.py \ docs/en/.mdx \ docs/zh/.mdx ``` Before requesting review, run the same full command used by the current CI workflow: ```bash theme={"system"} uvx pre-commit run --all-files --show-diff-on-failure ``` The configured hooks run Flake8, isort, YAPF, whitespace and line-ending checks, YAML validation, requirement sorting, and merge-conflict detection. Some hooks can modify files; inspect the diff and rerun until the command succeeds without unexpected changes. Documentation changes also require the checks in [Documentation Contributions](/en/developer_guide/contributing/documentation). ## Validate the Changed Surface First inspect components and configuration from the installed revision instead of relying on a remembered component list: ```bash theme={"system"} uv run agentcompass list benchmark uv run agentcompass list harness uv run agentcompass list env uv run agentcompass list analyzer ``` These commands confirm that package imports reach the registration decorator and that the expected ID is unique. They do not verify credentials, dependencies, provider availability, compatibility, or task execution. For a Benchmark, Harness, or Environment config dataclass, inspect the generated public schema: ```bash theme={"system"} uv run agentcompass config docs benchmark uv run agentcompass config docs harness uv run agentcompass config docs env ``` Confirm that field names, types, defaults, and descriptions match the implementation. Analyzer `conf` dictionaries and Recipe classes are not supported by `config docs`; validate their documented keys against source and runtime output instead. To inspect merged configuration without starting an evaluation, use: ```bash theme={"system"} uv run agentcompass config show \ --config \ --benchmark \ --harness \ --env ``` For a multi-request orchestration file, the implemented dry-run path is: ```bash theme={"system"} uv run agentcompass launch --dry-run ``` `agentcompass run` has no `--dry-run` option. An orchestration dry run validates and prints resolved requests but does not load Benchmark tasks, open Environments, execute Harnesses, evaluate results, or prove cleanup. It cannot replace a single-task smoke run. Next run one known task. Use a stable task ID from a locally available public Benchmark and minimize unrelated variability: ```bash theme={"system"} uv run agentcompass run \ \ \ "$MODEL_NAME" \ --env \ --benchmark-params '{"sample_ids":[""]}' \ --model-base-url "$MODEL_BASE_URL" \ --model-api-key "$MODEL_API_KEY" \ --task-concurrency 1 \ --max-retries 0 \ --progress plain \ --log-level DEBUG ``` Use the exact supported combination and component-specific parameters documented for that Benchmark. Keep `--max-retries 0` for initial diagnosis so a retry does not hide the first failure. Redact credentials and private endpoint details before sharing the command. Inspect the resulting `run_info.json`, `params.json`, `details/`, `summary.md`, and run log as applicable. A process exit alone is insufficient evidence: verify the task ID, resolved execution plan, status, error category, final answer or artifacts, Benchmark score, cleanup events, and denominator represented by the summary. Finally, add checks for behavior unique to the changed surface: | Changed surface | Additional focus | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Benchmark | Prepared input and ground truth; exact task selection; expected status, score shape, and artifacts for success and a representative evaluator failure | | Harness | Supported Model protocol and dependency path; normalized final answer, trajectory, tool calls, and metrics; timeout and error mapping; session cleanup | | Environment | Open, execute, transfer, and network switching where supported; timeout and close behavior; no leaked resource after an injected failure | | Recipe | Matching and non-matching inputs; expected `applied_recipes` ID; resolved fallbacks; preservation of compatible explicit user values | | Analyzer and results | Dataset and data-requirement eligibility; expected `analysis_result` payload; skip and error paths; old and new persisted results; declared summary fields | | runtime and configuration | Consistent request resolution from CLI, SDK, and configuration inputs; override precedence; retries, cancellation, persistence, cleanup, and affected public Environment combinations | For lifecycle or compatibility changes, also test the relevant boundary: optional versus required input, a valid zero score versus evaluator failure, or timeout and cancellation followed by cleanup. Keep Harness, Environment, and Model failures distinguishable in persisted status and errors instead of normalizing them into one generic exception. Expand to other public Environment combinations only when a shared runtime change affects them. Before intentionally rewriting a summary, first run: ```bash theme={"system"} uv run agentcompass summary --dry-run ``` Add narrower deterministic checks in the pull request when the change contains pure parsing, merging, matching, plan transformation, aggregation, or serialization logic. If no reusable test harness exists, include the exact invocation or small fixture used instead of describing it as an automated suite. ## Record Validation Evidence For every reported check, include: * the exact redacted command, tested revision, relevant dependency or provider versions, and selected public Benchmark, task ID, Harness, Environment, and Model protocol; * the pass/fail outcome and important output fields or artifact paths; * any checks not run, with the concrete reason and effect on risk assessment; * any score comparison, with matching task coverage, settings, and denominator. Do not commit credentials, private endpoints, downloaded datasets, complete result directories, or large trajectories. Use small sanitized excerpts or stable external artifacts when a reviewer needs more than the pull request description can hold.