> ## Documentation Index
> Fetch the complete documentation index at: https://agent-compass.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting Runs

> Reduce a failed evaluation, locate the failing phase, inspect evidence, and apply the correct fix.

Diagnose an AgentCompass failure by reducing it to one task and identifying the first lifecycle phase that failed.
Avoid changing several limits or components at once: that can hide the original cause and make the recovered result
incomparable with the intended evaluation.

## Start With a Minimal Reproduction

Use the same model, benchmark, harness, environment, and component parameters as the failed run, but select one failed
task, disable retries, and retain detailed logs:

```bash theme={"system"}
export MODEL_NAME=""

agentcompass run <benchmark> <harness> "$MODEL_NAME" \
  --env <environment> \
  --benchmark-params '{"sample_ids":["<failed-task-id>"]}' \
  --task-concurrency 1 \
  --max-retries 0 \
  --log-level INFO \
  --file-log-level DEBUG
```

Add `--keep-environment` only when the failure requires inspecting files or processes inside the sandbox. Do not enable
automatic dependency installation until you have confirmed that a missing optional extra is the actual problem.

## Locate the First Failing Phase

Search the persistent run log for the task id and the last started phase. The normal sequence is:

| Phase or log message      | What owns the failure                                                                                          |
| ------------------------- | -------------------------------------------------------------------------------------------------------------- |
| Task loading or selection | Benchmark parameters, dataset access, cache, or optional benchmark dependency                                  |
| `Execution plan building` | Benchmark/harness/environment compatibility or recipe inputs                                                   |
| Environment setup         | Provider credentials, image, snapshot, quota, resource request, network initialization, or startup timeout     |
| Material preparation      | Benchmark workspace layout, input files, or task metadata                                                      |
| Harness setup             | Harness dependency, executable installation, credentials, or setup network access                              |
| `run_harness` / inference | Model endpoint, agent loop, tool execution, run network policy, or harness timeout                             |
| Artifact collection       | Expected output path, file download, or workspace permission                                                   |
| Evaluation / verification | Tests, verifier dependency, verifier timeout, judge model, or verifier network policy                          |
| Analysis                  | Analyzer dependency, analyzer model, or optional post-processing; the measured task result may still be usable |

The first failure is usually more useful than later cleanup warnings. For example, an environment-close warning after a
model authentication error is not the root cause.

## Inspect the Run Evidence

The run directory contains different levels of evidence:

| Artifact                        | Inspect it for                                                                                               |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| `run_info.json`                 | Sanitized original request, reuse source, and recipe-resolved execution plans by task and attempt            |
| `params.json`                   | Sanitized persistence and effective parameter payload used for results and summaries                         |
| `logs/*.log`                    | Full phase messages, commands, provider responses, and tracebacks                                            |
| `progress.jsonl`                | Ordered task and phase events, including retry and reuse events                                              |
| `progress.json`                 | Current totals and the latest run state                                                                      |
| `details/<task-id>.json`        | Attempts, resolved execution plan, prediction, trajectory, metrics, verification result, and analyzer output |
| `details/_error_<task-id>.json` | A retryable or incomplete task that can be scheduled again by reuse                                          |
| `retry_details/*.json`          | Why a retry was consumed and which result was discarded                                                      |
| `summary.md`                    | Aggregate completion, correctness, and benchmark metrics                                                     |

Use the [local result browser](/en/user_guide/results#local-result-browser) for an interactive view, or inspect the
files directly. Preserve `run_info.json`, `params.json`, the relevant detail file, and the log when asking another
person to reproduce the issue.

## Common Failures

| Symptom                                                                | Likely cause                                                                                              | Correct next step                                                                                                               |
| ---------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `Unused Tokens` followed by `command not found`                        | A shell line continuation contains spaces after `\`.                                                      | Remove all characters after the backslash and rerun the command.                                                                |
| JSON parsing or unknown option error                                   | A JSON flag is malformed or a field was passed as a top-level CLI option.                                 | Validate quoting and inspect `agentcompass run --help` plus the component schema.                                               |
| `OptionalDependencyError`                                              | A declared extra is missing from the AgentCompass host Python environment.                                | Run the printed `uv` or `pip` command; opt into auto-install only for trusted components.                                       |
| Harness does not support the environment or protocol                   | The selected combination is incompatible.                                                                 | Use the benchmark's recommended harness and a protocol listed on the harness page.                                              |
| Generic Daytona or Modal sandbox cannot run the benchmark              | The benchmark requires a prebaked task image, snapshot, or workspace layout.                              | Let a compatible recipe infer it, or pass a deliberate image/snapshot override for a custom run.                                |
| Docker daemon permission denied                                        | The current user cannot access `/var/run/docker.sock`.                                                    | Start Docker and configure daemon access or use the provider's `use_sudo_docker` option where appropriate.                      |
| Image pull or manifest error                                           | Registry authentication, architecture, or image name is wrong.                                            | Authenticate, verify the image, and set a compatible platform if required.                                                      |
| `401` / `403` from a model endpoint                                    | Credential or endpoint authorization is wrong.                                                            | Check `MODEL_API_KEY`, base URL, model access, and any provider-specific headers.                                               |
| `404` or model not found                                               | The model id or endpoint path is wrong.                                                                   | Query the endpoint's model list and verify whether the base URL should include `/v1`.                                           |
| `429`, throttling, or rising latency                                   | Model concurrency or request/token rate exceeds capacity.                                                 | Lower task concurrency and environment open QPS; retry only the transient response.                                             |
| Jina Reader reports `401`, `402`, `403`, or `429`                      | The Jina credential is invalid, unauthorized, or out of quota.                                            | Check `JINA_API_KEY` and quota. The search harness logs the first fatal status at `ERROR` and suppresses duplicate occurrences. |
| A search `visit` attempt is absent from normal console logs            | Per-attempt Jina and summarizer retries are intentionally logged at `DEBUG`; only exhausted retries warn. | Inspect the persistent DEBUG run log before concluding that the tool did not retry.                                             |
| Model requests fail only inside the sandbox                            | The run network allowlist omits the model endpoint.                                                       | Add the exact endpoint hostname or use a local harness execution mode.                                                          |
| Harness install fails under isolation                                  | Setup network is restricted or the image lacks the harness.                                               | Keep setup public, allowlist package hosts, or use a prebuilt image.                                                            |
| Agent command timeout                                                  | A harness command or rollout limit expired.                                                               | Change the corresponding harness field after confirming the command is legitimately still progressing.                          |
| Verifier timeout after a completed rollout                             | A benchmark verifier limit expired.                                                                       | Change the benchmark verifier setting, not the harness command timeout.                                                         |
| Whole run ends while tasks remain                                      | `--timeout-seconds` is below the complete run duration.                                                   | Increase the whole-run budget or reduce the selected task set.                                                                  |
| Cost tracking rejects an unknown model id                              | The harness cost database does not map the custom model name.                                             | Use that harness's documented ignore-errors cost mode only when cost accounting is not required.                                |
| A completed task is unexpectedly rerun                                 | Reuse was disabled, the source file is missing/error-prefixed, or its task id/filename does not match.    | Check the reuse source, task id, category, and detail filename.                                                                 |
| `launch` rejects implicit reuse for duplicate benchmark/model requests | More than one request shares a result hierarchy, so “latest matching run” is ambiguous.                   | Set `runtime.reuse_run_id` explicitly for each affected request or disable reuse there.                                         |
| Console is noisy despite a high log level                              | A dependency configures its own logger before or outside AgentCompass logging.                            | Keep the file log, identify the logger name, and use the integration's documented verbosity control.                            |

## Inspect Effective Configuration

When a value appears to be ignored, compare the merged configuration with the component schema:

```bash theme={"system"}
agentcompass config show \
  --benchmark <benchmark> \
  --harness <harness> \
  --env <environment>

agentcompass config docs benchmark <benchmark>
agentcompass config docs harness <harness>
agentcompass config docs env <environment>
```

Then inspect the per-task resolved execution plan. Recipes run after normal configuration layers and may adapt images,
workspaces, resources, and network policies while preserving explicit user overrides. The resolved plan is the source
of truth for what that task actually used.

## Provider Checks

Run the provider's smallest independent check before debugging AgentCompass internals:

| Provider     | Useful check                                                                                            |
| ------------ | ------------------------------------------------------------------------------------------------------- |
| Docker       | `docker version`, `docker info`, and `docker run --rm hello-world`                                      |
| Daytona      | Confirm the key, target, quota, and failed sandbox in the Daytona dashboard.                            |
| Modal        | Run `modal token info` and inspect the AgentCompass app and sandbox in the Modal dashboard.             |
| Host process | Confirm the current working directory, executable paths, file permissions, and required local services. |

Provider credentials can succeed while a particular image or resource request fails. Keep the one-task AgentCompass
smoke test after the independent provider check because it also validates recipes, workspaces, harness setup, and
verification.

## Decide Whether to Retry, Reuse, or Restart

| Situation                                                           | Action                                                                   |
| ------------------------------------------------------------------- | ------------------------------------------------------------------------ |
| Temporary API or sandbox failure with unchanged settings            | Retry the task with a narrow retry pattern.                              |
| Interrupted large run with valid completed details                  | Use `--reuse` only after manually confirming the same measured settings. |
| Wrong model, benchmark, harness, network policy, or scoring setting | Start a new run; do not merge it into the old result.                    |
| Deterministic task failure                                          | Keep it as a measured failure unless the integration itself is broken.   |
| Integration bug fixed after the run                                 | Rerun affected tasks and document the code revision and reuse procedure. |

See [agentcompass run](/en/user_guide/cli/run#control-task-execution) for retry and reuse semantics and
[Network Policy](/en/user_guide/modules/environments/network) for phase-specific network diagnosis.

## Report a Reproducible Issue

Include the following when opening an issue:

* AgentCompass revision and Python version;
* operating system and environment provider;
* benchmark, harness, model protocol, and task id;
* exact command with secrets removed;
* relevant component parameters and whether a recipe was applied;
* first failing phase and complete traceback from the file log;
* sanitized `run_info.json`, `params.json`, and task detail; and
* whether the failure reproduces with concurrency `1` and retries disabled.

Do not upload API keys, provider tokens, private base URLs, proxy credentials, or proprietary task data.