Troubleshooting Runs

Reduce a failed evaluation, locate the failing phase, inspect evidence, and apply the correct fix.

Diagnose an AgentCompass failure by reducing it to one task and identifying the first lifecycle phase that failed. Avoid changing several limits or components at once: that can hide the original cause and make the recovered result incomparable with the intended evaluation.

Start With a Minimal Reproduction

Use the same model, benchmark, harness, environment, and component parameters as the failed run, but select one failed task, disable retries, and retain detailed logs:

export MODEL_NAME=""

agentcompass run <benchmark> <harness> "$MODEL_NAME" \
  --env <environment> \
  --benchmark-params '{"sample_ids":["<failed-task-id>"]}' \
  --task-concurrency 1 \
  --max-retries 0 \
  --log-level INFO \
  --file-log-level DEBUG

Add --keep-environment only when the failure requires inspecting files or processes inside the sandbox. Do not enable automatic dependency installation until you have confirmed that a missing optional extra is the actual problem.

Locate the First Failing Phase

Search the persistent run log for the task id and the last started phase. The normal sequence is:

Phase or log message	What owns the failure
Task loading or selection	Benchmark parameters, dataset access, cache, or optional benchmark dependency
Execution plan building	Benchmark/harness/environment compatibility or recipe inputs
Environment setup	Provider credentials, image, snapshot, quota, resource request, network initialization, or startup timeout
Material preparation	Benchmark workspace layout, input files, or task metadata
Harness setup	Harness dependency, executable installation, credentials, or setup network access
run_harness / inference	Model endpoint, agent loop, tool execution, run network policy, or harness timeout
Artifact collection	Expected output path, file download, or workspace permission
Evaluation / verification	Tests, verifier dependency, verifier timeout, judge model, or verifier network policy
Analysis	Analyzer dependency, analyzer model, or optional post-processing; the measured task result may still be usable

The first failure is usually more useful than later cleanup warnings. For example, an environment-close warning after a model authentication error is not the root cause.

Inspect the Run Evidence

The run directory contains different levels of evidence:

Artifact	Inspect it for
run_info.json	Sanitized original request, reuse source, and recipe-resolved execution plans by task and attempt
params.json	Sanitized persistence and effective parameter payload used for results and summaries
logs/*.log	Full phase messages, commands, provider responses, and tracebacks
progress.jsonl	Ordered task and phase events, including retry and reuse events
progress.json	Current totals and the latest run state
details/<task-id>.json	Attempts, resolved execution plan, prediction, trajectory, metrics, verification result, and analyzer output
details/_error_<task-id>.json	A retryable or incomplete task that can be scheduled again by reuse
retry_details/*.json	Why a retry was consumed and which result was discarded
summary.md	Aggregate completion, correctness, and benchmark metrics

Use the local result browser for an interactive view, or inspect the files directly. Preserve run_info.json, params.json, the relevant detail file, and the log when asking another person to reproduce the issue.

Common Failures

Symptom	Likely cause	Correct next step
Unused Tokens followed by command not found	A shell line continuation contains spaces after \.	Remove all characters after the backslash and rerun the command.
JSON parsing or unknown option error	A JSON flag is malformed or a field was passed as a top-level CLI option.	Validate quoting and inspect agentcompass run --help plus the component schema.
OptionalDependencyError	A declared extra is missing from the AgentCompass host Python environment.	Run the printed uv or pip command; opt into auto-install only for trusted components.
Harness does not support the environment or protocol	The selected combination is incompatible.	Use the benchmark’s recommended harness and a protocol listed on the harness page.
Generic Daytona or Modal sandbox cannot run the benchmark	The benchmark requires a prebaked task image, snapshot, or workspace layout.	Let a compatible recipe infer it, or pass a deliberate image/snapshot override for a custom run.
Docker daemon permission denied	The current user cannot access /var/run/docker.sock.	Start Docker and configure daemon access or use the provider’s use_sudo_docker option where appropriate.
Image pull or manifest error	Registry authentication, architecture, or image name is wrong.	Authenticate, verify the image, and set a compatible platform if required.
401 / 403 from a model endpoint	Credential or endpoint authorization is wrong.	Check MODEL_API_KEY, base URL, model access, and any provider-specific headers.
404 or model not found	The model id or endpoint path is wrong.	Query the endpoint’s model list and verify whether the base URL should include /v1.
429, throttling, or rising latency	Model concurrency or request/token rate exceeds capacity.	Lower task concurrency and environment open QPS; retry only the transient response.
Jina Reader reports 401, 402, 403, or 429	The Jina credential is invalid, unauthorized, or out of quota.	Check JINA_API_KEY and quota. The search harness logs the first fatal status at ERROR and suppresses duplicate occurrences.
A search visit attempt is absent from normal console logs	Per-attempt Jina and summarizer retries are intentionally logged at DEBUG; only exhausted retries warn.	Inspect the persistent DEBUG run log before concluding that the tool did not retry.
Model requests fail only inside the sandbox	The run network allowlist omits the model endpoint.	Add the exact endpoint hostname or use a local harness execution mode.
Harness install fails under isolation	Setup network is restricted or the image lacks the harness.	Keep setup public, allowlist package hosts, or use a prebuilt image.
Agent command timeout	A harness command or rollout limit expired.	Change the corresponding harness field after confirming the command is legitimately still progressing.
Verifier timeout after a completed rollout	A benchmark verifier limit expired.	Change the benchmark verifier setting, not the harness command timeout.
Whole run ends while tasks remain	--timeout-seconds is below the complete run duration.	Increase the whole-run budget or reduce the selected task set.
Cost tracking rejects an unknown model id	The harness cost database does not map the custom model name.	Use that harness’s documented ignore-errors cost mode only when cost accounting is not required.
A completed task is unexpectedly rerun	Reuse was disabled, the source file is missing/error-prefixed, or its task id/filename does not match.	Check the reuse source, task id, category, and detail filename.
launch rejects implicit reuse for duplicate benchmark/model requests	More than one request shares a result hierarchy, so “latest matching run” is ambiguous.	Set runtime.reuse_run_id explicitly for each affected request or disable reuse there.
Console is noisy despite a high log level	A dependency configures its own logger before or outside AgentCompass logging.	Keep the file log, identify the logger name, and use the integration’s documented verbosity control.

Inspect Effective Configuration

When a value appears to be ignored, compare the merged configuration with the component schema:

agentcompass config show \
  --benchmark <benchmark> \
  --harness <harness> \
  --env <environment>

agentcompass config docs benchmark <benchmark>
agentcompass config docs harness <harness>
agentcompass config docs env <environment>

Then inspect the per-task resolved execution plan. Recipes run after normal configuration layers and may adapt images, workspaces, resources, and network policies while preserving explicit user overrides. The resolved plan is the source of truth for what that task actually used.

Provider Checks

Run the provider’s smallest independent check before debugging AgentCompass internals:

Provider	Useful check
Docker	docker version, docker info, and docker run --rm hello-world
Daytona	Confirm the key, target, quota, and failed sandbox in the Daytona dashboard.
Modal	Run modal token info and inspect the AgentCompass app and sandbox in the Modal dashboard.
Host process	Confirm the current working directory, executable paths, file permissions, and required local services.

Provider credentials can succeed while a particular image or resource request fails. Keep the one-task AgentCompass smoke test after the independent provider check because it also validates recipes, workspaces, harness setup, and verification.

Decide Whether to Retry, Reuse, or Restart

Situation	Action
Temporary API or sandbox failure with unchanged settings	Retry the task with a narrow retry pattern.
Interrupted large run with valid completed details	Use --reuse only after manually confirming the same measured settings.
Wrong model, benchmark, harness, network policy, or scoring setting	Start a new run; do not merge it into the old result.
Deterministic task failure	Keep it as a measured failure unless the integration itself is broken.
Integration bug fixed after the run	Rerun affected tasks and document the code revision and reuse procedure.

See agentcompass run for retry and reuse semantics and Network Policy for phase-specific network diagnosis.

Report a Reproducible Issue

Include the following when opening an issue:

• AgentCompass revision and Python version;
• operating system and environment provider;
• benchmark, harness, model protocol, and task id;
• exact command with secrets removed;
• relevant component parameters and whether a recipe was applied;
• first failing phase and complete traceback from the file log;
• sanitized run_info.json, params.json, and task detail; and
• whether the failure reproduces with concurrency 1 and retries disabled.

Do not upload API keys, provider tokens, private base URLs, proxy credentials, or proprietary task data.