> ## Documentation Index > Fetch the complete documentation index at: https://agent-compass.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Run Records and Diagnostics When an evaluation request starts writing results, AgentCompass creates a dedicated run directory for it. In addition to task details and aggregate results, the directory contains these run records: ```text theme={"system"} / ├── run_info.json ├── params.json ├── progress.json ├── progress.jsonl └── logs/ └── YYYYMMDD_HHMMSS.log ``` `run_info.json` records the request configuration and final state. `params.json` keeps the compact parameter set needed to save and re-aggregate results. `progress.json` provides the latest snapshot, `progress.jsonl` preserves the full event sequence, and the log records readable execution messages and errors. ## When Files Are Created | File | When it is created and updated | | --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `logs/.log` | Created when the run directory is reserved and receives logs from that point onward. | | `run_info.json` | Created before task loading. It is updated whenever a task attempt resolves its execution plan and again when the request ends. | | `progress.json`, `progress.jsonl` | Created with the first progress event. Every later event updates the snapshot and is appended to the event stream. | | `params.json` | Created or rewritten when an evaluation saves task details. It is rewritten after that evaluation's final aggregation succeeds, even when no tasks were selected. | Not every invocation leaves these files behind. The CLI and SDK check the request before creating its run directory; a failure at that point creates no run directory. `agentcompass launch --dry-run` also creates no output. A preparation error after the directory exists will usually leave the log and `run_info.json`. If error handling completes normally, AgentCompass also writes the final state and a `run_finished` event. If the process is forcibly terminated, the final state, the last progress events, or `params.json` may not have been written. ## `run_info.json` `run_info.json` answers two questions: which request configuration this evaluation used, and how the request ended. It is created before tasks are loaded, updated throughout the run, and updated with the final state when the request ends. ### Top-Level Fields | Field | Meaning | | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | `schema_version` | Currently fixed at `agentcompass.run_info.v1`. | | `run_id` | The final run ID assigned to this request. | | `started_at` | Time when this record was created, in time-zone-aware ISO 8601 format. It is not the start time of the AgentCompass process or the entire orchestration. | | `request` | Request produced after CLI, configuration-file, or SDK values are merged according to configuration precedence. Per-task Recipes have not yet been applied. | | `reused_from` | Present when a reuse-source run is resolved. It records the source run's `run_id`, `path`, or both, and can appear even when no task is ultimately reused. | | `resolved_execution_plans` | Present after at least one task attempt resolves a plan. It records plan summaries by task ID and attempt number. | | `status` | Final request state: `completed`, `failed`, `cancelled`, or `timed_out`. It may be absent until the request finishes normally. | | `finished_at` | Time when the final state was written, in time-zone-aware ISO 8601 format. | | `error` | Error information when the request ends with an error; omitted after successful completion. | ### `request` Structure `request` is divided into model, Benchmark, Harness, Environment, execution, runtime, output, and metadata sections. Each component's `params` is an open object whose fields depend on the selected component. | Field path | Meaning | | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `model.id` | ID of the model being evaluated. | | `model.base_url` | Base address of the model API; it may be empty when not configured. | | `model.api_key` | Model API credential. It is redacted according to sensitive-key rules when written, so the original secret cannot be recovered from this value. | | `model.api_protocol` | Model API protocol name or ordered protocol list. `auto` and an unspecified value are normalized to an empty string while the request is built, so the literal `auto` is not persisted here. | | `model.params` | Request or generation parameters passed to the model client. | | `benchmark.id` | Component ID of the selected Benchmark. | | `benchmark.params` | Benchmark-specific parameters after configuration and request overrides are merged. | | `harness.id` | Component ID of the selected Harness. | | `harness.params` | Harness-specific parameters after configuration and request overrides are merged. | | `environment.id` | Component ID of the selected Environment. | | `environment.params` | Environment-specific parameters after configuration and request overrides are merged. Per-task Recipe changes are not included yet. | | `environment.network_policy` | Network policy used while preparing the Environment. | | `environment.run_network_policy` | Optional network policy used by the Harness or task execution. It may be omitted when not configured separately. | | `environment.verifier_network_policy` | Optional network policy used during Benchmark scoring. It may be omitted when not configured separately. | | `execution.task_concurrency` | Number of tasks that a direct evaluation request can run concurrently. In a multi-request orchestration, the orchestration-level `task_concurrency` sets the global limit. | | `execution.enabled_recipes` | Recipe IDs eligible for matching. An empty list leaves all candidate Recipes eligible. | | `execution.keep_environment` | Whether to preserve an Environment after the task for debugging. | | `execution.enable_analysis` | Whether to run analyzers during evaluation. | | `execution.analysis_params` | Analyzer selection, analysis model, and analyzer-specific settings. | | `execution.max_retries` | Maximum number of runtime retries within each evaluation attempt. | | `execution.retry_pattern_list` | Regular expressions used to decide whether an error triggers a retry. With `null`, any non-empty error can trigger a retry. | | `runtime.reuse` | Whether to reuse normal task details from an existing run: `details/*.json` files without the `_error_` prefix. | | `runtime.reuse_run_id` | Run ID to use as the reuse source. When empty, AgentCompass may find the latest compatible run. | | `output.run_name` | Optional namespace below the result root. | | `output.run_id` | Directory ID ultimately used for this run. | | `metadata.config_path` | Configuration files loaded to build the request. One file is stored as a path string; multiple files are stored as a JSON-array string containing every path. Omitted when no configuration file was loaded. | | `metadata.recipe_dirs` | External Recipe directories loaded to build the request; omitted when absent. | Each network-policy object contains `network_mode`, which selects the network-access mode, and `allowed_hosts`, which lists permitted hosts. When AgentCompass writes JSON, it removes `null` values and empty objects or lists. An empty `allowed_hosts` may therefore be absent from the file. `request` is not a copy of the original command line. It also excludes process-level settings such as `results_dir`, the whole-request timeout, log levels, and Environment provider concurrency limits. To verify these settings, consult the invocation, configuration, and log together. See the [Model](/en/user_guide/modules/models/overview), [Benchmark](/en/user_guide/modules/benchmarks/overview), [Harness](/en/user_guide/modules/harnesses/overview), and [Environment](/en/user_guide/modules/environments/overview) documentation for component-specific fields. When present, `reused_from` has these fields: | Field | Meaning | | -------- | --------------------------------- | | `run_id` | Run ID of the reuse source. | | `path` | Path to the source run directory. | ### `resolved_execution_plans` Structure `resolved_execution_plans` records the Environment, network policies, and Recipes resolved for each task attempt. Its structure is: ```json theme={"system"} { "resolved_execution_plans": { "": { "attempts": { "1": { "environment": { "id": "", "network_policy": { "network_mode": "public", "allowed_hosts": [] } }, "evaluation_environment": null, "run_network_policy": { "network_mode": "public", "allowed_hosts": [] }, "verifier_network_policy": { "network_mode": "public", "allowed_hosts": [] }, "applied_recipes": [] } } } } } ``` | Field or key | Meaning | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `` | Task ID provided by the Benchmark. | | `attempts` | Plan records for the task, keyed by attempt number. Attempt numbers start at `1`. | | `environment` | Environment planned for task execution. It records only the component ID and preparation network policy. | | `evaluation_environment` | Separate Environment planned for scoring. It records only the component ID and preparation network policy, and is `null` when no separate scoring environment is needed. | | `run_network_policy` | Network policy planned for Harness or task execution. | | `verifier_network_policy` | Network policy planned for Benchmark scoring. | | `applied_recipes` | Recipe IDs matched for this attempt. | The plan summary is written after resolution but before the Environment is opened. It tells you what the attempt planned to use; it does not prove that the Environment was created successfully. It also excludes the complete Recipe-resolved image, snapshot, working directory, resources, and Environment provider parameters. A task reused from an existing run and not executed again receives no new plan entry. Its original plan remains in the reused task detail. ## `params.json` `params.json` stores only the parameters needed to write task details and regenerate summaries. AgentCompass rewrites it when saving task details or generating the final summary. The file may not exist if the request fails before either operation. Running `agentcompass summary` separately updates only the summary files and leaves an existing `params.json` unchanged. | Field path | Meaning | | -------------------- | --------------------------------------------------------------------------------------------- | | `model.id` | Model ID used for result paths, display, and recovery. | | `model.params` | Persisted copy of model request parameters. | | `model.base_url` | Model API base address, saved when non-empty. | | `model.api_key` | Redacted credential placeholder, saved when non-empty; it cannot recover the original secret. | | `model.api_protocol` | Model API protocol name or list, saved when non-empty. | | `benchmark.id` | Benchmark ID used to select the aggregation behavior. | | `benchmark.params` | Effective Benchmark parameters needed to save task details and regenerate summaries. | | `output.run_name` | Result namespace, saved when non-empty. | | `output.run_id` | Directory ID ultimately used for this run. | Unset fields directly under `model`, `benchmark`, and `output` are omitted; values such as empty strings can still remain inside nested `params` objects. `params.json` does not contain the Harness, Environment, execution controls, reuse settings, metadata, or complete Recipe-resolved configuration. You therefore cannot use it to reconstruct the complete evaluation configuration. When regenerating a summary, AgentCompass reads `run_info.json.request` first and uses `params.json` to fill in missing values. The two files serve these purposes: | File | Scope | Primary purpose | | --------------- | ----------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | | `run_info.json` | Broader merged request, reuse source, limited execution-plan summaries, and final request state | Verify how a run was started and how it ended | | `params.json` | Compact model, Benchmark, and output subset | Support result writes and supply compatibility information during summary regeneration | ## `progress.json` `progress.json` stores the latest run state and task counts. AgentCompass replaces the snapshot with the latest state whenever a progress event occurs, so a status page or script can poll it. | Field | Meaning | | ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `run_id` | This request's run ID. | | `model`, `benchmark`, `harness`, `environment` | IDs of the selected components. | | `status` | Current run state. The file is created after the first event, so it normally starts at `running` and may later change to `summarizing` and a final request state. The internal initial value `created` is normally not written. | | `total_tasks` | Number of tasks selected by the benchmark. | | `reused_tasks` | Number of tasks reused from the source run. | | `pending_tasks` | Tasks not yet started during execution; the count decreases at each `task_started` event. When the request ends, it is recalculated as `total_tasks - finished_tasks`, so it then also includes tasks that started but did not finish. | | `running_tasks` | Tasks that have started but have not emitted `task_finished`. | | `finished_tasks` | Reused tasks plus tasks that emitted `task_finished`. | | `completed_tasks` | Tasks whose `task_finished` event records `completed`, plus reused tasks. | | `failed_tasks` | Tasks marked as failed in the progress record. Any of these conditions counts: a top-level or attempt `status` exactly equal to `error`, a non-empty `error`, or an attempt with `meta.status` equal to `error`. A bare status string such as `run_error` or `eval_error` without error text does not count by itself. | | `skipped_tasks` | Tasks that emitted `task_finished` with an explicit `skipped` status. Reused tasks are not rerun, but they count as completed rather than skipped. | | `attempts_started`, `attempts_finished` | Evaluation attempts started and finished. Runtime retries within an attempt do not increase these counters. | | `partials_saved` | Task-level partial results successfully persisted. | | `current_phase_counts` | Object counting active tasks by their current phase; cleared when the request ends. | | `active_tasks` | Object keyed by task ID that records the current state of each active task; cleared when the request ends. | | `elapsed_seconds` | Seconds from progress-tracker creation to the latest event, rounded to three decimal places. | | `updated_at` | Unix timestamp of the latest event, in seconds. | Each `active_tasks.` object contains `category`, `phase`, `attempt`, and `updated_at`. After a task starts but before it enters a specific phase, `phase` is `running`. A missing category or attempt number is stored as `null`. `completed_tasks` means that execution ended normally; it does not mean that the benchmark marked the answer correct. Use task details and `summary.md` for correctness, scores, and benchmark metrics. ## `progress.jsonl` `progress.jsonl` stores the complete progress event stream. Each line is one JSON object, appended in emission order. To reconstruct a task's phases, attempts, and retries, read this file instead of relying only on the latest snapshot. The CLI's `--progress auto|plain|none` and the SDK's `progress="auto"|"plain"|"none"` control only the live terminal display. They do not disable `progress.json` or `progress.jsonl`. When an SDK caller provides a custom progress reporter, file creation depends on that reporter's output configuration. In the fields below, an orchestration means one `launch` invocation that schedules multiple evaluation requests. For a standalone request, the orchestration-related fields are `null`. ### Fields on Every Event | Field | Meaning | | --------------------- | ------------------------------------------------------------------------------------------------------------------ | | `run_id` | Run ID. | | `event` | Event name. | | `timestamp` | Unix timestamp when the event was emitted, in seconds. | | `task_id`, `category` | Task and category associated with the event; `null` for run-level events. | | `attempt` | Evaluation attempt associated with the event, starting at `1`; `null` when the event is not tied to an attempt. | | `phase` | Current phase recorded by the event; `null` when not applicable. | | `status` | Status recorded by the event; `null` when not applicable. | | `payload` | Additional data specific to the event; an empty object when there is no additional data. | | `orchestration_id` | Parent orchestration ID; `null` without orchestration context. | | `request_key` | Unique scheduling key for the request within the orchestration; `null` without orchestration context. | | `request_name` | Request name declared in the orchestration configuration; `null` without orchestration context. | | `request_index` | Position of the request in the orchestration configuration, starting at `0`; `null` without orchestration context. | All fields above are always serialized. Missing values are written as `null`, and `payload` is always an object. ### Events and Event-Specific Fields | `event` | Fields and `payload` | Meaning | | ------------------------- | --------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `run_started` | `payload`: `model`, `benchmark`, `harness`, `environment` | The request begins loading tasks. | | `tasks_loaded` | `payload.total_tasks` | Task loading and selection completed. | | `reuse_loaded` | `payload.reused_tasks`, `payload.tasks_to_run` | Reusable results were loaded and the remaining task count was determined. | | `task_started` | `task_id`, `category`; `payload.index`, `payload.total` | The task entered scheduled execution. | | `phase_changed` | `task_id`, `category`, optional `attempt`; `phase` | A task entered a new phase. | | `attempt_started` | `task_id`, `category`, `attempt` | One evaluation attempt started. | | `execution_plan_resolved` | `task_id`, `category`, `attempt`, `phase: "plan"`; `payload` is the resolved-plan summary | Plan resolution completed for the attempt; the payload matches the summary written to `run_info.json`. | | `attempt_retry` | `task_id`, `category`, `attempt`; `payload.retry`, `max_retries`, `stage`, `scope`, `matched_pattern`, `retry_detail` | The current result was saved as retry diagnostics and will be executed again according to the matched rule. | | `attempt_finished` | `task_id`, `category`, `attempt`; `status` is `completed` or `failed` | Attempt processing ended. Here, `completed` means only that the processing path returned; it does not mean the answer was correct. | | `partial_saved` | `task_id`, `category` | A task-level result was persisted. | | `task_finished` | `task_id`, `category`; `status` is `completed`, `failed`, or `skipped`; `payload.index`, `payload.total` | The task left scheduled execution. | | `summary_started` | No additional fields | Final aggregation started. | | `run_finished` | `status` is `completed`, `failed`, `cancelled`, or `timed_out`; `payload.error` may be present when error text is available | The request reached a terminal state. | Matching `task_started` and `task_finished` events use the same `payload.index` and `payload.total`. These values are the sequence number and task count used during scheduling; they are not task identifiers. Always use `task_id` to identify a task. A multi-evaluation orchestration normally preserves positions in the original selected list, so reuse can leave gaps. A single evaluation request may instead renumber the remaining tasks. The fields in `attempt_retry.payload` mean: | Field | Meaning | | ----------------- | ------------------------------------------------------------------------------------------------------------ | | `retry` | Number of runtime retries already used within the current evaluation attempt, starting at `1`. | | `max_retries` | Maximum runtime retries allowed within the current evaluation attempt. | | `stage` | Execution stage in which the error was detected. | | `scope` | Retry scope. `attempt` reruns the entire evaluation attempt; `evaluate` reruns only scoring or verification. | | `matched_pattern` | Error regular expression that matched, or `` when no filter list was configured. | | `retry_detail` | Path to the diagnostic file containing the discarded result and error. | Current `phase_changed.phase` values are: | Phase | Meaning | | ---------------------- | --------------------------------------------------------------------------------------------------- | | `plan` | Resolve the task-level execution plan and recipes. | | `open_environment` | Create the run environment. | | `prepare_task` | Prepare task material in the environment. | | `start_harness` | Start the harness session. | | `run_harness` | Run the agent through a harness. | | `run_task` | Let a harness-free benchmark run inference directly. | | `collect_artifacts` | Collect run artifacts. | | `evaluate_environment` | Create a separate verification environment when the benchmark requires one. | | `evaluate` | Score or verify the result. | | `save_partial` | Persist the task-level result. | | `analyze` | Update saved analysis output while re-analyzing existing results. It appears only in that workflow. | Events from concurrent tasks can interleave. Filter by `task_id` and `attempt` to follow one task. Do not assume that every task enters the same phases, and do not infer dependencies from adjacent events belonging to different tasks. When [`agentcompass analysis`](/en/user_guide/using_agentcompass/cli/analysis) runs, AgentCompass removes both existing progress files from the target result directory before recording the new analysis events. Without `--override`, the target is a newly created result copy, so the source directory is not modified. Re-analysis keeps the original request's `run_id`, but it does not rebuild `run_info.json`, `params.json`, or the run-directory log. Those files continue to describe the original evaluation request. ## `logs/*.log` Each normally executed `run` or `launch` request creates one `logs/YYYYMMDD_HHMMSS.log` in its run directory. If that timestamp is already in use, AgentCompass advances it one second at a time until it finds an available name. A directory copied from another run can therefore contain multiple log files. Logging begins after the run directory is created, before `run_info.json` and later runtime checks. Earlier output from the CLI or SDK is not copied into this file. Each line uses this structure: ```text theme={"system"} HH:MM:SS LEVEL logger-name message ``` * `--file-log-level` controls the minimum level in the run-directory log and defaults to `DEBUG`; `--log-level` controls console output independently. * Third-party loggers are filtered to `WARNING` and above by default, even when the file level is `DEBUG`. * The file contains messages recorded by AgentCompass and integrated components. It does not guarantee every shell command, provider response, or internal third-party event. * Logs are not structured results and are not inputs to summary generation, reuse, or analysis regeneration. `run_info.json` and `params.json` redact recognized credential fields and remove underscore-prefixed runtime fields from parameter objects. This is not a general sensitive-data scanner, and it does not apply to logs. Custom fields, free-form text, progress events, and logs may still contain paths, URLs, task data, provider information, or tracebacks. Review and remove sensitive content before sharing a run directory. ## Diagnose a Failed Run Check the files in this order to narrow down a failure: 1. Inspect `progress.json` for the request state and task counts. While the request is active, it also shows current phases. 2. Filter `progress.jsonl` by `task_id` to reconstruct the failing task's last phase, attempts, and retry path. A terminal snapshot clears active tasks, so use the event stream for the last phase after the request ends. 3. Inspect `run_info.json` for the merged request, reuse source, and that attempt's recipe and network-policy summary. 4. If the failure involves result persistence or summary regeneration, inspect `params.json`. 5. Search `logs/*.log` by task ID, phase, or exception type for detailed messages and tracebacks. Progress files describe the running process. A write failure produces a warning but does not stop the evaluation, so these files can be stale or incomplete. A forced exit can also leave `run_info.json` and the progress files in different states. Use persisted task details and summaries to determine the final evaluation results. See [Task Results](/en/user_guide/other_features/results/task_results) for task-level fields and [Summary and Analysis Results](/en/user_guide/other_features/results/summary_analysis) for aggregate metrics. See [Troubleshooting Runs](/en/user_guide/other_features/troubleshooting) for a diagnosis workflow and [Run Controls](/en/user_guide/using_agentcompass/run_controls#logs-and-progress) for log-level and progress-display options.