Task Results

Each JSON file under details/ records the result of one Benchmark task; this page calls it a task detail file. It contains the final answer, score, trajectory, errors, and the task’s evaluation attempts. If a runtime retry is triggered, AgentCompass also writes the discarded execution to retry_details/ so you can identify why it was retried.

Before reading these files, distinguish two concepts:

• An attempt is an independent evaluation attempt controlled by k and included in the final task result.
• A retry reruns recoverable work within the same evaluation attempt. It does not add another attempt or directly contribute to Benchmark metrics.

See Shared Benchmark Fields for how to configure and aggregate k and avgk.

File	When it is generated	What it stores
details/<task-id>[_<category>].json	The task produced a detail without an execution or evaluation error	The final answer, score, trajectory, and evaluation attempts. An incorrect answer or a skipped task can still use this filename.
details/_error_<task-id>[_<category>].json	At least one recorded evaluation attempt has an execution or evaluation error	The same task information as a normal detail; the _error_ prefix indicates that the file contains an error.
retry_details/*.json	The runtime determines that the current error can be retried and retry budget remains	The discarded result and the error that triggered the retry. The filename also records the evaluation-attempt number, retry number, and failure stage.

The category segment appears only when the task has a category. / and : in task IDs, categories, and stage names are replaced with _. A normal run stores multiple evaluation attempts in the attempts object of one detail file.

Only the replacements described above are applied; this is not complete path sanitization. Custom components should produce trusted, stable task IDs, categories, and stage names without backslashes, control characters, or directory segments. The combination of task_id and category must also remain unique after / and : are replaced, or different tasks can write to the same path.

Task Detail Files

Normal details and _error_ details use the same JSON structure. Top-level fields describe the complete task, while attempts stores the result of each evaluation attempt. Field contents depend on the selected Benchmark, Harness, and analyzers, so some values can be null and optional fields may be absent.

If a task fails before a result can be formed and saved, it may have no corresponding task detail file. However, the first summary at the end of the evaluation uses the results collected during that run and can still count the task as an error. See Summary and Analysis Results for the difference between the initial and regenerated summaries.

{
  "task_id": "<task-id>",
  "category": "<category>",
  "correct": true,
  "solved_at": 1,
  "attempts_tried": 1,
  "k": 1,
  "retry_count": 2,
  "retry_counts": {
    "1": 2
  },
  "attempts": {
    "1": {
      "correct": true,
      "final_answer": "<answer>",
      "ground_truth": "<reference-answer>",
      "trajectory": {},
      "status": "completed",
      "score": 1.0,
      "error": "",
      "artifacts": {},
      "extra": {},
      "analysis_result": {},
      "meta": {
        "resolved_execution_plan": {}
      }
    }
  }
}

Task-Level Fields

Field	Meaning
task_id	The task identifier provided by the Benchmark. AgentCompass uses it to identify tasks during aggregation and reuse.
category	An optional task category provided by the Benchmark. The normal runtime writes an empty string for an uncategorized task; compatible external or older results may omit it or use null.
correct	Whether at least one recorded evaluation attempt passed scoring or verification. Omitted when a non-null avgk_value is used.
solved_at	The first evaluation attempt that passed scoring or verification, numbered from 1; null if none passed. Omitted when a non-null avgk_value is used.
attempts_tried	The number of evaluation attempts actually recorded in attempts. When avgk is disabled, execution can stop after the first success, so this value can be lower than k.
k	The maximum number of evaluation attempts allowed for this task.
max_score	An optional task-level maximum score supplied by an adapter. It is absent when the producer does not provide one.
avgk_value	Optional precomputed task-level avg@k, primarily for compatibility with externally generated results. When it is non-null, the top level no longer uses correct or solved_at, and aggregation reads this value first. Normal runs omit this field and calculate avg@k from attempts.
retry_count	The total number of runtime retries actually triggered across all evaluation attempts.
retry_counts	The number of retries triggered by each evaluation attempt. Keys are string-form attempt numbers. This map is sparse: an attempt with no retry has no key.
attempts	A map of evaluation attempts keyed by string numbers such as "1" and "2". Each value uses the attempt-level structure below.

A task detail has no top-level status or score; each evaluation attempt records its own status and score. The first summary generated at the end of an evaluation uses the results collected by that run. A later, separate agentcompass summary command reads the saved detail files and recalculates the summary.

Attempt-Level Fields

To inspect one evaluation attempt, first check status and error to determine whether execution was valid, then use correct and score to review the evaluation outcome. trajectory, artifacts, extra, and meta provide further execution and diagnostic context.

Field	Meaning
correct	Whether this evaluation attempt passed the Benchmark’s scoring or verification.
final_answer	The final answer produced by the model or agent. It can be text, a patch, or structured JSON defined by the Benchmark.
ground_truth	The reference answer provided by the Benchmark. It can be null for tasks that use a hidden verifier.
trajectory	A record normalized by the Harness to the standard AgentCompass trajectory structure; null when no trajectory is available. See Trajectory Fields for its structure.
status	The execution status of this evaluation attempt. See Status Values.
score	The Benchmark score for this evaluation attempt; it can be null when only a pass/fail result is available.
max_score	An optional maximum score for this evaluation attempt. It is absent when not provided.
error	An error produced during execution or evaluation. It is normally an empty string or null; on failure it can include a stack trace.
artifacts	Additional artifact content or indexes collected by the Benchmark or Harness. Its shape is integration-specific.
extra	Additional structured data written by the Benchmark or Harness. Its fields are not consistent across Benchmarks.
analysis_result	Analyzer output generated during evaluation, keyed by analyzer family. See Analysis Results for its structure.
meta	Supplementary information written by the runtime or an integration. Besides resolved_execution_plan, component-specific fields can include plan, extra, harness_metrics, status, and scoring.

Do not rely on internal Harness metrics as a stable attempt-level field. A Benchmark or Harness that needs to retain integration-specific metrics normally writes them under meta.harness_metrics, extra, or artifacts. meta.resolved_execution_plan is only a compact summary; other component-defined fields under meta can contain more complete configuration or diagnostic information.

Status Values

status	Meaning
completed	Execution and evaluation produced a valid result. This does not mean that the answer is correct.
run_error	The task execution phase failed.
eval_error	The scoring or verification phase failed.
run_error_or_eval_error	Both execution and evaluation failed, or the failure cannot be assigned to only one of them.
skipped	This evaluation attempt was skipped.

Trajectory Fields

ACTF_v1.0 is a trajectory schema version defined by AgentCompass. It gives different Harness implementations a common representation for agent execution records; it is not a protocol defined by a model provider or third-party agent framework.

trajectory uses this structure to record model input and output, tool calls, Environment observations, timing, and token metrics in execution order. Which fields contain values depends on the Harness; when a Harness does not produce a trajectory, trajectory is null.

Field	Meaning
schema_version	The AgentCompass trajectory schema version. The current default is ACTF_v1.0.
steps	An array of interaction steps in execution order.
started_at	The start time of the complete trajectory.
finished_at	The finish time of the complete trajectory.

Each element of steps[] contains:

Field	Meaning
step_id	The step number within the trajectory.
system_prompt	The system prompt used for this step.
user_content	User content or subsequent input sent to the model.
tools	Tool information recorded for this step; the exact contents depend on the Harness.
assistant_content.content	The assistant’s visible content for this step.
assistant_content.reasoning_content	Optional reasoning content supplied by the Harness.
assistant_content.tool_calls	Tool calls requested by the assistant during this step.
observation	Observations returned by tool or Environment actions.
metric.prompt_tokens_len	The number of input tokens for this step; null when unavailable.
metric.completion_tokens_len	The number of output tokens for this step; null when unavailable.
metric.llm_infer_ms	Model inference time in milliseconds.
metric.env_action_ms	Environment action time in milliseconds.
metric.stop_reason	The reason the model response stopped.
started_at	The start time of this step.
finished_at	The finish time of this step.

Resolved Execution Plan

attempts.<N>.meta.resolved_execution_plan records the Environments, network policies, and Recipes resolved for this evaluation attempt. This summary is created before the Environment is opened. It therefore shows that the plan was resolved, but does not prove that the Environment was created successfully or include its complete configuration.

Field	Meaning
environment	The Environment planned for task execution. It contains its id and the network_policy used for Environment startup, Benchmark preparation, and Harness setup.
evaluation_environment	The separate Environment planned for scoring. It contains its id and startup network_policy, and can be null when none is configured.
run_network_policy	The network policy used while the Harness or Benchmark performs model and tool operations.
verifier_network_policy	The network policy used during Benchmark scoring or verification.
applied_recipes	The Recipe IDs actually applied to this task.

Each network_policy object above contains network_mode and allowed_hosts. network_mode identifies the network mode, while allowed_hosts lists the hosts that can be accessed. See Network Policies for the meaning of each setting.

Analysis Results

When agentcompass analysis runs with an evaluation, analysis_result stores each evaluation attempt’s output by analyzer family. A successful analysis can contain the fields below; a failed analysis may contain only a subset:

Field	Meaning
is_badcase	Whether the analyzer classified this result as a bad case. Statistics-only analyzers can return null.
details	A structured explanation object produced by the analyzer; normally an empty object when no details are available.
score	An optional score produced by the analyzer.
error	An error from the analyzer itself. It is normally absent when no error occurred.
extra	Optional additional data produced by the analyzer.

If a selected analyzer’s analysis() call raises an exception, AgentCompass normally writes is_badcase: false and the exception under that family’s error, but omits details. The error does not change the Benchmark’s existing status, correct, or score. If the failure occurs while creating or matching the analyzer, or while checking its requirements, that family may not appear in analysis_result; consult the logs to identify the cause.

Error Detail Files

The _error_ prefix marks a task detail that contains an execution or evaluation error. It is used when any recorded evaluation attempt meets either condition:

• status is run_error, eval_error, or run_error_or_eval_error;
• error is non-empty.

To support result structures produced by different integrations, meta.status: "error" also causes this prefix to be used.

_error_ does not mean that the answer was merely incorrect. It means the detail contains an execution or evaluation error and therefore cannot be reused. If multiple evaluation attempts include both completed and error states, one attempt that meets a condition above is enough to give the complete task detail this prefix. A task with status: "completed" and correct: false uses a normal detail filename.

With --reuse, AgentCompass reuses only normal details. Tasks that have only an _error_ detail are run again in the new run, and the source run is not modified. If a normal detail for that task is later written in the target directory, its stale error counterpart is removed.

Retry Detail Files

The runtime reruns work and writes a retry detail only when an error matches the retry rules and retry budget remains. Therefore, the absence of a retry detail does not mean that the task did not fail. A final failure that is not retried normally remains in an _error_ task detail; if no result could be formed and saved when the failure occurred, there may be no detail file. See Retry Only Transient Failures for rules and budgets.

{
  "schema_version": "agentcompass.retry.v1",
  "task_id": "<task-id>",
  "category": "<category>",
  "attempt": 1,
  "retry": 1,
  "max_retries": 2,
  "stage": "evaluate",
  "scope": "evaluate",
  "matched_pattern": "<matched-regex>",
  "error": "<error-message>",
  "discarded_result": {}
}

Field	Meaning
schema_version	The retry-detail schema version. The current value is agentcompass.retry.v1.
task_id	The Benchmark task ID whose work was retried.
category	The optional task category.
attempt	The evaluation attempt to which this retry belongs, numbered from 1.
retry	The retry number within the current evaluation attempt, numbered from 1. It resets for the next evaluation attempt.
max_retries	The maximum number of runtime retries available to each evaluation attempt.
stage	The lifecycle stage in which the retry was triggered. Common values are listed below.
scope	The amount of work restarted by the retry: attempt or evaluate.
matched_pattern	The first regular expression that matched the error text. If no retry patterns were configured, any non-empty error matches and this field is <default:any-error>.
error	The error text that triggered the retry. For exceptions, it normally includes a stack trace.
discarded_result	The discarded result snapshot, including meta.resolved_execution_plan. If no result existed yet, the runtime constructs an error result.

discarded_result is for diagnosis only and preserves as much of the discarded result as possible, so it can have more fields than an evaluation attempt in details/*.json. It normally contains the status, correct, score, final_answer, ground_truth, trajectory, error, artifacts, extra, and meta fields described above. It can also contain:

Field	Meaning
task_id	The task ID associated with the discarded result.
category	The optional task category associated with the discarded result.
metrics	Raw metrics returned by the Harness, for diagnosis only. This is not a stable field in normal task details.
Other fields	A Benchmark or Harness that returns a dictionary result can retain its own additional fields, whose shape is integration-specific.

Use scope to determine which work the retry repeats:

scope	Behavior
attempt	Restart the complete current evaluation attempt.
evaluate	Rerun only scoring or verification without creating another evaluation attempt.

Use stage to identify the earliest phase that failed:

stage	Phase
plan	The retry occurred before a more specific task stage was entered.
open_environment	Create the task execution Environment.
prepare_task	Prepare Benchmark input and the workspace.
run_task	Run a Benchmark task that does not use a Harness.
start_harness	Start the Harness session.
run_harness	Execute the task through the Harness.
collect_artifacts	Collect task artifacts.
evaluate_environment	Create a separate scoring Environment.
evaluate	Perform scoring or verification.
attempt	Fallback when no more specific stage is available.

Handle Sensitive Content

Before writing task and retry details, AgentCompass recursively redacts credential fields that it recognizes. Answers, prompts, observations, stack traces, and integration-specific data can still contain task content or other sensitive text. Protect these files like logs, and review their contents before publishing a run directory.

details/*.json feeds aggregation, and normal detail files can also be reused. retry_details/*.json is for diagnostics only. To correct evaluation configuration or results, rerun the task instead of editing these files directly.

Related Pages

• Results Overview
• Run Records and Diagnostics
• Summary and Analysis Results
• Run Controls
• agentcompass analysis
• Network Policies