Results

Inspect task details, summaries, progress, logs, and run artifacts.

Results are the durable record of an AgentCompass run. They are designed for both human inspection and later automation: summary regeneration, post-analysis, badcase mining, and interrupted-run recovery all start from the run directory.

Directory Layout

results/
  [<run-name>/]
    <benchmark>/
      <model>/
        <run-id>/
          details/
          retry_details/
          logs/
          run_info.json
          params.json
          progress.json
          progress.jsonl
          .summary_counts.json
          summary.md
          analysis_summary.json
          analysis_summary.md

details/*.json files are the source of truth. Completed samples use the task id, with an optional category suffix, as the filename. Failed results use the _error_ prefix so a reuse run can schedule them again. Summary files are derived views. run-name is omitted from the path when it is empty.

Artifact Roles

Artifact	Purpose
details/*.json	Per-task measured attempts, benchmark verdicts and scores, predictions, errors, trajectories, and analyzer output.
retry_details/*.json	Discarded failed executions, matched retry patterns, and the stage that consumed each retry.
run_info.json	Sanitized run request, reuse source, and recipe-resolved execution plans by task and attempt.
summary.md	Human-readable run-level metrics and status.
params.json	Sanitized persistence and effective parameter payload used to write and summarize results.
.summary_counts.json	Internal aggregate counts used to regenerate the summary.
progress.jsonl	Append-only structured progress stream.
progress.json	Latest progress snapshot.
logs/*.log	Runtime logs for setup, execution, release, and errors.
analysis_summary.*	Aggregated analyzer output when analysis is enabled or re-run.

details/*.json Field Reference

One detail file represents one benchmark task, not one model call. Its outer envelope is stable across integrations, while values under artifacts, extra, and parts of meta are intentionally extensible. A run with k > 1 normally stores all measured executions in the same file under the string-keyed attempts object.

File Names

Pattern	Meaning
<task-id>.json	A normal result for a task without a category.
<task-id>_<category>.json	A normal result whose category participates in result grouping and reuse matching.
_error_<task-id>[_<category>].json	The task contains an invalid execution: at least one attempt has an error status or a non-empty error. Reuse ignores this file and schedules the task again.

/ and : in task ids or categories are replaced with _ when the filename is created. The normal runner keeps attempts inside the JSON object rather than creating one file per attempt. An _error_ file can still contain a model answer, trajectory, score, or verifier output; the prefix classifies execution validity, not answer quality alone. A later valid result is written to the normal filename and removes the stale error file.

The following abbreviated object shows how the field groups fit together. Fields that are unavailable for a particular benchmark, harness, or failure point may be null, empty, or omitted.

{
  "task_id": "example-task",
  "category": "category-name",
  "correct": true,
  "solved_at": 1,
  "attempts_tried": 1,
  "k": 1,
  "retry_count": 0,
  "retry_counts": {},
  "attempts": {
    "1": {
      "correct": true,
      "status": "completed",
      "score": 1.0,
      "final_answer": "...",
      "ground_truth": "...",
      "trajectory": {
        "schema_version": "ACTF_v1.0",
        "steps": [],
        "started_at": "...",
        "finished_at": "..."
      },
      "error": "",
      "artifacts": {},
      "extra": {},
      "analysis_result": {},
      "meta": {
        "resolved_execution_plan": {}
      }
    }
  }
}

Task-level Fields

Field	Type	Meaning
task_id	string	Stable benchmark task identifier. It is also the primary key used for result lookup and reuse matching.
category	string or null	Optional benchmark grouping, such as a task type or domain. When non-empty, it is included in the detail filename and category aggregates.
correct	boolean	Aggregate task verdict for the standard result mode. It is true when at least one measured attempt succeeds. It is omitted when a precomputed avgk_value is persisted instead.
solved_at	integer or null	One-based index of the first successful measured attempt. null means no recorded attempt succeeded. It is omitted with precomputed avgk_value output.
attempts_tried	integer	Number of entries actually stored in attempts. It may be smaller than k when non-avg@k execution stops after the first success. Runtime retries do not increase it.
k	integer	Maximum number of countable, independently measured attempts requested for this task.
avgk_value	number	Optional precomputed per-task avg@k value supported by the persistence and summary path. When present, it replaces the task-level correct and solved_at fields.
max_score	number	Optional task-level score ceiling supplied by an adapter. The standard runner normally keeps benchmark scoring at attempt level instead.
retry_count	integer	Total number of transient failed executions discarded and retried before the measured attempts were finalized.
retry_counts	object	Transient retry counts keyed by the one-based measured attempt number, for example {"1": 2}. The corresponding diagnostics are stored in retry_details/.
attempts	object	Measured attempt payloads keyed by string indices ("1", "2", and so on). These entries, unlike runtime retries, contribute to task metrics.

Task-level status and score are removed at the persistence boundary. Their countable values belong to individual attempts so multi-attempt results remain unambiguous.

Attempt-level Fields

Field	Type	Meaning
correct	boolean or null	Benchmark verifier verdict for this attempt. null is possible when execution ended before a verdict was produced.
status	string or null	Execution state, typically completed, run_error, eval_error, run_error_or_eval_error, or skipped.
score	number or null	Scalar score returned by the benchmark evaluator when one is available. Binary benchmarks commonly use correct and may leave this field null.
max_score	number or null	Optional upper bound or score scale supplied by an integration. It is omitted unless the result producer provides it.
final_answer	any JSON value	Harness output submitted for evaluation. Depending on the benchmark, this can be text, a patch, a structured answer, or a reference to generated output.
ground_truth	any JSON value	Benchmark reference answer or evaluation target when the integration exposes it. It may be absent for hidden-verifier benchmarks.
trajectory	object or null	Normalized AgentCompass trajectory. It is null when a failure occurs before the harness produces a trace. Raw harness-native traces normally live in artifacts.
error	string or null	Execution or evaluation error, often including a traceback. A non-empty value makes the task result error-prefixed even if other fields contain usable partial output.
artifacts	object or null	Integration-specific durable outputs, such as generated files, raw trajectories, verifier captures, or reports. Artifact names and value shapes are not a cross-benchmark contract.
extra	object or null	Integration-specific structured metadata that does not belong in the stable result envelope, such as raw evaluator data or harness metrics.
analysis_result	object or null	Results produced by eligible post-run analyzers, keyed by analyzer family. It is empty when analysis is disabled, skipped, or not applicable.
meta	object	Runtime audit metadata. Current runs attach the recipe-resolved plan under resolved_execution_plan; older result files may use a different compatibility shape.

Normalized Trajectory

When trajectory is available, AgentCompass uses the ACTF trajectory envelope. The default schema version is ACTF_v1.0; integrations may preserve additional values inside individual steps.

Field	Meaning
schema_version	Trajectory schema identifier used by readers and analyzers.
started_at, finished_at	Serialized timestamps bounding the complete harness run.
steps	Ordered list of model-and-environment interaction steps.

Each normalized step can contain the following fields:

Field	Meaning
step_id	Harness-assigned step index.
system_prompt	System instructions visible for the step, when retained by the harness.
user_content	User or task input sent for the step.
assistant_content.content	Assistant-visible response text.
assistant_content.reasoning_content	Reasoning content when the model endpoint and retention policy expose it.
assistant_content.tool_calls	Tool invocation payloads normalized or retained by the harness.
tools	Tool definitions or tool context available at that step.
observation	Environment or tool output returned to the agent.
metric.prompt_tokens_len	Input-token count for the model call, when available.
metric.completion_tokens_len	Output-token count for the model call, when available.
metric.llm_infer_ms	Model inference latency in milliseconds.
metric.env_action_ms	Environment action latency in milliseconds.
metric.stop_reason	Model or harness stop reason for the step.
started_at, finished_at	Serialized timestamps bounding the step.

For harness-specific fields, inspect the raw trajectory referenced by artifacts and the corresponding Harness module.

Resolved Execution Plan

attempts.<index>.meta.resolved_execution_plan records the security- and recipe-relevant plan that was actually used, after configuration precedence and recipe adaptation were applied. It is more useful for auditing than the original CLI request alone.

Field	Meaning
environment	Primary environment id and its setup-phase network_policy.
evaluation_environment	Separate verifier environment and setup policy when a benchmark uses one; otherwise null.
run_network_policy	Network policy applied while the harness runs.
verifier_network_policy	Network policy applied during benchmark verification.
applied_recipes	Ordered recipe ids that adapted the task execution plan.

Each network policy contains network_mode (public, no-network, or allowlist) and allowed_hosts. The detail record intentionally stores only the environment identity and network policy here, rather than every provider parameter. Use run_info.json and params.json for the sanitized run request and persisted effective parameters.

Analyzer Results

Each analysis_result.<analyzer-family> object has a small common shape:

Field	Meaning
is_badcase	Whether the analyzer classified this attempt as a bad case.
details	Human-readable or structured explanation produced by the analyzer.
score	Optional analyzer-specific score.
error	Optional analyzer failure message. Analyzer failure does not replace the benchmark attempt result.
extra	Optional analyzer-specific structured output.

Detail payloads are written atomically and recursively redact recognized credential fields before persistence. Nevertheless, answers, prompts, observations, and benchmark artifacts can still contain sensitive task data; apply the same access controls you use for run logs. Treat details/ as generated source data: edit analyzers or regenerate derived summaries instead of manually changing these files.

Local Result Browser

tools/result-browser provides a local web UI for inspecting completed or in-progress run directories. It is served from a source checkout and requires Node.js and npm on the machine running the UI service.

From the repository root:

cd tools/result-browser
npm install
npm run dev

Open the Vite URL printed by npm run dev, usually http://localhost:5173, then enter the absolute run directory path that contains summary.md and details/, for example /path/to/AgentCompass/results/swebench_verified/$MODEL_NAME/20260703_120000.

The entered path is resolved on the machine running npm run dev. If you access the UI through SSH port forwarding or a remote forwarded URL, still enter the server-side absolute path to the run directory.

To check the production build locally:

npm run build
npm run preview

Summary vs Analysis

Both commands derive new views from an existing run without rerunning the agent:

Command	Reads	Writes	Detailed usage
agentcompass analysis	Existing task attempts, trajectories, metrics, and errors	Per-task analysis_result plus analysis_summary.json and analysis_summary.md	Analysis CLI
agentcompass summary	Existing task details and run metadata	Recomputed benchmark aggregates and summary.md	Summary CLI

summary is lightweight and can be previewed without writing. analysis may invoke configured qualitative models and copies the run by default; in-place mutation requires an explicit --override.

Data, Cache, and Output Directories

AgentCompass keeps downloaded or prepared benchmark data separate from durable evaluation results. Logs live inside each run directory, alongside task details and generated summaries.

Setting	Default	Meaning
runtime.data_dir / --data-dir	data	Benchmark datasets and prepared data cache.
runtime.results_dir / --results-dir	results	Root directory for run outputs.
--run-name	empty	Optional namespace under the result root.
--run-id	timestamp	Explicit run id for the new result directory.

runtime.data_dir and runtime.results_dir are YAML keys in config/defaults.yaml with matching CLI flags. --run-name and --run-id are per-run CLI flags (fields of the RunRequest output spec, not configuration file keys).

Override the data and result roots for one run when needed:

export MODEL_NAME=""

agentcompass run <benchmark> <harness> "$MODEL_NAME" \
  --data-dir <data_dir> \
  --results-dir <results_dir>

Related Pages

• agentcompass run
• agentcompass analysis
• agentcompass summary
• CLI Overview