Summary and Analysis Results

This page explains the Benchmark summaries and analyzer summaries in a run directory, so you can choose the right file and understand its fields.

The two result families answer different questions:

• Benchmark summaries show how many tasks were evaluated and which metrics the run achieved. They are stored in summary.md and .summary_counts.json.
• Analyzer summaries describe patterns found in trajectories, errors, or runtime metrics. They are stored in analysis_summary.json and analysis_summary.md. Analysis helps diagnose results; it does not change Benchmark verdicts.

All four files summarize the complete run rather than storing the original record for one task. details/*.json stores the per-task records written to disk and serves as the input for later summary and analysis runs. The first Benchmark summary at the end of an evaluation instead uses the results collected during that run.

Files at a Glance

File	When it is generated	What to read it for
summary.md	An evaluation reaches the summary phase and Benchmark aggregation succeeds, or agentcompass summary runs without --dry-run	Task counts, Benchmark metrics, and optional grouped details
.summary_counts.json	Generated from the same Benchmark aggregation as summary.md	Machine-readable total, evaluated, and error counts
analysis_summary.json	Analysis is enabled and at least one saved task contains an aggregatable analysis_result, or agentcompass analysis produces aggregatable results	Analyzer statistics, analyzer error counts, bad-case file indexes, and data distributions
analysis_summary.md	Generated from the same analysis aggregation as analysis_summary.json	Human-readable overall, category, and distribution analysis

summary.md may be absent while a run is active, when it stops before the summary phase, or when Benchmark aggregation fails. Enabling analysis also does not guarantee analysis_summary.*: AgentCompass does not write an analysis summary when there are no task details, no attempts, or no analysis_result in any attempt.

Each Markdown/JSON pair shares one aggregation result, but the two files are written one after the other rather than at the same time. If the process exits while saving, the directory may contain only one file. Rerun the corresponding summary or analysis command to restore the pair.

Benchmark Summaries

summary.md

summary.md is the readable Benchmark summary. Use it to check task counts first, then review the Benchmark metrics and optional details.

The file contains these sections in order:

Section	Contents
Title	Uppercase Benchmark ID followed by Evaluation Results
Model	Model ID recorded for the run
Common counts	Total, Evaluated, and Error
Metrics	Metric names and values returned by the Benchmark
Details: <name>	Optional grouped or supplemental Benchmark details, rendered as a table when possible and otherwise as a JSON code block

Abbreviated structure:

# <BENCHMARK> Evaluation Results

**Model:** `<model>`

**Total:** <total>
**Evaluated:** <evaluated>
**Error:** <error>

## Metrics

| Metric | Value |
| --- | --- |
| <metric-name> | <value> |

## Details: <optional-detail-name>
...

The Markdown content comes from the Benchmark aggregate result’s counts, metrics, and details. The result object also contains schema_version (the schema version) and extra (additional Benchmark-provided data), but neither field is written to summary.md.

The common counts mean:

Count	Meaning
total	Total tasks covered by this aggregation
evaluated	Tasks that produced a result countable by the Benchmark metrics
error	Tasks that the Benchmark aggregation classified as execution or evaluation errors

Do not assume that evaluated + error = total. A Benchmark may also distinguish skipped tasks, tasks without a valid verdict, or other states. The Benchmark defines the exact counting rules. Metric names, calculations, and scales also vary; see the relevant Benchmark documentation.

At the end of a normal evaluation, AgentCompass aggregates the task results collected during that run, which can include an early error that has not been written to a detail file. A separate agentcompass summary run reads details/*.json instead. The outputs normally agree, but a regenerated summary has no corresponding detail for such an early failure and can report different counts.

.summary_counts.json

.summary_counts.json is a machine-readable snapshot of the three common counts. It does not contain Benchmark metrics or grouped details:

{
  "total": 100,
  "evaluated": 96,
  "error": 4
}

Tools can use this file to read the run size and error count quickly. It does not replace per-task details, and it cannot reconstruct summary.md by itself. agentcompass summary rereads details/*.json and runs Benchmark aggregation instead of using the old counts directly.

Analyzer Summaries

Analyzer output is first saved for each attempt under attempts.<index>.analysis_result.<analyzer-family>. AgentCompass then aggregates it by task, category, and analyzer family into run-level files.

<analyzer-family> is usually an analyzer ID, but several analyzer implementations can share one family ID. The analyzer fields below refer to this ID.

analysis_summary.json

analysis_summary.json is intended for programmatic use. It also contains bad-case file indexes that the Markdown version does not show. Its top-level fields are:

Field	Contents
per_category_per_analyzer	One statistics row for each category and analyzer combination
per_category_overall	One row per category, combining all analyzers in that category
overall_per_analyzer	One row per analyzer across all categories, with an items list of matching bad-case detail files
overall	Statistics combined across all categories and analyzers
distributions	Analyzer-declared value counts or numeric distributions, organized by analyzer, category, and field

Rows in the first four fields use the same basic shape:

Field	Meaning
category	Task category; overall rows use __overall__, and uncategorized tasks use (no category)
analyzer	Analyzer family ID; rows that combine all analyzers use __overall__
total	Tasks in the current scope that contain this analysis result
badcase_count	Tasks for which is_badcase=true
error_count	Tasks whose selected analysis result contains a non-empty error
badcase_ratio	badcase_count / total, or 0 when no tasks are present
avg_score	Average numeric analyzer score, or null when no score is available
items	Present only in overall_per_analyzer; lists the details/*.json files marked as bad cases by that analyzer

Abbreviated example:

{
  "per_category_per_analyzer": [
    {
      "category": "coding",
      "analyzer": "ExceptionAnalyzer",
      "total": 12,
      "badcase_count": 2,
      "error_count": 0,
      "badcase_ratio": 0.1667,
      "avg_score": null
    }
  ],
  "per_category_overall": [
    {
      "category": "coding",
      "analyzer": "__overall__",
      "total": 12,
      "badcase_count": 2,
      "error_count": 0,
      "badcase_ratio": 0.1667,
      "avg_score": null
    }
  ],
  "overall_per_analyzer": [
    {
      "category": "__overall__",
      "analyzer": "ExceptionAnalyzer",
      "total": 20,
      "badcase_count": 3,
      "error_count": 1,
      "badcase_ratio": 0.15,
      "avg_score": null,
      "items": ["task-a.json", "_error_task-b.json"]
    }
  ],
  "overall": [
    {
      "category": "__overall__",
      "analyzer": "__overall__",
      "total": 20,
      "badcase_count": 3,
      "error_count": 1,
      "badcase_ratio": 0.15,
      "avg_score": null
    }
  ],
  "distributions": {}
}

How Multiple Attempts Are Combined

When a task has multiple attempts, AgentCompass builds the task-level analysis result as follows:

1. It starts with the attempt selected by solved_at. If no attempt succeeded, it uses the last saved attempt.
2. It then checks the other attempts. If an analyzer returns is_badcase=true, the task-level verdict for that analyzer is set to true. That attempt’s score and details are copied only when the selected attempt has no result for the analyzer. false or null results from other attempts are not added.
3. A task is counted at most once for the same analyzer.

In rows that combine all analyzers, badcase_count is the number of tasks marked by at least one analyzer, and error_count is the number with at least one analyzer error. These fields are therefore not sums of the per-analyzer counts. For combined avg_score, each task contributes the highest of its available analyzer scores.

The summary also omits some rows without useful content:

• A bad-case analyzer that finds no bad cases and reports no errors anywhere in the run is omitted. Statistics-only analyzers and analyzers with errors remain visible.
• For an analyzer that remains in the summary, a category row is hidden if the category has Boolean results that are all false and no errors. If the category has no result from that analyzer, the current structure can still retain a total: 0 row.

distributions

An analyzer can declare fields to aggregate through distribution_fields. Results are organized as distributions.<analyzer-id>.<category>.<field> and support two methods:

Method	JSON contents
value_counts	total is the number of collected values, while distribution stores up to the 50 most frequent values and their counts; each element of a list-valued field is counted separately
numeric_stats	count is the number of collected numeric values, followed by min, mean, p50, p90, p95, and max

Cross-category results use __overall__ as the category key, while uncategorized tasks use an empty string. For a retained analyzer with a matching distribution declaration, value_counts appears with total: 0 and an empty distribution even when no values were collected. numeric_stats appears only after at least one numeric value is collected.

Tasks in one run should use categories consistently: either every detail has a non-empty category, or no detail uses categories. Mixing categorized and uncategorized tasks in a custom Benchmark can prevent analysis summaries from being generated.

analysis_summary.md

analysis_summary.md is the readable view generated from the same analysis aggregation. It contains, in order:

1. The Benchmark and model heading.
2. An Overall table with Total, Badcase, Error, Badcase Ratio, and Avg Score for each analyzer, plus an __overall__ row that combines all analyzers.
3. A table with the same columns and an __overall__ row for each task category.
4. A Distributions section, when distribution data exists, with numeric-statistics and value-count tables.

The Markdown file does not list every detail filename in overall_per_analyzer[].items. Read analysis_summary.json when you need to locate bad cases by analyzer.

Generate and Regenerate Results

Generate Results During Evaluation

agentcompass run and agentcompass launch write summary.md and .summary_counts.json after each evaluation request completes Benchmark aggregation successfully. When analysis is enabled and aggregatable results exist, they also write analysis_summary.json and analysis_summary.md.

Regenerate a Benchmark Summary

agentcompass summary reads existing details/*.json, run metadata, and the recovered Benchmark configuration. By default, it replaces summary.md and .summary_counts.json in place. It does not run the agent, Benchmark verifier, or analyzers, and it does not modify task details.

With agentcompass summary --dry-run, the command prints Markdown to the terminal without changing files in the run directory.

Rerun Analyzers

agentcompass analysis reconstructs its input from saved attempt fields, normalized trajectories and their step metrics, and errors. It runs analyzers on each readable attempt. When an analyzer returns new output, the command updates analysis_result and then generates both analysis summary files.

The command does not rerun the agent or Benchmark verifier, and it does not recompute summary.md. An existing analysis_result can remain when an analyzer skips an attempt or the analysis process fails before producing new output.

By default, agentcompass analysis copies the input run and writes results to a new timestamped sibling directory. Use --output to choose the copy destination. Only --override updates analysis fields and summaries in the original directory.

Reanalysis rebuilds analyzer input from saved fields, but it cannot restore all evaluation-time context, such as tool definitions in trajectory steps, meta, and the resolved plan for each attempt. An analyzer that depends on this context can produce a different result from inline analysis during evaluation.

If the current analysis produces no aggregatable results, AgentCompass does not delete an existing analysis_summary.* in the target directory. File presence alone therefore does not prove that the current pass updated it. sample_ids passed through --benchmark-params limits which tasks rerun analyzers, but final aggregation still scans every detail in the target directory and can include existing analysis_result from unselected tasks.

Benchmark summaries and analyzer summaries are independent. Regenerating summary.md with different aggregation parameters does not rerun analyzers, and reanalysis does not update Benchmark metrics.

Use and Share Results Safely

These four files do not receive another general redaction pass or complete Markdown escaping. Free-form Benchmark details, analyzer distribution values, categories, and detail filenames under items can contain task identifiers or sensitive content and can affect Markdown structure. Inspect the files before sharing them. Do not open untrusted results with a renderer that permits raw HTML.

All four files are generated artifacts. To correct the results, rerun the tasks or regenerate the files after changing the Benchmark aggregation logic or analyzer configuration; do not edit these summary artifacts directly.

Related Pages

• Results Overview
• Task Results
• agentcompass summary
• agentcompass analysis
• Benchmarks