> ## Documentation Index > Fetch the complete documentation index at: https://agent-compass.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Summary and Analysis Results This page explains the Benchmark summaries and analyzer summaries in a run directory, so you can choose the right file and understand its fields. The two result families answer different questions: * Benchmark summaries show how many tasks were evaluated and which metrics the run achieved. They are stored in `summary.md` and `.summary_counts.json`. * Analyzer summaries describe patterns found in trajectories, errors, or runtime metrics. They are stored in `analysis_summary.json` and `analysis_summary.md`. Analysis helps diagnose results; it does not change Benchmark verdicts. All four files summarize the complete run rather than storing the original record for one task. [`details/*.json`](/en/user_guide/other_features/results/task_results) stores the per-task records written to disk and serves as the input for later summary and analysis runs. The first Benchmark summary at the end of an evaluation instead uses the results collected during that run. ## Files at a Glance | File | When it is generated | What to read it for | | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | | `summary.md` | An evaluation reaches the summary phase and Benchmark aggregation succeeds, or `agentcompass summary` runs without `--dry-run` | Task counts, Benchmark metrics, and optional grouped details | | `.summary_counts.json` | Generated from the same Benchmark aggregation as `summary.md` | Machine-readable `total`, `evaluated`, and `error` counts | | `analysis_summary.json` | Analysis is enabled and at least one saved task contains an aggregatable `analysis_result`, or `agentcompass analysis` produces aggregatable results | Analyzer statistics, analyzer error counts, bad-case file indexes, and data distributions | | `analysis_summary.md` | Generated from the same analysis aggregation as `analysis_summary.json` | Human-readable overall, category, and distribution analysis | `summary.md` may be absent while a run is active, when it stops before the summary phase, or when Benchmark aggregation fails. Enabling analysis also does not guarantee `analysis_summary.*`: AgentCompass does not write an analysis summary when there are no task details, no attempts, or no `analysis_result` in any attempt. Each Markdown/JSON pair shares one aggregation result, but the two files are written one after the other rather than at the same time. If the process exits while saving, the directory may contain only one file. Rerun the corresponding `summary` or `analysis` command to restore the pair. ## Benchmark Summaries ### `summary.md` `summary.md` is the readable Benchmark summary. Use it to check task counts first, then review the Benchmark metrics and optional details. The file contains these sections in order: | Section | Contents | | ----------------- | ------------------------------------------------------------------------------------------------------------------------ | | Title | Uppercase Benchmark ID followed by `Evaluation Results` | | Model | Model ID recorded for the run | | Common counts | `Total`, `Evaluated`, and `Error` | | `Metrics` | Metric names and values returned by the Benchmark | | `Details: ` | Optional grouped or supplemental Benchmark details, rendered as a table when possible and otherwise as a JSON code block | Abbreviated structure: ```markdown theme={"system"} # Evaluation Results **Model:** `` **Total:** **Evaluated:** **Error:** ## Metrics | Metric | Value | | --- | --- | | | | ## Details: ... ``` The Markdown content comes from the Benchmark aggregate result's `counts`, `metrics`, and `details`. The result object also contains `schema_version` (the schema version) and `extra` (additional Benchmark-provided data), but neither field is written to `summary.md`. The common counts mean: | Count | Meaning | | ----------- | --------------------------------------------------------------------------------- | | `total` | Total tasks covered by this aggregation | | `evaluated` | Tasks that produced a result countable by the Benchmark metrics | | `error` | Tasks that the Benchmark aggregation classified as execution or evaluation errors | Do not assume that `evaluated + error = total`. A Benchmark may also distinguish skipped tasks, tasks without a valid verdict, or other states. The Benchmark defines the exact counting rules. Metric names, calculations, and scales also vary; see the relevant [Benchmark documentation](/en/user_guide/modules/benchmarks/overview). At the end of a normal evaluation, AgentCompass aggregates the task results collected during that run, which can include an early error that has not been written to a detail file. A separate `agentcompass summary` run reads `details/*.json` instead. The outputs normally agree, but a regenerated summary has no corresponding detail for such an early failure and can report different counts. ### `.summary_counts.json` `.summary_counts.json` is a machine-readable snapshot of the three common counts. It does not contain Benchmark metrics or grouped details: ```json theme={"system"} { "total": 100, "evaluated": 96, "error": 4 } ``` Tools can use this file to read the run size and error count quickly. It does not replace per-task details, and it cannot reconstruct `summary.md` by itself. `agentcompass summary` rereads `details/*.json` and runs Benchmark aggregation instead of using the old counts directly. ## Analyzer Summaries Analyzer output is first saved for each attempt under `attempts..analysis_result.`. AgentCompass then aggregates it by task, category, and analyzer family into run-level files. `` is usually an analyzer ID, but several analyzer implementations can share one family ID. The `analyzer` fields below refer to this ID. ### `analysis_summary.json` `analysis_summary.json` is intended for programmatic use. It also contains bad-case file indexes that the Markdown version does not show. Its top-level fields are: | Field | Contents | | --------------------------- | --------------------------------------------------------------------------------------------------- | | `per_category_per_analyzer` | One statistics row for each category and analyzer combination | | `per_category_overall` | One row per category, combining all analyzers in that category | | `overall_per_analyzer` | One row per analyzer across all categories, with an `items` list of matching bad-case detail files | | `overall` | Statistics combined across all categories and analyzers | | `distributions` | Analyzer-declared value counts or numeric distributions, organized by analyzer, category, and field | Rows in the first four fields use the same basic shape: | Field | Meaning | | --------------- | ------------------------------------------------------------------------------------------------------------- | | `category` | Task category; overall rows use `__overall__`, and uncategorized tasks use `(no category)` | | `analyzer` | Analyzer family ID; rows that combine all analyzers use `__overall__` | | `total` | Tasks in the current scope that contain this analysis result | | `badcase_count` | Tasks for which `is_badcase=true` | | `error_count` | Tasks whose selected analysis result contains a non-empty `error` | | `badcase_ratio` | `badcase_count / total`, or `0` when no tasks are present | | `avg_score` | Average numeric analyzer `score`, or `null` when no score is available | | `items` | Present only in `overall_per_analyzer`; lists the `details/*.json` files marked as bad cases by that analyzer | Abbreviated example: ```json theme={"system"} { "per_category_per_analyzer": [ { "category": "coding", "analyzer": "ExceptionAnalyzer", "total": 12, "badcase_count": 2, "error_count": 0, "badcase_ratio": 0.1667, "avg_score": null } ], "per_category_overall": [ { "category": "coding", "analyzer": "__overall__", "total": 12, "badcase_count": 2, "error_count": 0, "badcase_ratio": 0.1667, "avg_score": null } ], "overall_per_analyzer": [ { "category": "__overall__", "analyzer": "ExceptionAnalyzer", "total": 20, "badcase_count": 3, "error_count": 1, "badcase_ratio": 0.15, "avg_score": null, "items": ["task-a.json", "_error_task-b.json"] } ], "overall": [ { "category": "__overall__", "analyzer": "__overall__", "total": 20, "badcase_count": 3, "error_count": 1, "badcase_ratio": 0.15, "avg_score": null } ], "distributions": {} } ``` #### How Multiple Attempts Are Combined When a task has multiple attempts, AgentCompass builds the task-level analysis result as follows: 1. It starts with the attempt selected by `solved_at`. If no attempt succeeded, it uses the last saved attempt. 2. It then checks the other attempts. If an analyzer returns `is_badcase=true`, the task-level verdict for that analyzer is set to `true`. That attempt's `score` and `details` are copied only when the selected attempt has no result for the analyzer. `false` or `null` results from other attempts are not added. 3. A task is counted at most once for the same analyzer. In rows that combine all analyzers, `badcase_count` is the number of tasks marked by at least one analyzer, and `error_count` is the number with at least one analyzer error. These fields are therefore not sums of the per-analyzer counts. For combined `avg_score`, each task contributes the highest of its available analyzer scores. The summary also omits some rows without useful content: * A bad-case analyzer that finds no bad cases and reports no errors anywhere in the run is omitted. Statistics-only analyzers and analyzers with errors remain visible. * For an analyzer that remains in the summary, a category row is hidden if the category has Boolean results that are all `false` and no errors. If the category has no result from that analyzer, the current structure can still retain a `total: 0` row. #### `distributions` An analyzer can declare fields to aggregate through `distribution_fields`. Results are organized as `distributions...` and support two methods: | Method | JSON contents | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `value_counts` | `total` is the number of collected values, while `distribution` stores up to the 50 most frequent values and their counts; each element of a list-valued field is counted separately | | `numeric_stats` | `count` is the number of collected numeric values, followed by `min`, `mean`, `p50`, `p90`, `p95`, and `max` | Cross-category results use `__overall__` as the category key, while uncategorized tasks use an empty string. For a retained analyzer with a matching distribution declaration, `value_counts` appears with `total: 0` and an empty `distribution` even when no values were collected. `numeric_stats` appears only after at least one numeric value is collected. Tasks in one run should use categories consistently: either every detail has a non-empty `category`, or no detail uses categories. Mixing categorized and uncategorized tasks in a custom Benchmark can prevent analysis summaries from being generated. ### `analysis_summary.md` `analysis_summary.md` is the readable view generated from the same analysis aggregation. It contains, in order: 1. The Benchmark and model heading. 2. An `Overall` table with `Total`, `Badcase`, `Error`, `Badcase Ratio`, and `Avg Score` for each analyzer, plus an `__overall__` row that combines all analyzers. 3. A table with the same columns and an `__overall__` row for each task category. 4. A `Distributions` section, when distribution data exists, with numeric-statistics and value-count tables. The Markdown file does not list every detail filename in `overall_per_analyzer[].items`. Read `analysis_summary.json` when you need to locate bad cases by analyzer. ## Generate and Regenerate Results ### Generate Results During Evaluation [`agentcompass run`](/en/user_guide/using_agentcompass/cli/run) and [`agentcompass launch`](/en/user_guide/using_agentcompass/cli/launch) write `summary.md` and `.summary_counts.json` after each evaluation request completes Benchmark aggregation successfully. When analysis is enabled and aggregatable results exist, they also write `analysis_summary.json` and `analysis_summary.md`. ### Regenerate a Benchmark Summary [`agentcompass summary`](/en/user_guide/using_agentcompass/cli/summary) reads existing `details/*.json`, run metadata, and the recovered Benchmark configuration. By default, it replaces `summary.md` and `.summary_counts.json` in place. It does not run the agent, Benchmark verifier, or analyzers, and it does not modify task details. With [`agentcompass summary --dry-run`](/en/user_guide/using_agentcompass/cli/summary#preview-the-summary), the command prints Markdown to the terminal without changing files in the run directory. ### Rerun Analyzers [`agentcompass analysis`](/en/user_guide/using_agentcompass/cli/analysis#re-run-on-existing-results) reconstructs its input from saved attempt fields, normalized trajectories and their step metrics, and errors. It runs analyzers on each readable attempt. When an analyzer returns new output, the command updates `analysis_result` and then generates both analysis summary files. The command does not rerun the agent or Benchmark verifier, and it does not recompute `summary.md`. An existing `analysis_result` can remain when an analyzer skips an attempt or the analysis process fails before producing new output. By default, `agentcompass analysis` copies the input run and writes results to a new timestamped sibling directory. Use `--output` to choose the copy destination. Only `--override` updates analysis fields and summaries in the original directory. Reanalysis rebuilds analyzer input from saved fields, but it cannot restore all evaluation-time context, such as tool definitions in trajectory steps, `meta`, and the resolved plan for each attempt. An analyzer that depends on this context can produce a different result from inline analysis during evaluation. If the current analysis produces no aggregatable results, AgentCompass does not delete an existing `analysis_summary.*` in the target directory. File presence alone therefore does not prove that the current pass updated it. [`sample_ids` passed through `--benchmark-params`](/en/user_guide/using_agentcompass/cli/analysis#options) limits which tasks rerun analyzers, but final aggregation still scans every detail in the target directory and can include existing `analysis_result` from unselected tasks. Benchmark summaries and analyzer summaries are independent. Regenerating `summary.md` with different aggregation parameters does not rerun analyzers, and reanalysis does not update Benchmark metrics. ## Use and Share Results Safely These four files do not receive another general redaction pass or complete Markdown escaping. Free-form Benchmark `details`, analyzer distribution values, categories, and detail filenames under `items` can contain task identifiers or sensitive content and can affect Markdown structure. Inspect the files before sharing them. Do not open untrusted results with a renderer that permits raw HTML. All four files are generated artifacts. To correct the results, rerun the tasks or regenerate the files after changing the Benchmark aggregation logic or analyzer configuration; do not edit these summary artifacts directly. ## Related Pages * [Results Overview](/en/user_guide/other_features/results) * [Task Results](/en/user_guide/other_features/results/task_results) * [`agentcompass summary`](/en/user_guide/using_agentcompass/cli/summary) * [`agentcompass analysis`](/en/user_guide/using_agentcompass/cli/analysis) * [Benchmarks](/en/user_guide/modules/benchmarks/overview)