> ## Documentation Index
> Fetch the complete documentation index at: https://agent-compass.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Metrics and Aggregation

AgentCompass uses one metric pipeline for binary, scalar, and mixed Benchmarks. The Benchmark declares what each attempt measures; the run configuration selects how repeated attempts are executed and reduced; the result report keeps a separate value and coverage count for every metric series.

```text theme={"system"}
attempt.metrics → Metric Contract → Attempt Strategy → Metric Reducer → task value → Benchmark aggregation → MetricReport
```

## Configure Repeated Attempts

Repeated attempts are execution controls, not Benchmark parameters. Set them with CLI options or under `execution.attempts` in a configuration file:

```bash theme={"system"}
agentcompass run <benchmark> <harness> "$MODEL_NAME" \
  --k 3 \
  --attempt-strategy avg
```

```yaml theme={"system"}
execution:
  attempts:
    k: 3
    strategy: avg
```

| Field      | Default | Meaning                                                                                                                                   |
| ---------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `k`        | `1`     | Maximum number of independent evaluation attempts for each task.                                                                          |
| `strategy` | `avg`   | Built-in `avg` collects complete repeated observations; built-in `pass` stops after the Benchmark's binary primary metric first succeeds. |

## Understand Metric Contracts

Each Benchmark declares a Metric Contract. It assigns every key in `attempts.<N>.metrics` one of two kinds:

| Kind             | Attempt value          | Supported repeated-attempt reducers |
| ---------------- | ---------------------- | ----------------------------------- |
| `binary_success` | JSON `true` or `false` | `avg@k` and `pass@k`                |
| `scalar`         | Finite JSON number     | `avg@k` only                        |

`binary_success` means a yes/no success condition defined by the Benchmark, such as whether a verifier passed. It is not simply any numeric field whose current values happen to be 0 and 1. A scalar represents an amount or degree, including partial credit.

Every contract declares exactly one primary metric. Binary primaries use the canonical ID `correct`; scalar primaries use `score`; neither canonical ID can be auxiliary. Benchmark-specific names such as `reward` and `f2p` are auxiliary metrics. A mixed Benchmark can declare both binary and scalar observations, but its fixed primary metric always controls the execution strategy.

The contract is validated before tasks run. Selecting `strategy: pass` for a Benchmark with a scalar primary raises an error, even at `k=1`, because a numeric score does not define success. Only Benchmarks whose primary metric is `correct` can use `pass`.

## Know Which Series Are Produced

| Plan                   | Execution                                                                  | Exact output series                                                                          |
| ---------------------- | -------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
| `k=1`                  | One attempt                                                                | Native value for every declared metric.                                                      |
| `k>1`, `strategy=avg`  | Complete all `k` attempts.                                                 | `avg@k` for every compatible binary or scalar metric, plus `pass@k` for every binary metric. |
| `k>1`, `strategy=pass` | Stop after the binary primary metric first succeeds, or after attempt `k`. | Only the primary metric's `pass@k`.                                                          |

There is no attempt-1 or `first` headline when `k>1`. For an `avg` run with a binary primary, both its `avg@k` and `pass@k` are headline results. Other contract metrics are retained as auxiliary series in the full report.

The reducer definitions are:

* `native@1`: the single valid observation.
* `avg@k`: the arithmetic mean of exactly `k` valid observations. For a binary metric, `true` is `1` and `false` is `0`.
* `pass@k`: `1` as soon as any valid binary observation is `true`; `0` only after all `k` observations are valid and `false`.

## Treat Missing Attempts Explicitly

A missing, failed, skipped, or metric-less attempt is not silently converted to `0` or `false`.

* `avg@k` is unavailable unless all `k` observations are valid.
* `pass@k=1` is exact once a success exists, even if later attempts were not needed.
* `pass@k=0` is exact only when all `k` valid observations are false.

Each series therefore has independent `total`, `evaluated`, `error`, and `unavailable` task counts. Two series from the same run can have different denominators because an attempt can contain one metric but not another. Read those counts with the value in [`metrics.json`](/en/user_guide/other_features/results/summary_analysis#metricsjson).

For a series without an exact value, `error` means at least one required attempt is missing or errored; `unavailable` means all planned attempts are present and non-error, but too few contain a valid observation for that metric.

## Execution, Retry, and Reuse

`execution.task_concurrency` is the single per-run concurrency limit. It counts physical attempt executions, including retries, rather than treating all `k` attempts for one task as one slot. Inline analysis enabled by `agentcompass run` shares this limit; the standalone `agentcompass analysis` command schedules work with its own task concurrency.

With `strategy: avg`, attempts from the same task may run concurrently only when both the Benchmark and Harness declare that their per-attempt state is isolated. Otherwise AgentCompass runs those attempts serially. The user-facing concurrency setting does not change.

A retry belongs to one logical attempt. If attempt 3 is retried, completed attempts 1 and 2 are not executed again. AgentCompass checkpoints terminal attempts separately, so an interrupted run or a compatible `--reuse` run can continue from the missing `(task, attempt)` pairs. The saved task detail preserves `retry_count` and per-attempt `retry_counts`; retry executions do not add metric observations.

## Aggregate Tasks and Categories

The repeated-attempt reducer and the Benchmark aggregator solve different problems. A reducer combines the `k` observations of one task; after that, the runtime calls `Benchmark.aggregate_metrics()` to apply the Benchmark's official cross-task definition.

The default Benchmark implementation applies the following shared policies separately to every series:

| Setting                        | Run-level calculation                                                |
| ------------------------------ | -------------------------------------------------------------------- |
| `micro_weighted`               | Mean of valid task values; every task has equal weight.              |
| `category_mean`                | Mean of valid category means; every category has equal weight.       |
| Non-empty `category_hierarchy` | Uses the explicit tree and takes precedence over `aggregation_mode`. |

Every category and hierarchy node stores the same four series-specific counts as the overall value. Missing children have `value: null` and do not borrow another series' count. For hierarchy nodes, `unweighted`, explicit `weighted`, and `weighted_by_count` aggregation renormalize over children with valid values.

A Benchmark overrides the default hook when its official result is not a mean of task values. For example, SciCode computes subproblem accuracy as `sum(correct subproblems) / sum(total subproblems)`, DeepResearch FACT weights citation accuracy by checked citations, GDPVal divides the corpus total score by the corpus maximum, and Frontier Engineering derives medal and rank results from its reference tables. These formulas run after the selected task-level reducer, so they remain compatible with `native@1` and complete `avg@k` observations.

In `metrics.json`, each series records its actual `aggregation` formula. Shared series use `micro_weighted`, `category_mean`, or `category_hierarchy`; custom series can use `ratio_of_sums`, `sum`, or `benchmark`. Formula inputs and totals are stored in `series[].extra`, while larger Benchmark-specific diagnostics such as rank comparisons are stored in the report-level `extra`. Category and hierarchy breakdowns also expose `aggregation_weight` when a formula uses a denominator other than the number of evaluated tasks.

## Read the Outputs

Successful aggregation writes two complementary files:

| File           | Purpose                                                                                                                                                                                              |
| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `summary.md`   | Shared run counts and `Metrics` structure for every `k`; the traditional metric table at `k=1`, with attempt plan, series roles, formulas, and independent counts added in the metric area at `k>1`. |
| `metrics.json` | Canonical metric report containing all headline and auxiliary series, counts, categories, and hierarchy nodes.                                                                                       |

The CLI also prints the headline series. Use `metrics.json` for tooling and audits; do not parse Markdown as the data source. See [Task Results](/en/user_guide/other_features/results/task_results) for attempt observations and [Summary and Analysis Results](/en/user_guide/other_features/results/summary_analysis) for the complete output layout.