Metrics and Aggregation

AgentCompass uses one metric pipeline for binary, scalar, and mixed Benchmarks. The Benchmark declares what each attempt measures; the run configuration selects how repeated attempts are executed and reduced; the result report keeps a separate value and coverage count for every metric series.

attempt.metrics → Metric Contract → Attempt Strategy → Metric Reducer → task value → Benchmark aggregation → MetricReport

Configure Repeated Attempts

Repeated attempts are execution controls, not Benchmark parameters. Set them with CLI options or under execution.attempts in a configuration file:

agentcompass run <benchmark> <harness> "$MODEL_NAME" \
  --k 3 \
  --attempt-strategy avg

execution:
  attempts:
    k: 3
    strategy: avg

Field	Default	Meaning
k	1	Maximum number of independent evaluation attempts for each task.
strategy	avg	Built-in avg collects complete repeated observations; built-in pass stops after the Benchmark’s binary primary metric first succeeds.

Understand Metric Contracts

Each Benchmark declares a Metric Contract. It assigns every key in attempts.<N>.metrics one of two kinds:

Kind	Attempt value	Supported repeated-attempt reducers
binary_success	JSON true or false	avg@k and pass@k
scalar	Finite JSON number	avg@k only

binary_success means a yes/no success condition defined by the Benchmark, such as whether a verifier passed. It is not simply any numeric field whose current values happen to be 0 and 1. A scalar represents an amount or degree, including partial credit.

Every contract declares exactly one primary metric. Binary primaries use the canonical ID correct; scalar primaries use score; neither canonical ID can be auxiliary. Benchmark-specific names such as reward and f2p are auxiliary metrics. A mixed Benchmark can declare both binary and scalar observations, but its fixed primary metric always controls the execution strategy.

The contract is validated before tasks run. Selecting strategy: pass for a Benchmark with a scalar primary raises an error, even at k=1, because a numeric score does not define success. Only Benchmarks whose primary metric is correct can use pass.

Know Which Series Are Produced

Plan	Execution	Exact output series
k=1	One attempt	Native value for every declared metric.
k>1, strategy=avg	Complete all k attempts.	avg@k for every compatible binary or scalar metric, plus pass@k for every binary metric.
k>1, strategy=pass	Stop after the binary primary metric first succeeds, or after attempt k.	Only the primary metric’s pass@k.

There is no attempt-1 or first headline when k>1. For an avg run with a binary primary, both its avg@k and pass@k are headline results. Other contract metrics are retained as auxiliary series in the full report.

The reducer definitions are:

• native@1: the single valid observation.
• avg@k: the arithmetic mean of exactly k valid observations. For a binary metric, true is 1 and false is 0.
• pass@k: 1 as soon as any valid binary observation is true; 0 only after all k observations are valid and false.

Treat Missing Attempts Explicitly

A missing, failed, skipped, or metric-less attempt is not silently converted to 0 or false.

• avg@k is unavailable unless all k observations are valid.
• pass@k=1 is exact once a success exists, even if later attempts were not needed.
• pass@k=0 is exact only when all k valid observations are false.

Each series therefore has independent total, evaluated, error, and unavailable task counts. Two series from the same run can have different denominators because an attempt can contain one metric but not another. Read those counts with the value in metrics.json.

For a series without an exact value, error means at least one required attempt is missing or errored; unavailable means all planned attempts are present and non-error, but too few contain a valid observation for that metric.

Execution, Retry, and Reuse

execution.task_concurrency is the single per-run concurrency limit. It counts physical attempt executions, including retries, rather than treating all k attempts for one task as one slot. Inline analysis enabled by agentcompass run shares this limit; the standalone agentcompass analysis command schedules work with its own task concurrency.

With strategy: avg, attempts from the same task may run concurrently only when both the Benchmark and Harness declare that their per-attempt state is isolated. Otherwise AgentCompass runs those attempts serially. The user-facing concurrency setting does not change.

A retry belongs to one logical attempt. If attempt 3 is retried, completed attempts 1 and 2 are not executed again. AgentCompass checkpoints terminal attempts separately, so an interrupted run or a compatible --reuse run can continue from the missing (task, attempt) pairs. The saved task detail preserves retry_count and per-attempt retry_counts; retry executions do not add metric observations.

Aggregate Tasks and Categories

The repeated-attempt reducer and the Benchmark aggregator solve different problems. A reducer combines the k observations of one task; after that, the runtime calls Benchmark.aggregate_metrics() to apply the Benchmark’s official cross-task definition.

The default Benchmark implementation applies the following shared policies separately to every series:

Setting	Run-level calculation
micro_weighted	Mean of valid task values; every task has equal weight.
category_mean	Mean of valid category means; every category has equal weight.
Non-empty category_hierarchy	Uses the explicit tree and takes precedence over aggregation_mode.

Every category and hierarchy node stores the same four series-specific counts as the overall value. Missing children have value: null and do not borrow another series’ count. For hierarchy nodes, unweighted, explicit weighted, and weighted_by_count aggregation renormalize over children with valid values.

A Benchmark overrides the default hook when its official result is not a mean of task values. For example, SciCode computes subproblem accuracy as sum(correct subproblems) / sum(total subproblems), DeepResearch FACT weights citation accuracy by checked citations, GDPVal divides the corpus total score by the corpus maximum, and Frontier Engineering derives medal and rank results from its reference tables. These formulas run after the selected task-level reducer, so they remain compatible with native@1 and complete avg@k observations.

In metrics.json, each series records its actual aggregation formula. Shared series use micro_weighted, category_mean, or category_hierarchy; custom series can use ratio_of_sums, sum, or benchmark. Formula inputs and totals are stored in series[].extra, while larger Benchmark-specific diagnostics such as rank comparisons are stored in the report-level extra. Category and hierarchy breakdowns also expose aggregation_weight when a formula uses a denominator other than the number of evaluated tasks.

Read the Outputs

Successful aggregation writes two complementary files:

File	Purpose
summary.md	Shared run counts and Metrics structure for every k; the traditional metric table at k=1, with attempt plan, series roles, formulas, and independent counts added in the metric area at k>1.
metrics.json	Canonical metric report containing all headline and auxiliary series, counts, categories, and hierarchy nodes.

The CLI also prints the headline series. Use metrics.json for tooling and audits; do not parse Markdown as the data source. See Task Results for attempt observations and Summary and Analysis Results for the complete output layout.