> ## Documentation Index
> Fetch the complete documentation index at: https://agent-compass.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Results and Aggregation

`RunResult` should record both execution status and the Benchmark verdict. After all tasks finish, aggregate task-level results into a valid `MetricResult`.

## Do Not Confuse Status with Score

| Outcome                                                                            | `status`     | `correct`                     | `score`                    | `error`          |
| ---------------------------------------------------------------------------------- | ------------ | ----------------------------- | -------------------------- | ---------------- |
| Execution and evaluation succeed, and the answer is correct                        | `COMPLETED`  | `True`                        | Official score             | Empty            |
| Execution and evaluation succeed, but the answer is wrong or receives a valid zero | `COMPLETED`  | `False` or the official value | `0.0` or the official zero | Empty            |
| Harness or Benchmark execution fails                                               | `RUN_ERROR`  | `False` or `None`             | Usually `None`             | Execution error  |
| Execution succeeds, but the verifier crashes, times out, or returns invalid output | `EVAL_ERROR` | `False` or `None`             | Usually `None`             | Evaluation error |
| Execution and evaluation both fail                                                 | `ERROR`      | `False` or `None`             | Usually `None`             | Both errors      |

A failed test is not necessarily an `EVAL_ERROR`. For example, if the verifier contract defines exit code `1` as an ordinary test failure, the result is a valid zero. It becomes an evaluation error only when the verifier cannot complete scoring. Follow the pinned official verifier contract.

## Preserve the Existing Execution Result

Use `dataclasses.replace()` during evaluation so you do not discard the trajectory, artifacts, Model output, or `meta` already written by the Harness:

```python theme={"system"}
from dataclasses import replace

from agentcompass.runtime import RunResult, TaskStatus


def apply_verifier_result(result: RunResult, verifier) -> RunResult:
    eval_error = (
        verifier.timed_out or verifier.returncode not in {0, 1}
    )
    if eval_error:
        detail = (verifier.stderr or verifier.stdout).strip()
        status = (
            TaskStatus.ERROR
            if result.status == TaskStatus.RUN_ERROR
            else TaskStatus.EVAL_ERROR
        )
        error = "\n".join(
            part for part in [result.error, detail] if part
        )
        return replace(
            result,
            status=status,
            correct=False,
            score=None,
            error=error or "verifier failed",
        )

    if result.status != TaskStatus.COMPLETED or result.error:
        return replace(result, correct=False, score=None)

    passed = verifier.returncode == 0
    return replace(
        result,
        correct=passed,
        score=1.0 if passed else 0.0,
        metrics={**result.metrics, "pass": float(passed)},
    )
```

Do not change an error status to `COMPLETED` merely to let aggregation continue, and do not use a non-empty `error` for an ordinary wrong answer. When evaluation evidence is large, save a truncated summary or file artifact instead of copying the complete verifier log into several fields.

## Use the Default Binary Aggregation

`BaseBenchmark.aggregate_metrics()` calls `aggregate_binary_metrics()` by default. It fits a Benchmark whose every attempt produces a boolean `correct` value. It generates `accuracy` and applies the configured category, `k`, and `avg@k` or `pass@k` behavior.

An ordinary binary Benchmark does not need to override this method:

```python theme={"system"}
class ExampleExactMatchBenchmark(BaseBenchmark):
    # load_tasks(), prepare_task(), evaluate() ...
    pass
```

The default aggregator reads persisted result dictionaries, not in-memory `RunResult` objects. A custom implementation must not assume every field appears at the top level; results with multiple attempts may store them under `result["attempts"]`.

## Aggregate Scalar Scores

When the official primary metric is continuous, use the shared helper and state both the metric name and the missing-score behavior:

```python theme={"system"}
from typing import Any

from agentcompass.runtime import RunRequest
from agentcompass.runtime.metrics import MetricResult, aggregate_score_metrics


def aggregate_metrics(
    self,
    results: list[dict[str, Any]],
    req: RunRequest,
    config: Any,
) -> MetricResult:
    _ = req
    return aggregate_score_metrics(
        results,
        metric_name="mean_reward",
        score_key="score",
        missing_score_value=0.0,
        config=config,
    )
```

`missing_score_value=0.0` counts a missing score as zero. Do not use this default unchanged if the official rules exclude infrastructure failures or use a different denominator. First select attempts and the denominator according to the official rules, then construct `MetricResult` and preserve total, evaluated, and error counts in `counts`.

## Combine Multiple Primary Metrics

When you must report both accuracy and mean score, combine shared helpers:

```python theme={"system"}
from agentcompass.runtime.metrics import (
    aggregate_binary_metrics,
    aggregate_score_metrics,
    merge_metric_results,
)


def aggregate_metrics(self, results, req, config):
    _ = req
    accuracy = aggregate_binary_metrics(results, config=config)
    reward = aggregate_score_metrics(
        results,
        metric_name="mean_reward",
        config=config,
    )
    return merge_metric_results(accuracy, reward)
```

Merge two aggregations directly only when they use the same task set and denominator. If category weighting, hierarchical categories, different attempt-selection rules, or multiple official denominators are involved, implement those semantics explicitly and describe them in `details`.

## Minimum `MetricResult` Requirements

Custom aggregation must return `MetricResult`:

| Field              | Requirement                                                                  |
| ------------------ | ---------------------------------------------------------------------------- |
| `metrics`          | At least one primary metric with a non-empty name and finite numeric value   |
| `counts.total`     | Total number of tasks included in this aggregation                           |
| `counts.evaluated` | Number of tasks that received an official score; must not exceed `total`     |
| `counts.error`     | Number of tasks with execution or evaluation errors; must not exceed `total` |
| `details`          | Optional category, hierarchy, or submetric structure                         |
| `extra`            | Optional reproducibility metadata such as dataset and evaluator revisions    |

Custom logic should read attempt data through `attempt_payload()` and return new dictionaries or result objects instead of mutating the persisted structure supplied by the runtime. Shared helper implementations live under [`runtime/metrics`](https://github.com/open-compass/AgentCompass/tree/main/src/agentcompass/runtime/metrics).

## Check Before Aggregating

* Correct answers, wrong answers, valid zero scores, execution failures, and evaluation failures produce distinct expected statuses.
* `correct`, `score`, and the official primary metric have consistent meanings; do not infer a verifier crash from `score == 0`.
* Attempt selection, `k` semantics, and the failure denominator match the official implementation.
* Category aggregation neither drops unscored tasks nor counts one task in multiple mutually exclusive categories.
* `MetricResult` is serializable, every metric is finite, and every count satisfies its bounds.