Results and Aggregation

RunResult should record both execution status and the Benchmark verdict. After all tasks finish, aggregate task-level results into a valid MetricResult.

Do Not Confuse Status with Score

Outcome	status	correct	score	error
Execution and evaluation succeed, and the answer is correct	COMPLETED	True	Official score	Empty
Execution and evaluation succeed, but the answer is wrong or receives a valid zero	COMPLETED	False or the official value	0.0 or the official zero	Empty
Harness or Benchmark execution fails	RUN_ERROR	False or None	Usually None	Execution error
Execution succeeds, but the verifier crashes, times out, or returns invalid output	EVAL_ERROR	False or None	Usually None	Evaluation error
Execution and evaluation both fail	ERROR	False or None	Usually None	Both errors

A failed test is not necessarily an EVAL_ERROR. For example, if the verifier contract defines exit code 1 as an ordinary test failure, the result is a valid zero. It becomes an evaluation error only when the verifier cannot complete scoring. Follow the pinned official verifier contract.

Preserve the Existing Execution Result

Use dataclasses.replace() during evaluation so you do not discard the trajectory, artifacts, Model output, or meta already written by the Harness:

from dataclasses import replace

from agentcompass.runtime import RunResult, TaskStatus


def apply_verifier_result(result: RunResult, verifier) -> RunResult:
    eval_error = (
        verifier.timed_out or verifier.returncode not in {0, 1}
    )
    if eval_error:
        detail = (verifier.stderr or verifier.stdout).strip()
        status = (
            TaskStatus.ERROR
            if result.status == TaskStatus.RUN_ERROR
            else TaskStatus.EVAL_ERROR
        )
        error = "\n".join(
            part for part in [result.error, detail] if part
        )
        return replace(
            result,
            status=status,
            correct=False,
            score=None,
            error=error or "verifier failed",
        )

    if result.status != TaskStatus.COMPLETED or result.error:
        return replace(result, correct=False, score=None)

    passed = verifier.returncode == 0
    return replace(
        result,
        correct=passed,
        score=1.0 if passed else 0.0,
        metrics={**result.metrics, "pass": float(passed)},
    )

Do not change an error status to COMPLETED merely to let aggregation continue, and do not use a non-empty error for an ordinary wrong answer. When evaluation evidence is large, save a truncated summary or file artifact instead of copying the complete verifier log into several fields.

Use the Default Binary Aggregation

BaseBenchmark.aggregate_metrics() calls aggregate_binary_metrics() by default. It fits a Benchmark whose every attempt produces a boolean correct value. It generates accuracy and applies the configured category, k, and avg@k or pass@k behavior.

An ordinary binary Benchmark does not need to override this method:

class ExampleExactMatchBenchmark(BaseBenchmark):
    # load_tasks(), prepare_task(), evaluate() ...
    pass

The default aggregator reads persisted result dictionaries, not in-memory RunResult objects. A custom implementation must not assume every field appears at the top level; results with multiple attempts may store them under result["attempts"].

Aggregate Scalar Scores

When the official primary metric is continuous, use the shared helper and state both the metric name and the missing-score behavior:

from typing import Any

from agentcompass.runtime import RunRequest
from agentcompass.runtime.metrics import MetricResult, aggregate_score_metrics


def aggregate_metrics(
    self,
    results: list[dict[str, Any]],
    req: RunRequest,
    config: Any,
) -> MetricResult:
    _ = req
    return aggregate_score_metrics(
        results,
        metric_name="mean_reward",
        score_key="score",
        missing_score_value=0.0,
        config=config,
    )

missing_score_value=0.0 counts a missing score as zero. Do not use this default unchanged if the official rules exclude infrastructure failures or use a different denominator. First select attempts and the denominator according to the official rules, then construct MetricResult and preserve total, evaluated, and error counts in counts.

Combine Multiple Primary Metrics

When you must report both accuracy and mean score, combine shared helpers:

from agentcompass.runtime.metrics import (
    aggregate_binary_metrics,
    aggregate_score_metrics,
    merge_metric_results,
)


def aggregate_metrics(self, results, req, config):
    _ = req
    accuracy = aggregate_binary_metrics(results, config=config)
    reward = aggregate_score_metrics(
        results,
        metric_name="mean_reward",
        config=config,
    )
    return merge_metric_results(accuracy, reward)

Merge two aggregations directly only when they use the same task set and denominator. If category weighting, hierarchical categories, different attempt-selection rules, or multiple official denominators are involved, implement those semantics explicitly and describe them in details.

Minimum MetricResult Requirements

Custom aggregation must return MetricResult:

Field	Requirement
metrics	At least one primary metric with a non-empty name and finite numeric value
counts.total	Total number of tasks included in this aggregation
counts.evaluated	Number of tasks that received an official score; must not exceed total
counts.error	Number of tasks with execution or evaluation errors; must not exceed total
details	Optional category, hierarchy, or submetric structure
extra	Optional reproducibility metadata such as dataset and evaluator revisions

Custom logic should read attempt data through attempt_payload() and return new dictionaries or result objects instead of mutating the persisted structure supplied by the runtime. Shared helper implementations live under runtime/metrics.

Check Before Aggregating

• Correct answers, wrong answers, valid zero scores, execution failures, and evaluation failures produce distinct expected statuses.
• correct, score, and the official primary metric have consistent meanings; do not infer a verifier crash from score == 0.
• Attempt selection, k semantics, and the failure denominator match the official implementation.
• Category aggregation neither drops unscored tasks nor counts one task in multiple mutually exclusive categories.
• MetricResult is serializable, every metric is finite, and every count satisfies its bounds.