> ## Documentation Index
> Fetch the complete documentation index at: https://agent-compass.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Python SDK

> Call AgentCompass from Python.

Use `agentcompass.run_evaluation(...)` when AgentCompass needs to be embedded in another Python workflow.

```python theme={"system"}
import os

from agentcompass import run_evaluation

result = run_evaluation(
    benchmark="swebench_verified",
    harness="mini_swe_agent",
    model=os.environ["MODEL_NAME"],
    environment="docker",
    benchmark_params={"sample_ids": ["astropy__astropy-12907"]},
    model_base_url=os.environ["MODEL_BASE_URL"],
    model_api_key=os.environ["MODEL_API_KEY"],
    model_api_protocol="openai-chat",
    model_params={"temperature": 0},
    task_concurrency=1,
    results_dir="results",
    data_dir="data",
    progress="auto",
    log_level="INFO",
)
```

The return value contains aggregate metrics, output paths, and execution metadata. Per-task details are written to the run directory so large evaluations do not need to keep every artifact in memory.

## Multiple Evaluation Requests

`run_evaluation()` executes one request. Use `launch()` or `async_launch()` for an ordered set of explicitly named
requests:

```python theme={"system"}
from agentcompass import OrchestrationSpec, RunRequestSpec, launch

result = launch(
    OrchestrationSpec(
        task_concurrency=4,
        requests=[
            RunRequestSpec(
                name="first-evaluation",
                benchmark={"id": "<benchmark>"},
                harness={"id": "<harness>"},
                environment={"id": "<environment>"},
                model={
                    "id": "<model>",
                    "base_url": "<base-url>",
                    "api_key": "<api-key>",
                    "api_protocol": "openai-chat",
                },
            ),
        ],
    )
)
```

Each request has its own outcome, output paths, logs, and progress files while sharing one orchestration-level task
limit and timeout. See [`agentcompass launch`](/en/user_guide/cli/launch) for defaults, YAML, scheduling,
reuse, and failure-isolation semantics. Earlier batch helpers are not compatibility aliases for this interface.

## Common Arguments

| Argument                             | Meaning                                                                                                             |
| ------------------------------------ | ------------------------------------------------------------------------------------------------------------------- |
| `benchmark`                          | Registered benchmark id.                                                                                            |
| `harness`                            | Registered harness id.                                                                                              |
| `model`                              | Model id passed to the harness and used in result paths.                                                            |
| `environment`                        | Environment provider id.                                                                                            |
| `benchmark_params`                   | JSON-like benchmark configuration.                                                                                  |
| `harness_params`                     | Harness-specific options.                                                                                           |
| `environment_params`                 | Provider-specific environment options.                                                                              |
| `model_*`                            | Model endpoint connection and protocol settings.                                                                    |
| `wrap_api_key`                       | Enable the session-aware credential envelope for a compatible internal AgentCompass gateway; disabled by default.   |
| `task_concurrency`                   | Maximum benchmark tasks in flight for this run.                                                                     |
| `max_retries` / `retry_pattern_list` | Additional executions for matching transient failures.                                                              |
| `provider_limits`                    | Process-wide active-attempt limits by provider, for example `{"docker": 8}`.                                        |
| `env_open_qps`                       | Environment startup rates by provider. Defaults are local `0` and remote `10`; `0` disables pacing.                 |
| `timeout_seconds`                    | Wall-clock timeout for the complete evaluation run.                                                                 |
| `reuse` / `reuse_run_id`             | Reuse normal details with matching task ids from a previous run; the caller must keep measured settings compatible. |
| `keep_environment`                   | Retain task and verifier environments for debugging.                                                                |
| `progress` / `on_progress`           | Select terminal progress and receive structured progress events.                                                    |
| `log_level` / `file_log_level`       | Control console and persistent run-log verbosity independently.                                                     |
| `auto_install_dependencies`          | Install declared optional dependencies in the AgentCompass host Python environment; disabled by default.            |
| `recipe_dirs`                        | Trusted external recipe package directories for this run.                                                           |

Pass `network_policy`, `run_network_policy`, and `verifier_network_policy` inside `environment_params`, using the same
values documented in [Network Policy](/en/user_guide/modules/environments/network). See
[agentcompass run](/en/user_guide/cli/run#control-task-execution) for choosing safe values for concurrency, timeouts,
retries, and provider limits.

Use `on_progress` to receive structured progress events from long-running evaluations.