> ## Documentation Index > Fetch the complete documentation index at: https://agent-compass.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Python SDK The Python SDK and CLI use the same evaluation runtime. Choose an entry point based on the number of evaluation requests: | Request type | Synchronous entry point | Asynchronous entry point | CLI equivalent | | ---------------------------- | ----------------------- | ------------------------ | --------------------------------------------------------------------- | | Single evaluation request | `run_evaluation()` | `async_run_evaluation()` | [`agentcompass run`](/en/user_guide/using_agentcompass/cli/run) | | Multiple evaluation requests | `launch()` | `async_launch()` | [`agentcompass launch`](/en/user_guide/using_agentcompass/cli/launch) | ## Single Evaluation Request `run_evaluation()` executes one request composed of a [Benchmark](/en/user_guide/modules/benchmarks/overview), [Harness](/en/user_guide/modules/harnesses/overview), [Model](/en/user_guide/modules/models/overview), and [Environment](/en/user_guide/modules/environments/overview): ```python theme={"system"} import os from agentcompass import run_evaluation result = run_evaluation( benchmark="swebench_verified", harness="mini_swe_agent", model=os.environ["MODEL_NAME"], environment="docker", benchmark_params={"sample_ids": ["astropy__astropy-12907"]}, model_base_url=os.environ["MODEL_BASE_URL"], model_api_key=os.environ["MODEL_API_KEY"], model_api_protocol="openai-chat", model_params={"temperature": 0}, task_concurrency=1, results_dir="results", progress="auto", ) ``` All arguments are keyword-only. On success, the function returns a dictionary containing `metadata`, `metrics`, `summary`, and `paths`; per-task details are persisted in the result directory. A timeout or execution failure raises the corresponding exception. In an asynchronous application, use `await async_run_evaluation(...)`. It accepts the same arguments and returns the same value as the synchronous entry point. ## Multiple Evaluation Requests `launch()` accepts an `OrchestrationSpec`. Each `RunRequestSpec` represents one named evaluation request, while `OrchestrationDefaults` stores components and settings shared by every request: ```python theme={"system"} import os from agentcompass import ( OrchestrationDefaults, OrchestrationSpec, RunRequestSpec, launch, ) spec = OrchestrationSpec( name="terminal-evaluations", task_concurrency=4, defaults=OrchestrationDefaults( harness={"id": "terminus2", "max_turns": 300}, environment={"id": "docker"}, model={ "id": os.environ["MODEL_NAME"], "base_url": os.environ["MODEL_BASE_URL"], "api_key": os.environ["MODEL_API_KEY"], "api_protocol": "openai-chat", }, ), requests=[ RunRequestSpec( name="terminal-bench-2.1", benchmark={"id": "terminal_bench_2_1"}, ), RunRequestSpec( name="terminal-bench-2-verified", benchmark={"id": "terminal_bench_2_verified"}, ), ], ) result = launch(spec, progress="auto") ``` `task_concurrency` is the Benchmark-task concurrency limit shared by all requests. `launch()` returns an `OrchestrationResult`: `status` records the orchestration status, and `requests` stores each named request's status, result, error, and output paths. A failure in one request does not discard results from other requests. Multi-request parameters are divided among orchestration-wide settings, defaults shared by every request, and per-request overrides. Put them in `OrchestrationSpec`, `OrchestrationDefaults`, and the corresponding `RunRequestSpec`, respectively. In an asynchronous application, use `await async_launch(spec, ...)`. See [`agentcompass launch`](/en/user_guide/using_agentcompass/cli/launch#mapping-rules) for orchestration inheritance and mapping rules. ## CLI Parameter Mapping The CLI receives command-line strings, while the SDK uses `snake_case` keywords and native Python objects. The tables below first list parameters shared by the `run` and `launch` commands and both SDK entry points, followed by the input forms specific to a single request and a multi-request orchestration. ### Shared Runtime Parameters | CLI | Python SDK | Representation | | --------------------------------- | --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--config ` | `config_path` | Repeatable in the CLI; the SDK accepts one path or a sequence of paths. `launch()` accepts this argument only with an `OrchestrationSpec`. | | `--task-concurrency ` | `task_concurrency` | Limits Benchmark-task concurrency within one request for a single evaluation, or across the whole multi-request orchestration. | | `--results-dir ` | `results_dir` | Sets the result root directory. | | `--data-dir ` | `data_dir` | Sets the data and cache root directory. | | `--timeout-seconds ` | `timeout_seconds` | Limits one evaluation request or the whole orchestration. Single-request calls accept integer seconds; multi-request calls also accept fractional values. | | `--provider-limit =` | `provider_limits` | Repeatable in the CLI; the SDK accepts `dict[str, int]`. | | `--env-open-qps =` | `env_open_qps` | Repeatable in the CLI; the SDK accepts `dict[str, float]`. | | `--progress auto\|plain\|none` | `progress` | The SDK accepts the same string values. | | `--log-level ` | `log_level` | One of `DEBUG`, `INFO`, `WARNING`, `ERROR`, or `CRITICAL`. | | `--file-log-level ` | `file_log_level` | Accepts the same values as `log_level`. | | `--auto-install-dependencies` | `auto_install_dependencies` | The SDK accepts a Boolean. | | None | `log_file` | The SDK can set the log-file path. | | None | `on_progress` | The SDK can receive progress-event callbacks. | ### Direct Parameters for a Single Evaluation Request | `agentcompass run` | `run_evaluation()` / `async_run_evaluation()` | Representation | | --------------------------------- | --------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | `BENCHMARK` | `benchmark` | Benchmark ID; keyword-only in the SDK. | | `HARNESS` | `harness` | Harness ID; keyword-only in the SDK. | | `MODEL` | `model` | Model ID; keyword-only in the SDK. | | `--benchmark-params ` | `benchmark_params` | The CLI accepts a JSON object; the SDK accepts a `dict`. | | `--harness-params ` | `harness_params` | The CLI accepts a JSON object; the SDK accepts a `dict`. | | `--model-base-url ` | `model_base_url` | The value maps directly. | | `--model-api-key ` | `model_api_key` | The value maps directly. | | `--model-api-protocol ` | `model_api_protocol` | The SDK accepts a protocol name, `auto`, or a list of strings directly. | | `--model-params ` | `model_params` | The CLI accepts a JSON object; the SDK accepts a `dict`. | | `--env ` | `environment` | Environment ID. | | `--env-params ` | `environment_params` | The CLI accepts a JSON object; the SDK accepts a `dict`. | | `--max-retries ` | `max_retries` | The value maps directly. | | `--retry-pattern-list ` | `retry_pattern_list` | The CLI accepts a JSON string array; the SDK accepts `list[str]`. | | `--recipe ` | `enabled_recipes` | The CLI accepts a repeatable [Recipe](/en/user_guide/other_features/recipes) ID; the SDK accepts a string list. | | `--recipe-dir ` | `recipe_dirs` | Repeatable in the CLI; the SDK accepts a sequence of paths. | | `--run-name ` | `run_name` | The value maps directly. | | `--run-id ` | `run_id` | Sets the ID for a new result directory. | | `--reuse [run-id]` | `reuse`, `reuse_run_id` | The SDK separates the reuse switch from the ID of the run to reuse. | | `--keep-environment` | `keep_environment` | The SDK accepts a Boolean. | | `--enable-analysis` | `enable_analysis` | The SDK accepts a Boolean. | | `--analysis-params ` | `analysis_params` | The CLI accepts a JSON object; the SDK accepts a `dict`. | See the [`agentcompass run` parameter reference](/en/user_guide/using_agentcompass/cli/run#parameter-reference) for meanings and defaults. ### Orchestration Parameters for Multiple Evaluation Requests | `agentcompass launch` | `launch()` / `async_launch()` | Representation | | ----------------------------------------------- | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | | `ORCHESTRATION_PATH` | `orchestration` | The CLI reads a YAML or JSON file; the SDK accepts an `OrchestrationSpec` or resolved `Orchestration` object. | | `--cleanup-grace-seconds ` | `cleanup_grace_seconds` | Sets the cooperative cleanup grace period after cancellation. | | `--run-id ` | No same-named keyword | The CLI overrides every request's `output.run_id`; in the SDK, set each `RunRequestSpec.output`. | | `--reuse` | No same-named keyword | Corresponds to `reuse: true` in `OrchestrationDefaults.runtime`; an individual request can override the default. | | `--dry-run` | None | Only the CLI provides orchestration preflight and resolved-output display. | | `runtime.recipe_dirs` in the orchestration file | `OrchestrationSpec.runtime.recipe_dirs` | Multi-request evaluation has no corresponding CLI option or `launch()` keyword argument. | | None | `on_request_finished` | The SDK can receive a callback whenever a request finishes. | The top-level fields of `OrchestrationSpec` are `version`, `name`, `task_concurrency`, `runtime`, `defaults`, and `requests`. The only currently supported `version` is `1`. ### Request Fields in an Orchestration A single evaluation contains the same components and request settings shown below, but passes them directly to `agentcompass run` or `run_evaluation()`. In a multi-request orchestration, these values are not `launch()` keyword arguments: the CLI places them in the orchestration file, while the SDK places them in `OrchestrationDefaults` or `RunRequestSpec`. | Orchestration file | Python SDK | Fields | | ------------------------------------------------- | --------------------- | --------------------------------------------------------------------------------------------------------------------- | | `requests[].name` | `RunRequestSpec.name` | A required, unique name for each request. | | `defaults.benchmark` / `requests[].benchmark` | `benchmark` | `id` and same-level Benchmark configuration fields. | | `defaults.harness` / `requests[].harness` | `harness` | `id` and same-level Harness configuration fields. | | `defaults.model` / `requests[].model` | `model` | `id`, `base_url`, `api_key`, `api_protocol`, and `params`. | | `defaults.environment` / `requests[].environment` | `environment` | `id` and same-level Environment configuration fields. | | `defaults.execution` / `requests[].execution` | `execution` | `max_retries`, `retry_pattern_list`, `enabled_recipes`, `keep_environment`, `enable_analysis`, and `analysis_params`. | | `defaults.runtime` / `requests[].runtime` | `runtime` | `reuse` and `reuse_run_id`. | | `defaults.output` / `requests[].output` | `output` | `run_name` and `run_id`. | `task_concurrency` is orchestration-level only; it cannot appear under `defaults.execution` or `requests[].execution`. See [`agentcompass launch`](/en/user_guide/using_agentcompass/cli/launch#mapping-rules) for the complete field structure and inheritance rules. ## Related Pages * Concurrency, timeouts, retries, and provider limits: [Run Controls](/en/user_guide/using_agentcompass/run_controls) * Configuration-file loading and merging: [`agentcompass config`](/en/user_guide/using_agentcompass/cli/config)