Configure a Model

Configure a model id, endpoint, credential, API protocol, and endpoint-specific inference parameters.

AgentCompass does not maintain a fixed model-name registry. The model id comes from the endpoint you evaluate and is the third positional argument to agentcompass run:

agentcompass run <benchmark> <harness> "$MODEL_NAME"

The runtime stores that id together with its endpoint, credential, API protocol, and inference parameters in one ModelSpec. The selected harness decides how to consume the spec.

Model API Protocol List

Model ids vary by provider, but AgentCompass defines three protocol ids. They are also included in the output written by agentcompass list dump.

id	description
openai-chat	OpenAI-compatible Chat Completions protocol for /v1/chat/completions style endpoints.
openai-responses	OpenAI Responses API protocol for /v1/responses style endpoints and Responses-specific reasoning or stateful tool calls.
anthropic	Anthropic Messages protocol for Claude-style /v1/messages endpoints.

Protocol support also depends on the selected harness. An endpoint implementing OpenAI Chat does not make it compatible with a harness that requires Responses or Anthropic Messages behavior.

--model-api-protocol accepts any protocol id in the table. When the option is omitted or set to auto, the selected harness chooses the protocol; for example, codex defaults to openai-responses, while claude_code uses anthropic.

The option also accepts an ordered JSON array such as '["openai-responses","openai-chat"]'. The harness selects the first protocol it supports. The array is not a request-failure fallback and cannot contain auto.

Configure the Model Spec

The Run Parameter Reference introduces the model positional argument and --model-* flags. Together they construct these ModelSpec fields:

ModelSpec field	CLI input	Type and default	What it controls
id	Primary MODEL positional argument	Required string	Model name sent to the endpoint and model-name segment used in result paths.
base_url	--model-base-url <url>	String, default ""	API base URL. It may be empty when the selected client resolves a provider default.
api_key	--model-api-key <key>	String, default ""	Endpoint credential. Pass an environment-variable reference instead of a literal secret.
api_protocol	--model-api-protocol <protocol>	String or ordered string list, unspecified by default	Chooses how the harness communicates with the endpoint.
params	--model-params <json>	JSON object, default {}	Carries inference, client reliability, reasoning, and provider-specific request fields.

Configure Connection Details

One agentcompass run command contains one ModelSpec. To compare multiple model ids, declare one named request per model with agentcompass launch; this makes endpoint and inference-setting differences explicit instead of copying one implicit comparison template.

Export model connection values once and keep credentials out of command history:

export MODEL_NAME=""
export MODEL_BASE_URL=""
export MODEL_API_KEY=""

agentcompass run <benchmark> <harness> "$MODEL_NAME" \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat

Configure the params Field

The --model-params <json> object does not have one AgentCompass-wide generation schema. Its accepted fields are the intersection of three contracts:

model params
  ├─ fields consumed or translated by the selected harness
  ├─ request fields defined by the selected API protocol
  └─ fields supported by the endpoint and model deployment

agentcompass run <benchmark> <harness> "$MODEL_NAME" \
  --model-params '{
    "temperature": 0,
    "max_tokens": 4096
  }'

Common model parameters fall into the following categories:

Field family	Examples	How to choose it
Sampling	temperature, top_p, seed fields	Match the official benchmark setting. For debugging, use deterministic values only when the endpoint supports them.
Output budget	max_tokens, max_output_tokens	Use the field required by the protocol and keep it below the endpoint context budget while avoiding valid-output truncation.
Reasoning	reasoning_effort, thinking controls	Use the exact model/provider request shape and record it in alignment reports. Some providers require nested fields.
Client reliability	timeout, max_retries	Bound one model request and its transport retries. Keep these separate from harness timeout and AgentCompass task retries.
Provider extensions	extra_body and provider-defined fields	Pass only documented endpoint options. A field accepted by one OpenAI-compatible deployment may be rejected by another.

These names are not portable by default. For example, OpenAI Chat commonly uses max_tokens, OpenAI Responses uses Responses-specific request fields such as max_output_tokens, and Anthropic Messages has its own required request shape. The protocol pages document how AgentCompass forwards each request.

The selected harness page is also authoritative. A CLI-based harness may translate ModelSpec into its own config file instead of forwarding --model-params directly through AgentCompass’s native protocol clients.

--model-params must be valid JSON. CLI values deep-merge over matching keys in model.params from configuration files. Pass only the fields that differ from the effective endpoint and harness defaults.

Configure Judge and Analysis Models

Some benchmarks and analyzers use an additional model spec for judging or qualitative analysis. These nested specs use the same concepts—model id, base URL, API key, protocol, and params—but belong to their owning component:

• Benchmark judge models are normally under --benchmark-params, for example judge_model.
• Analyzer models are under --analysis-params, such as QualitativeAnalyzer.
• Tool-specific summarization models may belong to the selected harness.

Do not put judge credentials into the primary --model-params object unless the owning benchmark or harness explicitly documents that schema.

Diagnose Model Configuration

Symptom	Likely cause
Provider not detected	The underlying client expects a provider-qualified model id, or the selected harness uses a different protocol.
401 or authentication error	The API key is missing, expired, or belongs to another endpoint.
404 or model not found	The model id is not served by the configured base URL.
429 or rate-limit error	Task concurrency, harness parallelism, requests per minute, or token throughput exceeds endpoint capacity.
Unsupported protocol	The selected harness does not support the requested API protocol.
Invalid request field	A parameter was passed using the wrong protocol or provider-specific shape.
Repeated truncation	The output limit is too small or the total context budget is exhausted.

Reduce failures to one task and inspect the harness and endpoint error before changing multiple model parameters at once.