Configure a Model

Configure a model id, endpoint, credential, API protocol, and endpoint-specific inference parameters.

AgentCompass does not maintain a fixed model-name registry. The model id comes from the endpoint you evaluate and is the third positional argument to agentcompass run:

agentcompass run <benchmark> <harness> "$MODEL_NAME"

The runtime stores that id together with its endpoint, credential, API protocol, and inference parameters in one ModelSpec. The selected harness decides how to consume the spec.

Model API Protocol List

Model ids vary by provider, but AgentCompass defines three protocol ids. They are also included in the output written by agentcompass list dump.

id	description
openai-chat	OpenAI-compatible Chat Completions protocol for /v1/chat/completions style endpoints.
openai-responses	OpenAI Responses API protocol for response/stateful tool-call style endpoints.
anthropic	Anthropic Messages protocol for Claude-style /v1/messages endpoints.

Protocol support also depends on the selected harness. An endpoint implementing OpenAI Chat does not make it compatible with a harness that requires Responses or Anthropic Messages behavior.

Configure the Model Spec

The General Run Parameter Reference introduces the model positional argument and --model-* flags. Together they construct these ModelSpec fields:

ModelSpec field	CLI input	Type and default	What it controls
id	Primary MODEL positional argument	Required string	Model name sent to the endpoint and model-name segment used in result paths.
base_url	--model-base-url <url>	String, default ""	API base URL. It may be empty when the selected client resolves a provider default.
api_key	--model-api-key <key>	String, default ""	Endpoint credential. Pass an environment-variable reference instead of a literal secret.
wrap_api_key	--wrap-api-key	Boolean, default false	Enables the session-aware credential envelope required by a compatible internal AgentCompass gateway.
api_protocol	--model-api-protocol <protocol>	String or ordered string list, default harness selection	Chooses how the harness communicates with the endpoint.
params	--model-params <json>	JSON object, default {}	Carries inference, client reliability, reasoning, and provider-specific request fields.

One agentcompass run command contains one ModelSpec. To compare multiple model ids, declare one named request per model with agentcompass launch; this makes endpoint and inference-setting differences explicit instead of copying one implicit comparison template.

Export model connection values once and keep credentials out of command history:

export MODEL_NAME=""
export MODEL_BASE_URL=""
export MODEL_API_KEY=""

agentcompass run <benchmark> <harness> "$MODEL_NAME" \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat

Session-Aware Gateway Keys

--wrap-api-key is an opt-in compatibility mechanism for an internal gateway that understands the AgentCompass session envelope. At model-call time, AgentCompass combines the raw credential with the absolute run-directory id and encodes that envelope before sending it as the API key. This lets the gateway associate requests with one evaluation run.

Do not enable the flag for a normal OpenAI-compatible or Anthropic-compatible endpoint: those endpoints expect the raw credential and cannot decode the envelope. The encoding is a transport format, not encryption, so continue to protect the original credential through environment variables and normal secret-management practices.

Select the API Protocol

Pass one explicit protocol for a reproducible run:

--model-api-protocol openai-responses

If the value is empty or auto, the harness chooses its default. You can also pass an ordered JSON list; the harness selects the first protocol it supports:

--model-api-protocol '["openai-responses","openai-chat"]'

An ordered list expresses acceptable alternatives, not a fallback after a request fails. Unsupported harness/protocol combinations should fail during compatibility validation before task execution.

Configure Model Parameters

The --model-params <json> object does not have one AgentCompass-wide generation schema. Its accepted fields are the intersection of three contracts:

model params
  ├─ fields consumed or translated by the selected harness
  ├─ request fields defined by the selected API protocol
  └─ fields supported by the endpoint and model deployment

agentcompass run <benchmark> <harness> "$MODEL_NAME" \
  --model-params '{
    "temperature": 0,
    "max_tokens": 4096
  }'

Model Parameter Families

Field family	Examples	How to choose it
Sampling	temperature, top_p, seed fields	Match the official benchmark setting. For debugging, use deterministic values only when the endpoint supports them.
Output budget	max_tokens, max_output_tokens	Use the field required by the protocol and keep it below the endpoint context budget while avoiding valid-output truncation.
Reasoning	reasoning_effort, thinking controls	Use the exact model/provider request shape and record it in alignment reports. Some providers require nested fields.
Client reliability	timeout, max_retries	Bound one model request and its transport retries. Keep these separate from harness timeout and AgentCompass task retries.
Provider extensions	extra_body and provider-defined fields	Pass only documented endpoint options. A field accepted by one OpenAI-compatible deployment may be rejected by another.

These names are not portable by default. For example, OpenAI Chat commonly uses max_tokens, OpenAI Responses uses Responses-specific request fields such as max_output_tokens, and Anthropic Messages has its own required request shape. The protocol pages document how AgentCompass forwards each request.

The selected harness page is also authoritative. A CLI-based harness may translate ModelSpec into its own config file instead of forwarding --model-params directly through AgentCompass’s native protocol clients.

--model-params must be valid JSON. CLI values deep-merge over matching keys in model.params from configuration files. Pass only the fields that differ from the effective endpoint and harness defaults.

Configure Judge and Analysis Models

Some benchmarks and analyzers use an additional model spec for judging or qualitative analysis. These nested specs use the same concepts—model id, base URL, API key, protocol, and params—but belong to their owning component:

• Benchmark judge models are normally under --benchmark-params, for example judge_model.
• Analyzer models are under --analysis-params, such as QualitativeAnalyzer.
• Tool-specific summarization models may belong to the selected harness.

Do not put judge credentials into the primary --model-params object unless the owning benchmark or harness explicitly documents that schema.

Diagnose Model Configuration

Symptom	Likely cause
Provider not detected	The underlying client expects a provider-qualified model id, or the selected harness uses a different protocol.
401 or authentication error	The API key is missing, expired, or belongs to another endpoint.
404 or model not found	The model id is not served by the configured base URL.
429 or rate-limit error	Task concurrency, harness parallelism, requests per minute, or token throughput exceeds endpoint capacity.
Unsupported protocol	The selected harness does not support the requested API protocol.
Invalid request field	A parameter was passed using the wrong protocol or provider-specific shape.
Repeated truncation	The output limit is too small or the total context budget is exhausted.

Reduce failures to one task and inspect the harness and endpoint error before changing multiple model parameters at once.