Run A Judge-Model Benchmark

Pass an LLM-as-judge configuration through benchmark params.

Use this pattern for benchmarks whose correctness is determined or assisted by a judge model. Keep the evaluated model and judge model separate when possible.

Command

agentcompass run \
  browsecomp \
  researchharness \
  your-model \
  --env <env-provider> \
  --benchmark-params '{
    "sample_ids": ["<task_id>"],
    "judge_model": {
      "id": "judge-model",
      "base_url": "https://judge-endpoint/v1",
      "api_key": "sk-...",
      "api_protocol": "openai-chat",
      "params": {"temperature": 0}
    }
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY"

Notes

Topic	Guidance
Credentials	Prefer environment variables or private config for real keys.
Judge stability	Use deterministic judge params unless the benchmark requires otherwise.
Cost control	Start with sample_ids before running the full split.
Debugging	Inspect scoring payloads in details/*.json.

Related Pages

• Deep Research
• Model API setup