SWE-bench Multilingual

Multilingual repository repair tasks.

SWE-bench Multilingual extends repository repair evaluation to multilingual software projects.

Runtime Status

Field	Value
Benchmark id	swebench_multilingual
Tags	Agentic Coding, Repository Repair, Multilingual
Execution type	remote environment
Typical harness	mini_swe_agent, openhands, claude_code, codex, or openclaw
Typical environment	docker, daytona, or modal
Current status	registered in the direct runtime

When to Use

Use SWE-bench Multilingual when you need to measure agentic coding behavior with the task assumptions described by this benchmark. For large or remote benchmarks, prefer benchmark recipes so images, workspaces, and provider-specific defaults come from task metadata instead of manual CLI flags.

Parameters

Common parameters for this benchmark include:

• sample_ids
• split
• k
• avgk
• resume

Shared runtime controls such as k, avgk, sample_ids, resume, and category follow the conventions in Benchmark Parameters.

Run Example

agentcompass run \
  swebench_multilingual \
  mini_swe_agent \
  your-model \
  --env <env-provider> \
  --benchmark-params '{"sample_ids":["<task-id>"]}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat

Adjust the harness and environment to the supported combination for your branch and deployment.

Outputs

Per-task details are written to results/swebench_multilingual/<model>/<run>/details/. Aggregate metrics are written to summary.md in the same run directory.

Notes

Provider recipes derive images from task metadata when available.