TauBench (τ³)

TauBench (τ³, based on the upstream tau2-bench v1.0.1) evaluates an agent’s dual-control conversational tool-use ability: the agent must converse with a simulated user while operating a backend domain environment through tools, ultimately fulfilling the user’s request. It covers the four official text domains — airline, retail, telecom, and the banking_knowledge RAG domain (github).

Unlike benchmarks that depend on an external harness, τ³ owns the complete agent/user/domain-environment workflow, so it needs no external harness — pass the none placeholder. The workflow still runs through the selected AgentCompass EnvironmentSession: use an image-capable Environment with the built-in taubench Recipe, or host_process when tau2 and its dependencies are installed locally. AgentCompass uploads the matching worker bundle and versioned dataset archive when each task starts; the image contains runtime dependencies only. The model-under-test is the agent.

The worker supports the same native model protocols as the TauBench model backend: openai-chat, openai-responses, and anthropic. Agent, user, judge, embedding, and reranker credentials are supplied to the worker command at execution time and are not written into the uploaded request JSON.

For any image-capable Environment, the automatically matched taubench Recipe selects ailabdocker/ac-taubench:v1.0.1 and defaults the working directory to /workspace. Explicit image and working-directory settings take precedence. The image provides python3, tau2 v1.0.1, model protocol dependencies, and the banking sandbox binaries, so TauBench packages are not required in the controller. For --env host_process, install the taubench extra and pinned tau2 source by following Dependencies.

Tau2 temporary and banking sandbox directories are rooted inside the per-task workspace. The worker explicitly closes tracked sandboxes on normal completion and handled failures. When keep_environment=false, AgentCompass also removes the task workspace after evaluation, runner failure, or cancellation; the selected environment provider remains responsible for terminating a command that reaches its hard timeout.

​

Parameters

Parameters fall into three groups: task and simulation, model roles, and banking_knowledge retrieval (effective only for that category; ignored by all others).

build_config is strict about unknown parameters — keys not in the tables below (e.g. a typo) raise an error rather than being silently ignored, so a misspelled param never goes unnoticed.

​

Parameter overview

Param	Type	Default	Allowed values	Description
category	string | list	all	airline, retail, telecom, telecom-workflow, banking_knowledge, all; or a list combining any of them	Domain(s) to evaluate. all = the four text domains airline/retail/telecom/banking_knowledge. telecom-workflow is the workflow-policy version of telecom. Pass a list to run several domains at once.
task_split	string	base	base, test, train; telecom additionally supports small, full	Task split; base is the official leaderboard-submission split, and the pinned v1.0.1 dataset includes its evaluation criteria. banking_knowledge has no train/test split — only the released full set: it accepts only base (or omitted). So category=all (which includes banking) with a non-base split raises an error rather than silently ignoring the split — drop banking_knowledge explicitly or use base.
max_steps	int	200	integer ≥ 1	Maximum steps in a single simulation; truncated once exceeded.
max_errors	int	10	integer ≥ 0	Aborts the simulation early once accumulated errors reach this number.
solo_mode	bool	false	true / false	Solo mode: disables the user simulator; the agent interacts only with the environment. Only telecom / telecom-workflow support it (retail/airline/banking_knowledge do not). With category=all it narrows automatically to the solo-capable domains (with a warning); explicitly listing an unsupported category is an error rather than a silent drop.
user_model	dict	null	id, base_url, api_key, api_protocol	The LLM that role-plays the customer; reuses the model-under-test if omitted.
judge_model	dict	required	id, base_url, api_key, api_protocol	The judge LLM. Required — there is no fallback to the model-under-test; the run errors out if it is unset.
retrieval_variant	string	alltools	20 options (see retrieval_variant)	banking_knowledge only: how the agent accesses the knowledge base.
retrieval_kwargs	dict	{}	see the fields of retrieval_kwargs	banking_knowledge only: overrides passed to resolve_variant.
embedding_model	dict	null	id, base_url, api_key	banking_knowledge only, required when the variant is a dense-retrieval type: the embedding endpoint.
reranker_model	dict	null	id, base_url, api_key, api_protocol	banking_knowledge only, required when the variant is *_reranker*: the LLM rerank endpoint; reuses the model-under-test if left empty.

​

Model spec conventions & recommendations

judge_model is required and never falls back to the model-under-test — a run without it errors out up front (letting a model grade its own transcripts is neither fair nor comparable). Except for embedding_model and judge_model, the other secondary models (user_model, reranker_model) reuse the model-under-test itself (same id, same gateway) when not explicitly set. embedding_model is a further exception — a chat model cannot serve as an embedding model, so it never falls back to the model-under-test. user_model, judge_model, and reranker_model are dictionaries containing id, base_url, api_key, and api_protocol, pointing at each model’s dedicated endpoint; for the reusing roles, any missing endpoint fields fall back to the model-under-test’s gateway.

Recommended configuration:

• judge_model: required — must be passed explicitly. The evaluation is adjudicated by it, so it never falls back to the model-under-test (that would make the model grade its own answers — neither fair nor comparable across models). Specify a fixed, sufficiently strong judge; AgentCompass recommends gpt-5.5.
• user_model, reranker_model: recommended to leave unset. Let them fall back to and reuse the model-under-test, so the model-under-test also plays the conversational-user and knowledge-rerank roles — evaluating its overall capability more thoroughly and comprehensively.

​

banking_knowledge retrieval configuration

The parameters below apply only to the banking_knowledge category; all other domains ignore them. They decide how the agent accesses the bank knowledge base. Shell-type retrieval variants (terminal_use, terminal_use_write, alltools, alltools-qwen) additionally require the srt sandbox system dependency. These dependencies cannot be installed via pip and must be installed separately with the steps below (offline variants such as bm25_grep don’t need them):

# 1. sandbox-runtime (srt) — needs Node.js / npm
npm install -g @anthropic-ai/sandbox-runtime@0.0.23

# 2. system tools (Linux; bwrap / socat are Linux-only, macOS needs only brew install ripgrep)
sudo apt-get install -y ripgrep bubblewrap socat

# 3. verify (macOS will only have srt / rg)
which srt rg bwrap socat

​

retrieval_variant

Selects the retrieval method (default alltools). Each option declares the derived parameters it needs — embedding_model, reranker_model, and the srt sandbox system dependency (✔ = required, · = not needed):

Variant	embedding_model	reranker_model	srt sandbox	Description
no_knowledge	·	·	·	No knowledge base (baseline)
full_kb	·	·	·	Whole KB stuffed into the prompt (upper bound)
golden_retrieval	·	·	·	Only the relevant docs (oracle)
bm25	·	·	·	Pure BM25 retrieval (offline)
bm25_grep	·	·	·	BM25 + grep tool (offline)
grep_only	·	·	·	grep tool only (offline)
bm25_reranker	·	✔	·	BM25 + LLM rerank
bm25_reranker_grep	·	✔	·	BM25 + grep + LLM rerank
openai_embeddings	✔ (openai)	·	·	Dense vector retrieval
openai_embeddings_grep	✔ (openai)	·	·	Dense + grep
openai_embeddings_reranker	✔ (openai)	✔	·	Dense + rerank
openai_embeddings_reranker_grep	✔ (openai)	✔	·	Dense + grep + rerank
qwen_embeddings	✔ (openrouter)	·	·	Dense (qwen)
qwen_embeddings_grep	✔ (openrouter)	·	·	Dense (qwen) + grep
qwen_embeddings_reranker	✔ (openrouter)	✔	·	Dense (qwen) + rerank
qwen_embeddings_reranker_grep	✔ (openrouter)	✔	·	Dense (qwen) + grep + rerank
terminal_use	·	·	✔	Read-only shell retrieval
terminal_use_write	·	·	✔	Writable shell retrieval
alltools	✔ (openai)	·	✔	BM25 + dense + shell (official default / leaderboard)
alltools-qwen	✔ (openrouter)	·	✔	Same as above, dense via qwen

• ✔ (openai) uses an OpenAI embedder, ✔ (openrouter) uses an OpenRouter/Qwen embedder, decided by the variant name.
• The srt sandbox is a system dependency, needed only by shell-type variants (terminal_use, terminal_use_write, alltools, alltools-qwen); when missing it errors clearly rather than failing silently.
• For a fully offline run, choose an offline variant such as bm25_grep.

​

retrieval_kwargs

Overrides passed to resolve_variant (equivalent to the official --retrieval-config-kwargs). Only the four fields below are accepted; an unknown field is rejected rather than silently ignored.

Field	Type	Default	Applies to	Description
top_k	int	10	bm25* / *embeddings* / alltools*	Number of docs returned by KB search (dense/bm25)
grep_top_k	int	10	*_grep / grep_only	Number of results returned by the grep tool
case_sensitive	bool	false	*_grep / grep_only	Whether grep is case-sensitive
reranker_min_score	int	5	*_reranker*	Minimum score the reranker keeps

​

embedding_model

Required only by the dense-retrieval variants marked embedding_model = ✔ above; other variants don’t need it. If such a variant is run without embedding_model, it errors before the task starts and prompts you to pass one (a chat model cannot serve as an embedding model, and it will not silently fall back to a default model).

​

reranker_model

Required only by the variants marked reranker_model = ✔ (i.e. *_reranker*); other variants don’t need it. If omitted, it falls back to and reuses the model-under-test.

​

Run examples

The three positional arguments to agentcompass run are Benchmark, Harness, and Model. These examples use taubench, none (the Benchmark owns the run loop), and $MODEL_NAME, with docker as the Environment.

Set the following environment variables in your terminal before running the examples:

• Model under test: MODEL_NAME, MODEL_BASE_URL, and MODEL_API_KEY; see Model connection details.
• Judge Model: JUDGE_MODEL_NAME, JUDGE_MODEL_BASE_URL, and JUDGE_MODEL_API_KEY. Use a fixed, independent judge configuration.
• Embedding Model for the full run: EMBEDDING_MODEL_NAME, EMBEDDING_MODEL_BASE_URL, and EMBEDDING_MODEL_API_KEY. The default alltools variant uses the OpenAI embeddings API.

See the run command for configuration ownership and CLI overrides.

The full run uses the default alltools retrieval variant, which requires an embedding model and the srt sandbox dependencies described above. The telecom smoke test and the custom bm25_reranker_grep example need no embedding model.

Smoke test (single task end-to-end)

Verify the end-to-end flow works — sample_ids picks which case to run, with the other params at their defaults.

agentcompass run \
  taubench \
  none \
  "$MODEL_NAME" \
  --env docker \
  --benchmark-params '{
    "category": "telecom",
    "sample_ids": ["taubench_telecom_bf9cd8d0"],
    "judge_model": {
      "id": "'"$JUDGE_MODEL_NAME"'",
      "base_url": "'"$JUDGE_MODEL_BASE_URL"'",
      "api_key": "'"$JUDGE_MODEL_API_KEY"'"
    }
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY"

Custom parameters

Shows how to override the various params as needed: evaluate two categories retail and banking_knowledge at once, switch banking retrieval to bm25_reranker_grep and fine-tune it with retrieval_kwargs (top_k / grep_top_k / reranker_min_score), and relax the simulation limits max_steps / max_errors.

agentcompass run \
  taubench \
  none \
  "$MODEL_NAME" \
  --env docker \
  --benchmark-params '{
    "category": ["retail", "banking_knowledge"],
    "judge_model": {
      "id": "'"$JUDGE_MODEL_NAME"'",
      "base_url": "'"$JUDGE_MODEL_BASE_URL"'",
      "api_key": "'"$JUDGE_MODEL_API_KEY"'"
    },
    "retrieval_variant": "bm25_reranker_grep",
    "retrieval_kwargs": {
      "top_k": 20,
      "grep_top_k": 15,
      "reranker_min_score": 6
    },
    "max_steps": 300,
    "max_errors": 20
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --task-concurrency 16

AgentCompass recommended config

Evaluate the default base split across all domains. Supply a fixed judge_model and the embedding_model required by the default alltools retrieval variant.

agentcompass run \
  taubench \
  none \
  "$MODEL_NAME" \
  --env docker \
  --benchmark-params '{
    "judge_model": {
      "id": "'"$JUDGE_MODEL_NAME"'",
      "base_url": "'"$JUDGE_MODEL_BASE_URL"'",
      "api_key": "'"$JUDGE_MODEL_API_KEY"'"
    },
    "embedding_model": {
      "id": "'"$EMBEDDING_MODEL_NAME"'",
      "base_url": "'"$EMBEDDING_MODEL_BASE_URL"'",
      "api_key": "'"$EMBEDDING_MODEL_API_KEY"'"
    }
  }' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --task-concurrency 16

​

Evaluation Results

For shared result conventions, see Run Directory, Aggregate Scores, and Task Files and Shared Fields.

​

Scoring Metrics

TauBench’s primary metric is binary correct, with auxiliary scalar reward preserving the raw reward. correct=true when reward is within 1e-6 of 1.0, matching the upstream tau2-bench success check.

The reward is the product of the checks specified by the task’s reward_basis: it reaches 1.0 when all are satisfied, and any zero component makes the total zero. Any partial credit remains visible in reward but does not count as a pass.

With the default configuration, aggregate results show the task pass rate and mean reward; higher is better. Domain and task filters restrict scores to the selected tasks. Repeated attempts use correct as the success criterion; see Metrics and Aggregation for strategies and scoring failure rules.

​

Task Results and Scoring Evidence

Under the attempt’s artifacts, reward_info stores the raw reward and its check breakdown, including applicable natural-language assertion verdicts, action checks, and database-state checks. simulation preserves the upstream simulation record. Comparing them helps distinguish apparent completion in the conversation from failed backend checks.