TauBench (τ³, based on the upstream tau2-bench v1.0.1) evaluates an agent’s dual-control conversational tool-use ability: the agent must converse with a simulated user while operating a backend domain environment through tools, ultimately fulfilling the user’s request. It covers the four official text domains — airline, retail, telecom, and the banking_knowledge RAG domain (github).
Unlike benchmarks that depend on an external harness, τ³ owns the complete agent/user/domain-environment workflow, so it needs no external harness — pass the none placeholder. The workflow still runs through the selected AgentCompass EnvironmentSession: use docker with the built-in TauBench recipe, or host_process when tau2 and its dependencies are installed locally. AgentCompass uploads the matching worker bundle and versioned dataset archive when each task starts; the image contains runtime dependencies only. The model-under-test is the agent.
The worker supports the same native model protocols as the TauBench model backend: openai-chat, openai-responses, and anthropic. Agent, user, judge, embedding, and reranker credentials are supplied to the worker command at execution time and are not written into the uploaded request JSON.
For --env docker, the automatically matched taubench_docker recipe selects ailabdocker/ac-taubench:v1.0.1 unless an image was explicitly configured. The image provides python3, tau2 v1.0.1, model protocol dependencies, and the banking sandbox binaries. For --env host_process, install the taubench extra and pinned tau2 source by following Dependencies. Docker runs use the task image and do not require TauBench packages in the controller.
Tau2 temporary and banking sandbox directories are rooted inside the per-task workspace. The worker explicitly closes tracked sandboxes on normal completion and handled failures. When keep_environment=false, AgentCompass also removes the task workspace after evaluation, runner failure, or cancellation; the selected environment provider remains responsible for terminating a command that reaches its hard timeout.
Parameters
Parameters fall into three groups: task & simulation, model roles, and banking_knowledge retrieval (effective only for that category; ignored by all others).
build_config is strict about unknown parameters — keys not in the tables below (e.g. a typo) raise an error rather than being silently ignored, so a misspelled param never goes unnoticed.
Parameter overview
| Param | Type | Default | Allowed values | Description |
|---|
category | string | list | all | airline, retail, telecom, telecom-workflow, banking_knowledge, all; or a list combining any of them | Domain(s) to evaluate. all = the four text domains airline/retail/telecom/banking_knowledge. telecom-workflow is the workflow-policy version of telecom. Pass a list to run several domains at once. |
task_split | string | base | base, test, train; telecom additionally supports small, full | Task split; base is the official leaderboard-submission split, and the pinned v1.0.1 dataset includes its evaluation criteria. banking_knowledge has no train/test split — only the released full set: it accepts only base (or omitted). So category=all (which includes banking) with a non-base split raises an error rather than silently ignoring the split — drop banking_knowledge explicitly or use base. |
max_steps | int | 200 | integer ≥ 1 | Maximum steps in a single simulation; truncated once exceeded. |
max_errors | int | 10 | integer ≥ 0 | Aborts the simulation early once accumulated errors reach this number. |
solo_mode | bool | false | true / false | Solo mode: disables the user simulator; the agent interacts only with the environment. Only telecom / telecom-workflow support it (retail/airline/banking_knowledge do not). With category=all it narrows automatically to the solo-capable domains (with a warning); explicitly listing an unsupported category is an error rather than a silent drop. |
user_model | dict | null | {id, base_url, api_key, api_protocol} | The LLM that role-plays the customer; reuses the model-under-test if omitted. |
judge_model | dict | required | {id, base_url, api_key, api_protocol} | The judge LLM. Required — there is no fallback to the model-under-test; the run errors out if it is unset. |
retrieval_variant | string | alltools | 20 options (see retrieval_variant) | banking_knowledge only: how the agent accesses the knowledge base. |
retrieval_kwargs | dict | {} | see the fields of retrieval_kwargs | banking_knowledge only: overrides passed to resolve_variant. |
embedding_model | dict | null | {id, base_url, api_key} | banking_knowledge only, required when the variant is a dense-retrieval type: the embedding endpoint. |
reranker_model | dict | null | {id, base_url, api_key, api_protocol} | banking_knowledge only, required when the variant is *_reranker*: the LLM rerank endpoint; reuses the model-under-test if left empty. |
Model spec conventions & recommendations
judge_model is required and never falls back to the model-under-test — a run without it errors out up front (letting a model grade its own transcripts is neither fair nor comparable). Except for embedding_model and judge_model, the other secondary models (user_model, reranker_model) reuse the model-under-test itself (same id, same gateway) when not explicitly set. embedding_model is a further exception — a chat model cannot serve as an embedding model, so it never falls back to the model-under-test. user_model, judge_model, and reranker_model are each passed as a dict: {"id","base_url","api_key","api_protocol"} pointing at that model’s dedicated endpoint; for the reusing roles, any missing endpoint fields fall back to the model-under-test’s gateway.
Recommended configuration:
judge_model: required — must be passed explicitly. The evaluation is adjudicated by it, so it never falls back to the model-under-test (that would make the model grade its own answers — neither fair nor comparable across models). Specify a fixed, sufficiently strong judge; AgentCompass recommends gpt-5.5.user_model, reranker_model: recommended to leave unset. Let them fall back to and reuse the model-under-test, so the model-under-test also plays the conversational-user and knowledge-rerank roles — evaluating its overall capability more thoroughly and comprehensively.
banking_knowledge retrieval configuration
The parameters below apply only to the banking_knowledge category; all other domains ignore them. They decide how the agent accesses the bank knowledge base. shell-type retrieval variants (terminal_use, terminal_use_write, alltools, alltools-qwen) additionally require the srt sandbox system dependency. These dependencies cannot be installed via pip and must be installed separately with the steps below (offline variants such as bm25_grep don’t need them):
retrieval_variant
Selects the retrieval method (default alltools). Each option declares the derived parameters it needs — embedding_model, reranker_model, and the srt sandbox system dependency (✔ = required, · = not needed):
| Variant | embedding_model | reranker_model | srt sandbox | Description |
|---|
no_knowledge | · | · | · | No knowledge base (baseline) |
full_kb | · | · | · | Whole KB stuffed into the prompt (upper bound) |
golden_retrieval | · | · | · | Only the relevant docs (oracle) |
bm25 | · | · | · | Pure BM25 retrieval (offline) |
bm25_grep | · | · | · | BM25 + grep tool (offline) |
grep_only | · | · | · | grep tool only (offline) |
bm25_reranker | · | ✔ | · | BM25 + LLM rerank |
bm25_reranker_grep | · | ✔ | · | BM25 + grep + LLM rerank |
openai_embeddings | ✔ (openai) | · | · | Dense vector retrieval |
openai_embeddings_grep | ✔ (openai) | · | · | Dense + grep |
openai_embeddings_reranker | ✔ (openai) | ✔ | · | Dense + rerank |
openai_embeddings_reranker_grep | ✔ (openai) | ✔ | · | Dense + grep + rerank |
qwen_embeddings | ✔ (openrouter) | · | · | Dense (qwen) |
qwen_embeddings_grep | ✔ (openrouter) | · | · | Dense (qwen) + grep |
qwen_embeddings_reranker | ✔ (openrouter) | ✔ | · | Dense (qwen) + rerank |
qwen_embeddings_reranker_grep | ✔ (openrouter) | ✔ | · | Dense (qwen) + grep + rerank |
terminal_use | · | · | ✔ | Read-only shell retrieval |
terminal_use_write | · | · | ✔ | Writable shell retrieval |
alltools | ✔ (openai) | · | ✔ | BM25 + dense + shell (official default / leaderboard) |
alltools-qwen | ✔ (openrouter) | · | ✔ | Same as above, dense via qwen |
✔ (openai) uses an OpenAI embedder, ✔ (openrouter) uses an OpenRouter/Qwen embedder, decided by the variant name.- The srt sandbox is a system dependency, needed only by shell-type variants (
terminal_use, terminal_use_write, alltools, alltools-qwen); when missing it errors clearly rather than failing silently.
- For a fully offline run, choose an offline variant such as
bm25_grep.
retrieval_kwargs
Overrides passed to resolve_variant (equivalent to the official --retrieval-config-kwargs). Only the four fields below are accepted; an unknown field is rejected rather than silently ignored.
| Field | Type | Default | Applies to | Description |
|---|
top_k | int | 10 | bm25* / *embeddings* / alltools* | Number of docs returned by KB search (dense/bm25) |
grep_top_k | int | 10 | *_grep / grep_only | Number of results returned by the grep tool |
case_sensitive | bool | false | *_grep / grep_only | Whether grep is case-sensitive |
reranker_min_score | int | 5 | *_reranker* | Minimum score the reranker keeps |
embedding_model
Required only by the dense-retrieval variants marked embedding_model = ✔ above; other variants don’t need it. If such a variant is run without embedding_model, it errors before the task starts and prompts you to pass one (a chat model cannot serve as an embedding model, and it will not silently fall back to a default model).
reranker_model
Required only by the variants marked reranker_model = ✔ (i.e. *_reranker*); other variants don’t need it. If omitted, it falls back to and reuses the model-under-test.
Run examples
A τ³ run command looks like agentcompass run taubench none <model>; the three positional arguments are, in order:
taubench — the benchmark id;none — the harness placeholder (τ³ carries its own run loop and needs no external harness);<model> — the model-under-test, i.e. the agent; its access credentials are passed via --model-base-url / --model-api-key.
All other configuration (domains, split, the various models, etc. — see Parameter overview above) is passed as a single JSON blob through --benchmark-params; it can also go into a YAML file specified with --config, where CLI values take precedence for keys present in both.
Verify the end-to-end flow works — sample_ids picks which case to run, with the other params at their defaults. Shows how to override the various params as needed: evaluate two categories retail and banking_knowledge at once, switch banking retrieval to bm25_reranker_grep and fine-tune it with retrieval_kwargs (top_k / grep_top_k / reranker_min_score), and relax the simulation limits max_steps / max_errors. AgentCompass’s recommended config for a full evaluation. Besides the model-under-test, only two params need to be provided explicitly: the judge judge_model, and the embedding_model required by the default alltools retrieval variant. alltools depends on the srt sandbox — install it per the section above before running. Output
A run produces two kinds of results, both under results/taubench/<model>/<run>/: aggregate metrics (summary.md, overall performance) and per-task details (details/, per-task reward and breakdown).
Aggregate metrics (summary.md)
summary.md summarizes the overall performance of the run, in two parts — a run overview and the metrics.
Run overview
Metrics
There is a single headline metric, accuracy: the task pass rate, equivalent to pass^1. A task counts as passed (scored 1, otherwise 0) when its reward is within 1e-6 of the full 1.0 (matching upstream tau2-bench’s is_successful() — it earned the full reward and is considered complete); accuracy is the mean over all tasks. The reward is the product of the checks in the task’s reward_basis — database/environment-state checks, action checks, the NL-assertion judge, etc. — so it reaches the full 1.0 only when all of them pass, and drops sharply (usually to 0) if any one fails.
Per-task details (details/)
The raw per-task reward and its breakdown (reward_info, including each NL-assertion verdict and the action / database check decomposition) are saved in the corresponding task’s JSON under details/. 