Yao and colleagues: tau-bench tool-agent-user reliability evaluation

What changed

Evaluated policy-following API agents in conversations with simulated users, checking final database states and repeated-trial reliability.

Scientific scope

Evaluated policy-following API agents in conversations with simulated users, checking final database states and repeated-trial reliability. The user is model-simulated and domains are constrained. Repeated-trial pass^k differs from best-of-k pass@k; final-state checks do not prove every interaction is safe or appropriate.

Evidence

The linked original sources describe the stated contribution and its evidence limits. WRS has not independently rerun the experiments or certified a proof. The user is model-simulated and domains are constrained. Repeated-trial pass^k differs from best-of-k pass@k; final-state checks do not prove every interaction is safe or appropriate.

Contributors

• Source-listed authors and technical contributors · SCIENCE: Source-listed authors or explicitly credited technical contributors for the cited versions. Family entries retain source-specific attribution; a combined list is not one original-paper byline or equal individual work. Names alone do not establish resolved identities.

• $τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains

Provisional WRS assessment, edition 2026-10-16.1. Independent expert calibration is pending. This grade does not certify correctness or assign ownership.

Full scientific record · Browse contribution grades · Request a correction