3.3.3 (Multi-)Agent Evaluations
Original Problem in the Paper
Paper motivation: agentic systems can pursue high-level goals, directly influence the world, take goal-directed actions, and plan over long horizons; this can be economically useful but creates hard-to-predict risks. Dedicated open problems: evaluate and monitor agentic systems despite user customizability and tool integration; introduce best practices because current agent benchmarks lack holdout data and reproducibility; expand limited multi-agent evaluations because multi-agent systems add information asymmetries, destabilizing dynamics, trust/security issues, emergent collective capabilities/goals; and attribute downstream responsibility or liability to individual agents in multi-agent outcomes.
July 2026 Update & Trajectory
Since 2024, single-agent agent evaluation has gained benchmarks for real-computer use, tool-agent-user interactions, software engineering, ML R&D, and harmful tool-agent misuse; Inspect documentation also shows support for tool calling, sandboxed agent evaluations, multi-agent primitives, and external agent CLIs. The source paper frames multi-agent evaluation and responsibility attribution as open problems, and the evidence reviewed here does not show a mature, generally accepted benchmark or causal-attribution stack for distributed agent outcomes. The AI Agent Index also finds limited safety/risk-management transparency among the deployed agentic systems it documents. Verified tooling/governance evidence is narrower: Inspect documents operational support for tool-using and multi-agent evaluations; Anthropic’s 2026 RSP page documents frontier risk-governance processes, including risk reports and capability-threshold policy updates, but neither source shows that multi-agent dynamics or causal attribution are solved.
Deployed / Operationalized
- OSWorld: real desktop/web app environment with setup configurations and execution-based evaluation scripts for open-ended computer-use agents.
- τ-bench: dynamic tool-agent-user benchmark with domain APIs, policy rules, final goal-state database scoring, and pass^k reliability metric.
- RE-Bench: ML R&D agent benchmark comparing frontier agents and human experts in equivalent environments with scoring functions, resource budgets, GPUs in some tasks, and released transcripts.
- SWE-bench Verified: a human-validated 500-sample subset for software-engineering agents, released with/corresponding to a more reliable Docker-based evaluation harness; SWE-bench is described by OpenAI as one of the most popular software-engineering evaluation suites.
- AgentHarm: harmful multi-step tool-agent misuse benchmark covering fraud, cybercrime, harassment, and jailbreak robustness.
- Inspect AI: evaluation framework with agent evaluations, tool calling including MCP/custom tools, sandboxing for untrusted model code, multi-agent primitives, and bridges to external agents such as Claude Code, Codex CLI, and Gemini CLI.
- AI Agent Index: public database documenting currently deployed agentic systems’ components, application domains, and risk-management practices from public information and developer correspondence; it finds comparatively limited information about safety and risk-management practices.
New Tractable Vectors
- Evaluate tool-using agents with execution-based ground truth: final database state, tests passed, files changed, exploit success/refusal, or task completion logs.
- Measure reliability over repeated trials/pass^k rather than one-shot success, because agent behavior can vary across stochastic conversations and tool interactions.
- Benchmark real-world GUI/computer-use tasks with reproducible VM setup, custom initial states, and custom execution validators.
- Package bounded ML R&D/software/cyber tasks with human baselines, resource budgets, transcript capture, and objective scores.
- Audit deployed agent systems’ public disclosures about base model, tools, memory, autonomy level, guardrails, monitoring, and incident handling.
Key Open Questions
- Multi-agent ecological validity: simulate realistic markets, organizations, social platforms, finance, cyber, or supply-chain settings without oversimplifying incentives and feedback loops.
- Causal attribution/liability: trace which agent, tool, prompt, memory item, or human intervention contributed to an outcome in distributed agent systems.
- Benchmark gaming/overfitting: maintain holdout or rotating tasks and robust validators as public agent benchmarks become exposed to scaffolds, repeated tuning, or training-data contamination.
- Tool-security evaluation: test malicious MCP/tool behavior, prompt/tool injection, malicious code execution, remote access, credential theft, and related cross-tool misuse under realistic permissions.
- Long-horizon evaluation: measure tasks spanning days/weeks, sparse feedback, ambiguous goals, hidden dependencies, and changing human preferences.