3.3.3 (Multi-)Agent Evaluations
Original Problem in the Paper
Paper motivation: agentic systems can pursue high-level goals, directly influence the world, take goal-directed actions, and plan over long horizons; this can be economically useful but creates hard-to-predict risks. Dedicated open problems: evaluate and monitor agentic systems despite user customizability/tool integration; introduce best practices because current agent benchmarks lack holdout data and reproducibility; expand limited multi-agent evaluations because multi-agent systems add information asymmetries, destabilizing dynamics, trust/security issues, emergent collective capabilities/goals; and attribute downstream impact/liability to individual agents in multi-agent outcomes.
July 2026 Update & Trajectory
Single-agent evals have advanced quickly: real-computer, tool-use, software-engineering, ML-R&D, harmful-agent, and domain-policy-interaction benchmarks exist; Inspect-style frameworks now support tool and multi-agent evals. Multi-agent evaluation and causal attribution remain mostly open: few realistic benchmarks, weak liability/traceability tooling, and limited public evidence on deployed-agent safety practices. Verified 2026: Anthropic/Inspect pages show operational agent eval/risk governance; no verified 2026 claim that multi-agent dynamics are solved.
Deployed / Operationalized
- OSWorld: real desktop/web app environment with execution-based eval scripts for open-ended computer-use agents.
- τ-bench: dynamic tool-agent-user benchmark with domain APIs, policy rules, goal-state database scoring, and pass^k reliability metric.
- RE-Bench: ML R&D agent benchmark comparing frontier agents and human experts in equivalent environments with scoring functions, GPUs, and transcripts.
- SWE-bench Verified: validated software-agent benchmark and harness; widely used for coding-agent capability tracking.
- AgentHarm: harmful multi-step tool-agent misuse benchmark covering fraud, cybercrime, harassment, and jailbreak robustness.
- Inspect AI: built-in agent evaluations, tool calling, sandboxing, multi-agent primitives, and bridges to external agents such as Claude Code/Codex/Gemini CLI.
- AI Agent Index: public database documenting deployed agentic systems’ components, use domains, and risk-management disclosures; finds limited safety/risk-management transparency.
New Tractable Vectors
- Evaluate tool-using agents with execution-based ground truth: final DB state, tests passed, files changed, exploit success/refusal, or task completion logs.
- Measure reliability over repeated trials/pass^k rather than one-shot success, because agent behavior varies under stochastic conversations and tools.
- Benchmark real-world GUI/computer-use tasks with reproducible VM setup, custom initial states, and custom execution validators.
- Package bounded ML R&D/software/cyber tasks with human baselines, resource budgets, transcript capture, and objective scores.
- Audit deployed agent systems’ public disclosures: base model, tools, memory, autonomy level, guardrails, monitoring, incident handling.
Key Open Questions
- Multi-agent ecological validity: simulate realistic markets, organizations, social platforms, finance, cyber, or supply-chain settings without oversimplifying incentives and feedback loops.
- Causal attribution/liability: trace which agent, tool, prompt, memory item, or human intervention caused an outcome in distributed agent systems.
- Benchmark gaming/overfitting: keep agent benchmarks useful when scaffolds learn to exploit validators or public tasks leak into training.
- Tool-security evaluation: test prompt injection, malicious MCP/tools, credential theft, privilege escalation, and cross-tool data exfiltration under realistic permissions.
- Long-horizon evaluation: measure tasks spanning days/weeks, sparse feedback, ambiguous goals, hidden dependencies, and changing human preferences.