4.3.1 Facilitation of Third-Party Access to Models
Original Problem in the Paper
Paper motivation: external research/evaluation requires model access, yet many systems are not openly released and current APIs lack depth/flexibility; black-box-only evaluations can “produce misleading results” and offer “limited insights.” Dedicated open problems: map what auditing methods are possible across black/grey/white-box access; assess how access forms change misuse/model-theft risks; reconcile research/audit access with commercial/safety concerns; enable near-white-box auditing through PETs/TEEs/MPC; and maintain stable access to deprecated hosted models for reproducibility.
July 2026 Update & Trajectory
Third-party model access is institutionally more common than in 2024: UK AISI, company safety frameworks, METR-style external evals, and Seoul commitments normalize pre/post-deployment external testing and confidential sharing with trusted actors. Technical secure-access mechanisms exist but are not broadly standardized, and access remains voluntary/contractual for frontier labs except limited regulatory contexts. Grey/white-box risk tradeoffs, model-theft exposure from logits/fine-tuning/weights, evaluator privacy, and version-stable access remain unresolved. Verified 2026: Anthropic RSP v3.3 has external review hooks; OpenAI 2025 PF publishes ongoing eval/report commitments; no verified 2026 universal mandated near-white-box access regime found.
Deployed / Operationalized
- UK AISI conducts government-led advanced-model evaluations before and after deployment; it uses automated assessments, red-teaming, human uplift, agent evaluations, and safeguard evaluations, with confidential methodology.
- Seoul Frontier AI Safety Commitments include internal/external red-teaming, independent third-party/home-government evaluations where appropriate, public safety frameworks, and more detailed nonpublic sharing with trusted actors.
- Anthropic RSP evolved through 2026; v3.2 authorized external review of Risk Reports and v3.3 updated thresholds, showing operational governance for trusted external review.
- OpenAI Preparedness Framework v2 commits to scalable automated evals, expert deep dives, capability/safeguards reports, and published findings with frontier releases.
- OpenMined/Syft and TEE approaches offer prototypes for secure privileged evaluations without exposing weights or proprietary/user data.
New Tractable Vectors
- Define access tiers—black-box, tool/API, logit, activation, sandboxed fine-tune, enclave weights, reproducible snapshot—and match them to audit methods and risks.
- Run near-white-box evaluations inside TEEs/MPC so auditors can execute tests without copying weights or revealing all test prompts to providers.
- Create versioned model escrow/deprecation protocols for reproducibility of hosted models.
- Quantify marginal model-theft/misuse risk from logits, embeddings, fine-tuning APIs, activation access, or sandboxed weight access.
Key Open Questions
- Trusted-evaluator access is not the same as broad independent research access; who qualifies and who pays remains open.
- Provider-controlled APIs can change models, filters, sampling, and logging, undermining reproducibility unless snapshots are preserved.
- Secure evaluation stacks must protect three secrets simultaneously: model IP, evaluator test suites, and provider cybersecurity posture.
- External red-team/safety eval results are often partially disclosed, limiting independent verification and cross-model comparability.