How to scale the impact of embedded AI evaluators

Introduction

Over the last few weeks, there’s been a big shift in the debate about who gets to evaluate the world’s most powerful AI systems. Dario Amodei published an essay committing Anthropic, unilaterally and immediately, to granting independent evaluators permanent, employee-level access to its models. Sam Altman pledged that OpenAI would bring in outside evaluators and Elon Musk signalled his agreement.

Beyond the labs, nation states continue to build capacity to conduct evaluations of frontier AI, and new legislative proposals are emerging for creating pathways for independent oversight, such as Independent Verification Organisations.

That consensus is welcome, but there are practical problems that willingness alone cannot solve. The first problem is the significant friction currently involved in enabling external parties to examine model weights, training data, user logs and other components. AI owners are right to be wary of the privacy, security and commercial risks involved. This friction bottlenecks how scalable such an evaluation program can be. This is a particularly bad outcome for frontier AI, as a limited number of evaluators cannot cover the breadth of questions society needs answered about these systems. The second problem runs in the other direction. Evaluators who reveal their tests and prompts to the labs they scrutinise risk seeing those benchmarks gamed, quietly trained against or leaked — undermining the very independence that gives their results value.

Because of these constraints, even the most generous access commitments face a cost problem. Each seat in embedded evaluator programs — where a vetted external party is given direct access to a lab’s systems — is very expensive: it requires security clearance, privacy review, and ongoing access management. A program that can support ten embedded evaluators cannot necessarily support the diversity of evaluation required to ensure AI is safe across all aspects of society, but dramatically increasing the scale of an embedded researcher program can come with impractical costs. The result is a bottleneck: the willingness to grant access outpaces the infrastructure to do so safely and at scale.

What if every embedded evaluator could turn their work into a reusable, governed endpoint that larger numbers of external researchers call without a new approval cycle? That is the problem PySyft solves. An embedded evaluator writes a job against real data. The AI owner (e.g. a frontier lab employee) reviews and approves it. From that point on, any authorised external party can re-run that job (on fresh data, with parametrised inputs) and receive filtered results, without ever seeing the underlying assets and without the AI owner repeating the review.

This post is updated from a version we first published in 2023. It describes the limitations of the main approaches to external access (open release, onsite access and regular APIs), and introduces a method of external access we’ve developed, called PySyft. PySyft enables an external evaluator to propose a question about an AI system to its owner and related third parties and — if they approve the question — the evaluator is able to download the answer to that question without the evaluator, AI owner, or third parties learning anything beyond what the group explicitly approved.

This is no longer a proposal. Last month, PySyft was used by Google DeepMind, AVERI, the Singapore AI Safety Institute, and MLCommons to run the world’s first double-blind evaluation of a proprietary, frontier-class AI model. The evaluators never saw Gemini’s weights, and DeepMind never saw the evaluators’ prompts or intermediate findings. The rest of this post explains how that kind of arrangement works — and how to build one yourself.

The challenge of external access

Enabling external access to AI models is hard, just as it is hard for an untrusted external actor to answer a question about a private, secure, valuable piece of intellectual property inside of an organization. Solving this problem would mean that one party (an independent evaluator) would be able to answer a question about a second party’s assets (an AI system). In the ideal case, the first party would be the only party to learn anything in the process — the answer to their question (e.g., about an AI system’s safety) — while the second party learns nothing they didn’t already know (they only ever see their own data/model). Facilitating this shared outcome requires achieving an ideal called structured transparency.

Current approaches to external access

In the context of AI evaluations, there are three popular approaches to this challenge. They all fall short of the ideal of structured transparency. The approach we have developed — PySyft — combines the best properties from these approaches. So, before introducing PySyft, we will briefly summarize these other popular approaches as an important bridge to our proposal.

There three popular approaches to enabling external access to AI systems are:

1. Open access: release AI models and datasets to external parties via an insecure environment (such as a public webpage).
2. Onsite-access: store AI models and datasets in a secure environment, and bring external evaluators into this environment for study.
3. API access: store AI models and datasets in a secure environment, and build an API which allows external evaluators to extract specific pieces of information from the secure environment.

Open access

For most production AI systems, open access is a non-starter because of privacy, security, and IP concerns. Even if external parties sign various confidentiality agreements, it’s almost impossible to enforce what someone does with your information once you share it with them.

Onsite access

Onsite-access is the legacy approach of governments and corporations around the world for managing their information. Trusted employees access official, organization-owned devices inside of secure facilities which require security badges to enter. This has been extended to external parties through the combined use of both trust exercises (background checks, etc.) and — more or less — requiring trusted internal parties to be the ones who actually produce research results for untrusted external parties.

For example, the US government runs Federal Statistical Research Data Centers (FSRDCs). Researchers with special sworn status (SSS) can enter a physical building (without bringing electronics with them), sit down at a secure computer, perform analysis against sensitive data, and then submit their code to an analyst who works for the FSRDC. The researcher then leaves the facility (without their results), and waits for their project to be reviewed. If it passes the review, the internal employee runs the code, produces the result, and transmits it to the external researcher.

API access

API access is — in effect — an extension of this same philosophy as onsite-access, but with a greater potential for scale. If it is likely that many external researchers will run variations of the same query, and if variations of that query are approved by internal employees of the organization, the organization can build a custom web application allowing external researchers to ask and answer these specific research questions. This has all the benefits of onsite-access with one additional benefit, it can operate at much lower cost and higher scale for everyone involved. Researchers need not travel to physical locations and owners of secure information need not host them there. In the context of AI evaluations, this is often called structured access.

However, API access has a drawback in practice; it constrains researchers only to the questions which can be answered by the custom API. In this sense, if external researchers all want to ask the same types of questions, an API is more scalable. But if external researchers all want to ask different types of questions, an API is less scalable — an organization would be constantly hiring software engineers to extend its API with new workloads and endpoints for each research project.

Trusted intermediaries don’t scale

Trusted intermediaries are sometimes posited as a solution to enabling external access. But finding and working with trusted intermediaries is all but impractical because it requires just the right balance of incentives. Let’s say an external evaluator wants to perform analysis using data from an organization. It needs to find an organization that both parties trust enough to hold their valuable, secure, private, IP — but which is also somehow motivated and experienced enough in the domains of both to effectively steward the information. This problem worsens when an evaluator requires access to data held by more than one party. For example, if one wanted to perform analysis against data from a hospital and a bank, one would need to find a trusted intermediary which the bank and the hospital both trusted, and which also understood healthcare and banking well enough to know how to protect the data — to know what uses were appropriate or inappropriate. However, it turns out that expertise in healthcare and banking usually means that the organization operates in the healthcare and banking industries. If they operate in the healthcare and banking industries, it reduces the chances that they don’t have a conflict-of-interest with holding data from a hospital and a bank.

This is not impossible, and trusted intermediaries do exist. However, there’s a trust-at-scale issue. The more parties an AI owner or external evaluator seeks to work with, the harder it will be to find a trusted intermediary capable of jointly understanding and protecting the interests of all parties involved. And as the number of relationships increases, so too do the dynamics (will the intermediary give any of the parties preferential treatment over the others?).

For understanding the impacts of AI in society, this trust-at-scale issue is of paramount importance. Consider trying to understand the impacts of newsfeed, recommender, or credit scoring algorithms on the health and welfare of society’s members. “Health and welfare” means medical, financial, cultural, educational, political, and many other sensitive areas of possible impact. At the end of it all, is there an intermediary who is both experienced in — and neutrally separated from — all of these application areas? Surely such a party does not exist.

Our approach

At OpenMined, we are advancing a new method for external access called PySyft. PySyft enables an external evaluator to propose a question about an AI system to its owner and related third parties and — if they approve the question — the evaluator is able to download the answer to that question without the evaluator, AI owner, or third parties learning anything beyond what the group explicitly approved.

Three roles, defined

An external access program using PySyft involves three roles, each with different permissions and incentives. They are:

1. The embedded evaluator has direct access to the AI owner’s assets: model weights, training data, user logs, or whatever the program covers. They sit inside the AI owner’s security perimeter (physically or virtually) and write code against real data. Their output is a reusable job, rather than a one-time report: a script that future callers can re-run without repeating the review cycle. The embedded evaluator is the expensive seat; every one requires security clearance, privacy review, and ongoing access management.
2. The internal reviewer is an employee of the AI owner. They review the embedded evaluator’s proposed job, decide whether its outputs are safe to release, and approve. They can also set output controls: redaction rules, rate limits, and result review requirements. Once they approve, the job becomes a governed endpoint.
3. The external evaluator never sees the underlying assets. They discover approved jobs, call them (potentially with parameters like a date range or a demographic filter), and receive the filtered outputs. They trust that the embedded evaluator’s description of the job is accurate, and that the internal reviewer’s approval is meaningful. This is the role that scales: one embedded seat can serve hundreds of external callers.

An optional fourth tier, scheduled public reporting, runs approved jobs on a cadence and publishes results to regulators or the public without any human triggering the run.

This role separation is not new. The US government’s Federal Statistical Research Data Centers already work this way: a sworn-status researcher writes the query, an internal analyst reviews and runs it, and approved results are transmitted out. Using PySyft, this pattern is programmable and repeatable.

Workflow

An AI owner and evaluator must jointly execute several steps together. First, the AI owner launches a new type of user-extendable web server — a “domain server” — and loads it with relevant datasets and AI models. This could be an AI model’s training data, user log, or other relevant metadata.

An evaluator then “proposes a project”, which is a bit of code they’d like to be added to the Domain Server’s API. This code typically interacts with datasets and AI models in a specific way, so that the external evaluator can use it to answer important question in the future. The model owner reviews this project (reviews the code) and — if they approve — the code is added to the API. And against this new API, the external evaluator can download their results. Below, you can see this process broken apart into a series of 7 steps:

A flexible query API

PySyft uses a type of API that allows for particularly flexible use. Instead of an API which allows an external party to call specific functions on an AI model, the API offers mock (fake) versions of the private datasets and models behind the API. This allows an external evaluator to write code against these fake versions (as if they had access to the real datasets and models). Then, they can use the API to submit this code to the AI owner. When they do so, they’re asking permission to answer a specific question. And if the AI owner is willing to run their code and see the result, the AI owner is granting them permission to answer their question.

This approach is beneficial because it captures the best aspects of open access, onsite-access, and API access. It’s like open access in that an external evaluator downloads datasets and AI models, allowing them to write code locally. It’s like onsite-access in that an AI evaluator is largely unconstrained in the types of questions they can pose of the AI system. Yet, it is like an API in that the external evaluator only learns information specifically approved by the AI owner. Taken together, flexible query APIs offer a compelling-yet-familiar advantage over the three previous approaches.

Tiered queries

Not every question requires a fresh review cycle. Once an embedded evaluator’s job has been approved, PySyft supports three tiers of external access:

1. Pre-approved queries: an external evaluator calls an approved job exactly as registered. The result requires no human in the loop, as it’s returned automatically. This is the workhorse tier: it covers recurring evaluations, scheduled compliance checks, and, in general, any case where the question doesn’t change across runs.
2. Parametrised queries: an external evaluator calls an approved job but varies specific inputs that the embedded evaluator marked as safe to change: a date range, a demographic filter, a sample size, etc. However, the job runs within the bounds of the original approval. This lets a single approved script serve many different research questions without a new review.
3. Novel submissions: an external evaluator submits entirely new code. This enters the review queue and requires explicit approval from the internal reviewer before execution.

Note that this tiered model means that the internal reviewer’s time scales with the number of submitted jobs, rather than with the number of evaluators. This is a massive efficiency gain.

Hardware and software that provably follow instructions

There are multiple routes to verifying processes and outcomes of external access, but for the moment we will focus on the most accessible version to explain: a secure enclave. A secure enclave is a type of computer chip which is special in two ways. First, when it is manufactured, a “private key” is burned into the chip in a way that no-one else can know its value. This allows an external party to encrypt information using a “public key” with the knowledge that the only computer in the entire world that could decrypt this information and use it is the secure enclave chip.

Second, a secure enclave chip has a critical core capability built into its hardware: it produces a hash of what software it is running. This allows multiple owners to know — ahead of time — what the computer chip would do with their information if that information was sent into the secure enclave. In the context of our approach we call these owners “shareholders”, because we can design software which empowers multiple owners with joint control over what the enclave does.

There is more to a secure enclave in practice, but this is enough to understand the general principle. If parties are not inclined to trust a secure enclave, there are other software approaches they can use, broadly termed “secure multi-party computation”. This can facilitate exactly the same outcome, it just uses software to do so instead of hardware.

So far, this post has focused on evaluations involving one external party and one AI model owner. But some of the most important questions about AI systems require combining assets from multiple organisations; for example, demographic data from a third party to measure an AI system’s fairness across populations. This is a harder coordination problem: each data owner needs assurance that their information won’t be exposed to the others and, realistically, no single trusted intermediary can serve all parties. Secure enclaves solve this by giving multiple data holders joint control over what runs inside the enclave, without any party needing to trust another or relinquish control to an intermediary. This multi-party case is what SETUPs 3-5 are designed for, and we’ll cover them in future posts.

Conclusion

There is more to discuss when considering how to do this in practice. How do 3rd party shareholders come to meet one another, and what does the governance process look like? How do they approve or deny extensions of the static API when they cannot see the information contributed from other parties? When the evaluator receives their results, how do they know they haven’t been faked? And when a consumer later uses an AI product or service, how do they know they are receiving predictions from an AI model that has in fact been evaluated? This involves a myriad of important topics: meta-evaluations, evaluation registries, signed hashes, digital identity, and more.

On that last set of questions (verification and trust) there has been real progress since Part 1. In 2024, we collaborated with Anthropic and the UK AI Security Institute to demonstrate that two organizations could jointly approve and run an evaluation inside a secure enclave, with neither party able to see the other’s sensitive assets. That pilot used GPT-2 as a stand-in for a frontier model and a five-row evaluation as a proxy for a confidential benchmark. It proved the mechanism worked end-to-end. More recently, in August 2026, we worked with AVERI, Google DeepMind and MLCommons to conduct the world’s first double-blind evaluation of a proprietary frontier-class model. Gemini 2.5 Flash-Lite was evaluated against never-before-used prompts from the MLCommons AILuminate benchmark, inside a secure enclave where Google could not see the prompts and the evaluators could not see the model weights. This is built on the same structured transparency stack that this post describes.

There is also more to discuss with respect to domain specific applications. There are already a myriad of tools emerging for studying AI bias, fairness, plagiarism, value alignment, illegal content, disinformation, and copyright. For the most part, these tools assume that the access problem has already been solved, focusing instead on how to measure these properties using AI models they can already access. We’ll publish a hands-on tutorial for SETUP 1 — the simplest tier, where an evaluator trusts that the model owner’s assets are genuine — very soon. And as the double-blind evaluation pilot demonstrates, the same ideas scale up. SETUPs 3 through 5 (where the evaluator can cryptographically verify they’re working with genuine assets) are now not just theoretical. They’ve been demonstrated on a production frontier model. We’ll explore those higher SETUPs in future posts.

If you’d like to play with code yourself, you can find all the code here. If you’d like to get involved in developing and using the tools you’ve seen here, please join us on slack at slack.openmined.org

Authors

Alejandro Tlaie Boria, Andrew Trask, Jack Hardinges, Koen van der Veen, Lacey Strahm.

Acknowledgements

We’d like to again thank all those who provided feedback, editing, and guidance on the original 2023 tutorial, including: Dewey Murdick, Emma Bluemke, Heidi Mayer, Helen Toner, Hema Krishnamurthy, Jan Leike, Jiahao Chen, Joe O’Brien, John Farkas, Matt Prewitt, Nahema Marchal, Rumman Chowdhury, Sean McGregor, Sebastian Farquhar, Shahar Avin, Teddy Collins, William Isaac, and Yonadav Shavit.