Anthropic's evaluator update makes access and publication rights separate trust questions

An independent evaluator’s name is useful context, but it does not tell a buyer what the evaluator could inspect or say publicly. Anthropic’s October 15 update on independent evaluators offers a more specific arrangement: it has begun giving a six-person METR team employee-like access to assess its models and safety practices.

The useful question is not whether the word “independent” settles trust. It is how access depth, publication freedom and confidentiality limits fit together. The announcement describes evaluation machinery; it does not itself provide a favorable assessment of the systems being evaluated.

Employee-like does not mean unrestricted

Anthropic says the METR team receives company hardware, office space and internal-tool access roughly comparable to its own risk assessment teams. Staff are encouraged to share information as they would with colleagues working on alignment risk, outside specified categories.

Those exclusions include customer data, attorney-client privileged information and model-architecture information. A footnote adds that highly relevant architecture information may generally be shared with explicit internal approvals. The boundaries therefore should not be flattened into either complete access or a claim that architecture information can never be shared.

My inference is that this depth can address a limitation of purely external testing: an evaluator may need information about training and safeguards, not just model outputs. But access is an enabling condition for an assessment, not proof that the assessment will find risks acceptable.

Publication freedom has a confidentiality boundary

METR will publish analysis of Anthropic’s risk levels, incidents or practices, according to the announcement. Anthropic says it cannot veto publication, but can require redactions within predefined information categories. Crucially, the company ultimately decides what falls within those categories, while METR reserves the right to make certain statements about the access and information it received.

That is a more qualified proposition than either “the company approves every conclusion” or “the evaluator can publish everything.” The October 16 clarification noted in the article’s footnote is part of the current account and matters to that distinction.

For readers of a future assessment, the practical questions are what evidence the evaluator could examine and what limitations it reports. Confidentiality may have legitimate purposes, but it can also constrain what a public reader can independently inspect. Those are analytical considerations, not allegations of a specific withheld finding.

Do not treat every partnership as the same arrangement

Anthropic describes three evaluator tiers. At the first, investigations are narrow and publication needs its approval. The second allows publication without approval, subject to category-based redactions. The third is the employee-like METR arrangement, with publication rights similar to the second. Other developing partnerships therefore should not automatically inherit METR’s described access.

The embedded METR team is also siloed from other METR staff. This is not a route for unrelated organizations to obtain confidential information. Anthropic describes the embedding as voluntary.

For a product or procurement decision, the defensible takeaway is to ask for the actual assessment when available and read its scope and limitations. A stronger process can improve the basis for scrutiny without predetermining its conclusion. Evaluator access is a meaningful disclosure about how trust may be examined, not a safety certificate.