Who gets to look behind an AI company’s benchmark chart?

What the announcement establishes

In its 15 October account, Anthropic says a METR team is receiving employee-like access for risk evaluation. It says METR can publish assessments, subject to predefined redaction categories, and describes boundaries on the access. This is a company statement about an evaluation arrangement. It does not itself supply METR’s eventual findings.

A demonstration answers the question it was shown

Imagine two fictional assistants asked to organise a folder. One demonstration shows ten clearly named files sorted into the right places. Another test includes duplicate names, a missing date and a file that should remain untouched. Both tests may be honestly reported, but they answer different questions. The first shows success on a neat example. The second asks what happens when the job becomes ambiguous.

When comparing new agent products, look for the test’s starting conditions. Could the agent retry? Did a person choose the best output? Was the tool allowed to use search? Were failed attempts included? These details are part of the result. A single number with none of this context can make different systems appear comparable when they performed different jobs.

Independent access can change what is discoverable

An outside evaluator who only sees a public demonstration cannot inspect everything the builder knows. More access may allow different questions: which failure appeared during development, which safeguard caught it, and what remained unresolved? Publication freedom matters because a reader needs to hear more than the successful examples. Independence and access are useful properties to inspect separately; neither makes an evaluator infallible.

For your own small tool, use a modest version of this idea. Write the intended job and all test inputs before looking at the final output. Preserve failures beside successes. Let another person or assistant try a different case, and state who controlled the test. This is practical evidence about a bounded job. It does not turn a handful of trials into a general claim about intelligence.

Follow the evidence after the headline

Keep a short watchlist with three columns: announcement, evidence available now and next observation to look for. For this story, the next observation is an actual evaluator publication or a material change to the arrangement. For a new tool, it might be a reproducible task with reported costs. This gives you a better way to follow the frontier: notice which claims have gained evidence, which remain open, and which have changed.