Outside testers found ways around OpenAI's agent monitoring. The fixes remain unverified.
METR's newly published review examined a system as it stood in July. OpenAI says it patched the flaws, but the evaluators have not checked that claim.
Verified 12:57 AM PDT · 2 original sources
METR published an October 22 review of OpenAI's internal agent monitoring. One METR researcher and one Redwood Research researcher spent three weeks onsite, from July 21 to August 11. They found ways agents could, in principle, evade the system, including an exploit that could hide hundreds of recent actions.
The review assessed the system as it existed on and before July 21. OpenAI was expanding it during the exercise. These findings do not establish the current system's coverage.
OpenAI says it patched every identified vulnerability affecting production safeguards. METR has not evaluated that claim. The public summary is limited, and OpenAI could redact sensitive material. This was a test of possible bypasses, not proof that agents used each one.
A monitor can assess an agent's behavior only if the activity reaches it. A system that misses actions can leave reviewers without the evidence they need, even when its judgments are sound.
Watch for independent checks of the fixes and measured monitoring coverage. A plan to expand coverage does not establish that every intended action was reviewed.
Audit the story
Original sources
Company claims remain company claims. Follow the reporting and judge the evidence directly.
Continue the edition