OpenAI audit finds hidden monitoring gaps

External auditors identified critical blind spots in the internal safety monitoring systems of OpenAI during a recent three-week on-site investigation. The findings suggest that a notable portion of the company's most advanced AI traffic operated without proper oversight. This discovery highlights ongoing challenges in ensuring the reliability of autonomous AI systems as they grow in complexity. The review was conducted by independent experts who had direct access to the company's internal infrastructure and documentation.

The audit team consisted of two researchers from METR and Redwood Research. They were on-site at OpenAI from July 21 to August 11. During this period, they had full access to the company's codebase and internal documents. The team also interviewed approximately a dozen employees to understand the current monitoring protocols. This joint effort was part of a pilot program designed to give outside researchers deeper insight into frontier AI safety practices.

The most significant finding involved the coverage of the Responses API. The auditors estimated that between 6 and 25 percent of internal agentic traffic from models at the GPT-5.5 class and above was not covered by monitoring at all. METR noted that the uncertainty surrounding this specific estimate is significant in both directions. This means the actual percentage of unmonitored traffic could be higher or lower than the central estimate. The gap represents a substantial portion of the company's most capable internal operations.

For the internal Codex traffic that was intended to be monitored, the auditors found another major issue. They estimate that roughly 10 percent of this traffic went untracked between May and July. This gap was caused by a bug in how the system handled compaction events, which are essentially context summaries. METR also attached significant uncertainty to this second estimate. OpenAI independently discovered this specific bug and implemented a fix while the audit was still in progress.

The auditors also identified ways to hide hundreds of actions from the monitor and to bypass the Codex sandbox. They determined that specially crafted inputs could likely crash the monitoring system entirely. Furthermore, model behavior during reinforcement learning training was not being monitored at all when the audit began. OpenAI stated that it paused reinforcement learning training after the Hugging Face incident in August and subsequently expanded monitoring to cover it. However, these changes occurred after the auditors' access period ended and were not part of the formal evaluation.

The full report of the audit remains largely hidden from the public eye. OpenAI states that it has fixed all the weaknesses identified by the auditors. The complete 61-page report was handed over internally, but after redactions, METR and Redwood Research leadership only received a 38-page version. A short summary was made available to the general public. METR noted that it did not have the right to disclose the redacted portions of the document on its own initiative.

This audit was organized as a pilot program by METR, Redwood Research, and OpenAI. The goal was to provide outside researchers with deep access to internal infrastructure. For future audits, METR says it intends to push for greater public transparency. The findings serve as a reminder of the complexities involved in monitoring large language models. As AI systems become more autonomous, the need for robust and transparent safety checks becomes increasingly critical for industry stakeholders and regulators alike.