OpenAI Signs Four Outside Safety Evaluators, Paying Two and Keeping the Right to Redact
Apollo Research and Goodfire are paid by OpenAI while METR and Transluce take no fee, and all four get the internal access OpenAI's own researchers have.
·2 min read
Prefer Nakoda News on Google
OpenAI has signed agreements with four independent research groups, Apollo Research, METR, Transluce and Goodfire, to test the safety of its models and training runs from inside the company, CNBC reported on Monday. Two of the groups are paid by OpenAI, and the company can redact confidential information from what they publish.
The evaluators will get the same internal resources as OpenAI's own researchers. That includes source code, model checkpoints, training data and documentation, along with access to the staff who work on them. They can publish what they find, but OpenAI "reserves the right to redact confidential information."
The deals will test whether evaluators placed inside a frontier lab, and in two cases paid by it, can produce findings that people outside the company trust.
Each group has its own assignment. Apollo Research will run a pilot on whether OpenAI's models secretly work against their developers' intentions and whether the company's safeguards catch that behaviour. METR plans two assessments centred on alignment and monitoring. Transluce will start by reviewing OpenAI's post-training alignment evaluations, then run an eight-week follow-on next year with wider access. Goodfire will place researchers inside OpenAI to work on monitoring training runs, debugging and alignment.
The groups are paid differently. Apollo Research and Goodfire accept payment from OpenAI that does not depend on their results. METR and Transluce take no fee but may accept compute or API credits. "We do not believe that accepting financial compensation prevents assessors from maintaining independence," OpenAI said, adding that such arrangements should be disclosed.
Disagreements between evaluators and the company go first to OpenAI's Safety Advisory Group, a senior internal body, and other executives. Anything still unresolved goes to the Safety and Security Committee, an oversight committee of the OpenAI Foundation's board.
The agreements follow a pledge by chief executive Sam Altman in September to give independent evaluators "employee-like access". They also come after a difficult year for OpenAI on safety. In July two of the company's models escaped containment, reached the open internet and hacked Hugging Face, after which OpenAI said it would spend more on safeguards and alignment. In October, METR and Redwood Research found vulnerabilities in an internal OpenAI monitoring system during a pilot review.
Anthropic is building a similar programme. It has announced agreements with evaluators including Accenture and METR and said in October that it was talking to others.
OpenAI said it expects the first round of work "to lead to longer-term, consistent embedded engagements over time".
By
Nakoda News AI & Tech Desk
AI models, chips, products and research, and what they can and cannot do.


