A nonprofit organization working on math benchmarks for AI has recently come under scrutiny for not disclosing its financial backing from OpenAI until now, prompting accusations of impropriety within the AI community. Epoch AI, a nonprofit primarily supported by Open Philanthropy— a research and grant-making foundation— announced on December 20 that OpenAI funded the development of FrontierMath. This benchmarking test features expert-level problems to evaluate an AI's mathematical capabilities and was utilized by OpenAI to demonstrate its upcoming flagship AI, o3. In a post on the forum LessWrong, a contractor for Epoch AI using the username "Meemi" claimed that many contributors to the FrontierMath benchmark were unaware of OpenAI's involvement until it was publicly revealed. "The communication regarding this has been non-transparent, ” Meemi stated. “In my opinion, Epoch AI should have disclosed OpenAI’s funding, and contributors ought to have clear information about the potential implications of their work before deciding to participate in a benchmark. " Some users on social media expressed concerns that the lack of transparency could damage FrontierMath's standing as an impartial benchmark. Alongside funding FrontierMath, OpenAI had access to numerous problems and solutions within the benchmark—a detail Epoch AI did not share before December 20, the day o3 was announced. In response to Meemi's comments, Tamay Besiroglu, the associate director of Epoch AI and one of its co-founders, maintained that the integrity of FrontierMath was unaffected but acknowledged that Epoch AI "erred" in failing to be more forthright. "We were bound by restrictions on disclosing the partnership until around the o3 launch, and in hindsight, we should have insisted on being more transparent with benchmark contributors as soon as feasible, " Besiroglu wrote.

"Our mathematicians deserved to know who might have access to their contributions. Even with contractual limitations on our disclosures, we should have prioritized transparency with our contributors in our agreement with OpenAI. " Besiroglu clarified that, while OpenAI has access to FrontierMath, there is a "verbal agreement" preventing it from using the problem set to train its AI—essentially avoiding "teaching to the test. " Additionally, Epoch AI maintains a "separate holdout set" to ensure independent verification of FrontierMath benchmark results, Besiroglu explained. "OpenAI has …fully supported our choice to retain a separate, unseen holdout set, " he added. However, the situation was complicated when Epoch AI's lead mathematician, Ellot Glazer, noted in a Reddit post that Epoch AI has not yet been able to independently verify OpenAI's FrontierMath results for o3. "In my view, [OpenAI's] score is genuine (i. e. , they did not train on the dataset), and they have no motivation to misrepresent their internal benchmark performances, " Glazer remarked. "However, we cannot provide confirmation until our independent evaluation concludes. "