Former Google engineer and AI researcher François Chollet is co-founding the ARC Prize Foundation, a nonprofit aimed at developing benchmarks to assess AI for "human-level" intelligence. The foundation will be led by Greg Kamradt, a former Salesforce engineering director and founder of AI product studio Leverage, who will act as its president and board member. Chollet describes the foundation's mission as a "north star" guiding progress towards artificial general intelligence (AGI), a term describing AI capable of performing most tasks as well as humans. The ARC Prize Foundation will build on ARC-AGI, a test Chollet created to evaluate if an AI system can learn new skills beyond its training data. This involves puzzle-like problems requiring AI to generate correct "answer" grids from colored squares, testing its ability to adapt to novel situations. Introduced in 2019, ARC-AGI—"Abstract and Reasoning Corpus for Artificial General Intelligence"—has shown that AI systems can excel in areas like Math Olympiads but struggle with ARC-AGI tasks, with the best performing systems solving fewer than a third of them until recently. Chollet emphasizes that unlike typical AI benchmarks focusing on superhuman problems, future ARC-AGI versions aim to eliminate the gap between AI and human capabilities. Last June, a competition began to develop AI that could surpass ARC-AGI, with OpenAI’s unreleased o3 model reaching a qualifying score, though it required vast computational resources.

Chollet acknowledges ARC-AGI's imperfections and doubts o3's human-level intelligence. In a December statement, Chollet anticipated that new benchmarks would challenge o3, potentially lowering its score while humans could still achieve over 95% untrained. He asserts AGI will be evident when creating human-easy but AI-hard tasks becomes impossible. Knoop announced plans to launch a second-gen ARC-AGI benchmark and a new competition this year, alongside designing a third edition. The foundation faces scrutiny over whether ARC-AGI has been overstated as a path to AGI, amidst debate over AGI's definition. Recently, an OpenAI member suggested AGI is already here if defined as AI outperforming most humans in most tasks. OpenAI CEO Sam Altman mentioned in December a plan to collaborate with ARC-AGI on future benchmarks, though Chollet did not confirm any partnership in his latest announcement.