François Chollet Co-founds ARC Prize Foundation for Assessing AI Intelligence
Brief news summary
François Chollet, formerly with Google, co-founded the ARC Prize Foundation to create benchmarks for assessing AI's "human-level" intelligence. Managed by ex-Salesforce director Greg Kamradt, the foundation's goal is advancing artificial general intelligence (AGI), enabling AI to perform a variety of human-like tasks. In 2019, Chollet introduced the ARC-AGI test, designed to evaluate an AI’s ability to learn and adapt through complex puzzles, moving past preset training. While AI has progressed, most models solve less than one-third of ARC-AGI tasks. Chollet emphasizes that the test measures human intelligence advancement, not superhuman prowess. Current AI models frequently use brute-force methods for high scores, highlighting challenges in assessing genuine intelligence. OpenAI's unreleased o3 model gained a qualifying score but needed extensive computational power, questioning its human-level intelligence. This year, the ARC Prize Foundation aims to update the ARC-AGI benchmark, host a competition, and develop a third version. The benchmark's efficacy is debated due to the unclear concept of AGI. Despite interest from OpenAI’s CEO, Sam Altman, in collaboration, no partnerships have formed. The foundation continues refining AI benchmarks amid ongoing debate.Former Google engineer and AI researcher François Chollet is co-founding the ARC Prize Foundation, a nonprofit aimed at developing benchmarks to assess AI for "human-level" intelligence. The foundation will be led by Greg Kamradt, a former Salesforce engineering director and founder of AI product studio Leverage, who will act as its president and board member. Chollet describes the foundation's mission as a "north star" guiding progress towards artificial general intelligence (AGI), a term describing AI capable of performing most tasks as well as humans. The ARC Prize Foundation will build on ARC-AGI, a test Chollet created to evaluate if an AI system can learn new skills beyond its training data. This involves puzzle-like problems requiring AI to generate correct "answer" grids from colored squares, testing its ability to adapt to novel situations. Introduced in 2019, ARC-AGI—"Abstract and Reasoning Corpus for Artificial General Intelligence"—has shown that AI systems can excel in areas like Math Olympiads but struggle with ARC-AGI tasks, with the best performing systems solving fewer than a third of them until recently. Chollet emphasizes that unlike typical AI benchmarks focusing on superhuman problems, future ARC-AGI versions aim to eliminate the gap between AI and human capabilities. Last June, a competition began to develop AI that could surpass ARC-AGI, with OpenAI’s unreleased o3 model reaching a qualifying score, though it required vast computational resources.
Chollet acknowledges ARC-AGI's imperfections and doubts o3's human-level intelligence. In a December statement, Chollet anticipated that new benchmarks would challenge o3, potentially lowering its score while humans could still achieve over 95% untrained. He asserts AGI will be evident when creating human-easy but AI-hard tasks becomes impossible. Knoop announced plans to launch a second-gen ARC-AGI benchmark and a new competition this year, alongside designing a third edition. The foundation faces scrutiny over whether ARC-AGI has been overstated as a path to AGI, amidst debate over AGI's definition. Recently, an OpenAI member suggested AGI is already here if defined as AI outperforming most humans in most tasks. OpenAI CEO Sam Altman mentioned in December a plan to collaborate with ARC-AGI on future benchmarks, though Chollet did not confirm any partnership in his latest announcement.
Watch video about
François Chollet Co-founds ARC Prize Foundation for Assessing AI Intelligence
Try our premium solution and start getting clients — at no cost to you
