OpenAI's latest large language model, known as "o3, " has yet to be widely released, but preliminary tests hint at its abilities. The model was briefly introduced via a promotional video, with few details about its capabilities. Notably, o3 was tested using the "Abstraction and Reasoning Corpus for Artificial General Intelligence" (ARC-AGI), a benchmark designed to measure a model's ability to adapt to novel situations. In this test, o3 achieved a 76% accuracy, surpassing human scores recorded by Mechanical Turk workers, marking what some consider a breakthrough in AI's ability to tackle new tasks. François Chollet, who developed ARC-AGI, highlights that o3's score demonstrates a significant leap in AI capability, suggesting o3 could soon compete with human work. Although Chollet has previously been skeptical about AI reaching human-level intelligence, he acknowledges this development as a shift.

The ARC-AGI test involves solving visual puzzles that do not rely on language, challenging models like o3 in new ways. Despite its successes, o3's methods remain opaque due to its closed-source nature. Chollet speculates that the model's architecture differs significantly from its predecessors, employing a test-time search approach akin to that used by Google's AlphaZero. However, Chollet notes that OpenAI hasn't disclosed the resource expenditure for achieving ARC-AGI scores, which could affect perceived efficiency. Questions about o3's general adaptability remain, as it was specifically trained for the ARC-AGI test. Chollet emphasizes that while o3 shows promise, it still fails on some simple tasks, suggesting it's not yet at AGI (artificial general intelligence) level. He plans to release an updated version of ARC-AGI to further challenge models like o3, indicating that true AGI is still out of reach for now.