OpenAI's experimental chatbot model, o3, recently achieved an impressive 87. 5% on the ARC-AGI test, significantly surpassing the previous AI record of 55. 5%. This marks a significant step towards Artificial General Intelligence (AGI), which is defined as a system capable of reasoning, planning, and learning on par with humans. AI researcher François Chollet, who developed the ARC-AGI test, acknowledges o3's substantial reasoning and generalization capabilities, although he notes AGI has not yet been fully realized. The o3 model has excelled in various benchmarks, such as the challenging FrontierMath test by Epoch AI. However, David Rein and other experts remain skeptical about whether ARC-AGI accurately measures AI's reasoning and generalization abilities, urging the need for better assessments. OpenAI has not disclosed details about o3's operation, but it follows the o1 model's 'chain of thought' logic. Some speculate o3 generates multiple reasoning chains to derive the best answer. Despite its high performance, o3's testing process is costly and time-consuming, requiring about 14 minutes per task, raising concerns about sustainability. The concept of AGI lacks a precise definition, making consensus elusive on when AI might achieve it.

Various tests are being developed to track progress, such as Rein’s Google-Proof Q&A and OpenAI's upcoming 2024 MLE-bench, which challenges AI with real-world problems. Good benchmarks must ensure the AI hasn't encountered test questions during training and require true reasoning without shortcuts. Xiang Yue emphasizes the importance of messy, real-world conditions and energy efficiency in tests. Yue's MMMU benchmark assesses chatbots on university-level tasks, with OpenAI's o1 holding the current record score of 78. 2%. In contrast, ARC-AGI focuses on basic skills like math and pattern recognition, providing test-takers with design transformations to infer outcomes. Yue appreciates ARC-AGI's unique perspective in evaluating AI capabilities.