The AI industry may have reached "peak data, " according to OpenAI cofounder Ilya Sutskever, signaling a potential slowdown in AI advancements due to the depletion of useful data from the internet. This could impact the future growth of AI models, which rely heavily on pre-training with abundant data. Despite this, many AI experts are exploring ways to circumvent this issue. One promising approach is the "test-time" or "inference-time compute" technique, which improves AI's reasoning capabilities by breaking down complex queries into smaller tasks and processing each separately before progressing. This method allows AI models to generate higher-quality outputs, especially in tasks with clear-cut answers like math problems. The outputs from these reasoning models could become new training data, forming an iterative loop for model improvement.

This concept was backed by research from Google DeepMind, which envisions these outputs enhancing large language models (LLMs) even after hitting the peak-data wall. OpenAI and similar AI labs have begun deploying models employing this technique, such as OpenAI's "o1, " which shows superior performance in certain benchmarks. Microsoft CEO Satya Nadella has referred to this strategy as an essential scaling law for advancing AI models, as it provides a way to circumvent data limitations by feeding model outputs back into training processes. The effectiveness of test-time compute will be more thoroughly evaluated by 2025. While researchers like Charlie Snell are hopeful, they acknowledge challenges in generalizing the technique to tasks without definitive answers, such as essay writing. Nonetheless, there's optimism that synthetic data generated through this method could surpass existing internet data quality, potentially aiding in training future AI models. Already, some speculations suggest that companies like DeepSeek have used outputs from OpenAI's o1 to enhance their models, such as their latest "DeepSeek V3. " As the industry navigates these strategies, the potential to use test-time compute to overcome data limitations is cautiously promising but still under exploration.