Elon Musk agrees with other AI experts that there is a scarcity of real-world data left for training AI models. During a livestreamed discussion with Stagwell chairman Mark Penn on X, Musk remarked, “We’ve essentially used up the cumulative sum of human knowledge in AI training, " noting that this depletion occurred last year. Musk, who runs the AI company xAI, echoed sentiments from former OpenAI chief scientist Ilya Sutskever, who addressed this issue at the NeurIPS machine learning conference in December. Sutskever mentioned that the AI industry had reached “peak data, ” predicting that the shortage of training data would necessitate a change in current model development practices. Musk suggested that synthetic data, generated by AI itself, is the way forward, saying, “The only way to supplement [real-world data] is with synthetic data, where the AI creates [training data]. With synthetic data, [AI] will grade itself and engage in self-learning. ” Many companies, including Microsoft, Meta, OpenAI, and Anthropic, are already incorporating synthetic data to train their primary AI models.

According to Gartner, 60% of data used for AI and analytics projects in 2024 will be synthetically generated. Microsoft’s recently open-sourced Phi-4 was trained using both synthetic and real-world data, as were Google’s Gemma models. Anthropic also utilized synthetic data for its highly capable Claude 3. 5 Sonnet system, and Meta fine-tuned its latest Llama series of models with AI-generated data. Training models on synthetic data also offers cost benefits. The AI startup Writer claims its Palmyra X 004 model, largely built on synthetic sources, cost just $700, 000 to develop, compared to the estimated $4. 6 million for a similar-sized model from OpenAI.