Researchers have issued a warning about the potential consequences of training AI on machine-generated text, referring to it as "model collapse. " They use the analogy of photocopying a piece of paper multiple times, where artifacts start to appear. In a similar manner, future AI models trained on the internet, where AI-generated content is expected to become widespread, may absorb the flawed work of their predecessors. This occurrence, known as the AI ouroboros, could negatively impact the predictions made by AI chatbots, as they rely on internet data for training. The researchers from Canada and the U. K. express concern that as AI-generated content becomes more prevalent on the internet, it could corrupt the data that AI models learn from. This would lead to biased predictions by future AI chatbots, favoring common occurrences and neglecting outliers.

The phenomenon of model collapse intensifies with each iteration, resulting in a distorted representation of reality that undermines the reliability of the model's predictions. The researchers constructed mathematical models to investigate the theoretical consequences of this degenerative learning process. They highlight the significance of this issue, casting doubt on the prevailing notion that the rapid progress in large language models (LLMs) will continue without setbacks. Consequently, they propose potential solutions such as training models to differentiate between content produced by humans and AI, although this may become challenging as AI technology advances. Another approach could involve curating high-quality human-generated data, but coordination amidst increasing competition among chatbots presents a challenge. While the researchers believe that chatbots currently have access to sufficient human-generated data for their development, they caution that early indicators of LLM-induced data poisoning, such as information manipulation and biased amplification against marginalized populations, might not be far off. As a result, they emphasize the need to balance resources for addressing short-term concerns while preparing for the capabilities of increasingly advanced machines.