Generative AI models rely on large training data sets, typically composed of public data from the internet. However, organizations are increasingly restricting access to their data through robots. txt files, fearing the potential impact of generative AI on their businesses. This restriction poses challenges for AI companies that heavily rely on such data. The Data Provenance Initiative's report, titled "Consent in Crisis: The Rapid Decline of the AI Data Commons, " reveals that a significant portion of the data used to train AI models has been restricted in recent years.

This restriction not only affects the quality and freshness of the data but also creates a gap between models that respect robots. txt and those that disregard it. Some potential solutions proposed include licensing data directly from organizations, utilizing synthetic data, or finding ways to extract hidden data, such as that locked away in PDFs. The report emphasizes the need for industry standardization and improved mechanisms for expressing data usage preferences that balance the interests of various stakeholders.