A recent study by the Columbia Journalism Review's Tow Center for Digital Journalism reveals significant accuracy problems with generative AI models used for news searches. Testing eight AI-driven search tools, researchers found that over 60% of queries yielded incorrect information about news sources. About 25% of Americans currently use these AI models instead of traditional search engines, raising concerns about their reliability. The error rates varied among different tools. For instance, Perplexity made mistakes in 37% of queries, while ChatGPT Search saw a 67% error rate (134 out of 200 queries). Grok 3 had the highest at 94%. To conduct the tests, researchers provided direct excerpts from real news articles and asked the AI tools to identify corresponding details like headlines, publishers, dates, and URLs, totaling 1, 600 queries. A concerning trend noted was that instead of declining to answer when unsure, the models often offered plausible-sounding but erroneous responses, a pattern consistent across all tools tested. Premium versions of these AI tools, like Perplexity Pro ($20/month) and Grok 3's premium service ($40/month), sometimes performed worse, as they frequently provided incorrect answers despite correctly address a higher number of prompts. Their tendency to offer uncertain responses contributed to higher overall error rates. The study also raised issues regarding publishers' control over their content.

Some AI tools ignored the Robot Exclusion Protocols meant to prevent unauthorized access to certain content. For instance, Perplexity’s free version identified excerpts from paywalled National Geographic articles, even though access was explicitly disallowed. Furthermore, when AI tools did cite sources, they often linked to syndicated content on sites like Yahoo News instead of original publishers. A significant problem arose with URL fabrication—over half of the citations from Google's Gemini and Grok 3 led to broken or non-existent pages, with Grok 3 having 154 out of 200 citations result in error pages. This situation places publishers in a difficult position: blocking AI crawlers could eliminate attribution, while allowing access facilitates content reuse without benefiting the original sites. Mark Howard, COO of Time magazine, expressed concerns regarding transparency and control, but also suggested potential for improvement, stating that current AI tools will evolve positively. Howard pointedly criticized users who expect complete accuracy from free AI services, suggesting that skepticism is necessary. OpenAI and Microsoft acknowledged the study's findings but did not directly respond to the issues raised. OpenAI emphasized its commitment to supporting publishers, while Microsoft claimed compliance with Robot Exclusion Protocols. This report builds on earlier findings from November 2024, which similarly highlighted accuracy issues with ChatGPT's handling of news content. For further details, the full report is available on the Columbia Journalism Review's website.