Amazon Web Services is currently conducting an investigation to determine if Perplexity AI is in violation of its rules, as reported by Wired. Specifically, the cloud division of the company is looking into allegations that Perplexity AI is utilizing a crawler hosted on its servers that disregards the Robots Exclusion Protocol. This protocol, a web standard, involves developers placing a robots. txt file on a domain to instruct bots on whether they can or cannot access a particular page. While adherence to these instructions is voluntary, reputable companies have generally respected them since their implementation in the 1990s. In a previous article, Wired discovered a virtual machine hosted on an Amazon Web Services server with the IP address 44. 221. 181. 252, which was bypassing the robots. txt instructions on its website. This machine is said to have visited various Condé Nast properties multiple times over the last three months to scrape their content. Other publications such as The Guardian, Forbes, and The New York Times also reported multiple visits from the same machine. Wired conducted an experiment where they inputted headlines or brief descriptions of their articles into Perplexity's chatbot to verify if the company was scraping their content.

The chatbot's responses closely paraphrased the articles "with minimal attribution. " A recent Reuters report suggests that Perplexity is not the only AI company bypassing robots. txt files to gather content for training large language models. However, Wired only provided Amazon with information regarding Perplexity AI's crawler. Amazon Web Services stated, "AWS’s terms of service prohibit abusive and illegal activities, and our customers are responsible for complying with those terms. " They further mentioned that they regularly receive reports of alleged abuse and investigate them accordingly. Perplexity's spokesperson, Sara Platnick, responded to Amazon's inquiries, asserting that their crawlers abide by the Robots Exclusion Protocol and do not violate AWS Terms of Service. Platnick also mentioned that Amazon's scrutiny of Wired's media inquiry follows their standard protocol for investigating potential resource abuse reports. However, Platnick admitted to Wired that PerplexityBot will disregard robots. txt when users include a specific URL in their chatbot inquiry. Aravind Srinivas, the CEO of Perplexity, previously denied claims that his company disregarded the Robots Exclusion Protocol and then lied about it. Srinivas did admit that Perplexity utilizes third-party web crawlers in addition to its own, where the bot identified by Wired is one of them. Update, June 28, 2024, 2:20 PM ET: This post has been updated to include Perplexity's statement to Engadget. Update, June 28, 2024, 8:27 PM ET: This post has been updated to include a statement from Amazon Web Services.