None

Amazon Web Services is currently conducting an investigation to determine if Perplexity AI is in violation of its rules, as reported by Wired. Specifically, the cloud division of the company is looking into allegations that Perplexity AI is utilizing a crawler hosted on its servers that disregards the Robots Exclusion Protocol. This protocol, a web standard, involves developers placing a robots. txt file on a domain to instruct bots on whether they can or cannot access a particular page. While adherence to these instructions is voluntary, reputable companies have generally respected them since their implementation in the 1990s. In a previous article, Wired discovered a virtual machine hosted on an Amazon Web Services server with the IP address 44. 221. 181. 252, which was bypassing the robots. txt instructions on its website. This machine is said to have visited various Condé Nast properties multiple times over the last three months to scrape their content. Other publications such as The Guardian, Forbes, and The New York Times also reported multiple visits from the same machine. Wired conducted an experiment where they inputted headlines or brief descriptions of their articles into Perplexity's chatbot to verify if the company was scraping their content.
The chatbot's responses closely paraphrased the articles "with minimal attribution. " A recent Reuters report suggests that Perplexity is not the only AI company bypassing robots. txt files to gather content for training large language models. However, Wired only provided Amazon with information regarding Perplexity AI's crawler. Amazon Web Services stated, "AWS’s terms of service prohibit abusive and illegal activities, and our customers are responsible for complying with those terms. " They further mentioned that they regularly receive reports of alleged abuse and investigate them accordingly. Perplexity's spokesperson, Sara Platnick, responded to Amazon's inquiries, asserting that their crawlers abide by the Robots Exclusion Protocol and do not violate AWS Terms of Service. Platnick also mentioned that Amazon's scrutiny of Wired's media inquiry follows their standard protocol for investigating potential resource abuse reports. However, Platnick admitted to Wired that PerplexityBot will disregard robots. txt when users include a specific URL in their chatbot inquiry. Aravind Srinivas, the CEO of Perplexity, previously denied claims that his company disregarded the Robots Exclusion Protocol and then lied about it. Srinivas did admit that Perplexity utilizes third-party web crawlers in addition to its own, where the bot identified by Wired is one of them. Update, June 28, 2024, 2:20 PM ET: This post has been updated to include Perplexity's statement to Engadget. Update, June 28, 2024, 8:27 PM ET: This post has been updated to include a statement from Amazon Web Services.
Brief news summary
Amazon Web Services (AWS) is conducting an investigation into Perplexity AI for potentially violating its rules, according to Wired. The AWS cloud division is looking into allegations that Perplexity AI's crawler, hosted on AWS servers, is not adhering to the Robots Exclusion Protocol. This protocol allows developers to specify if bots can access certain web pages. Wired previously reported that a virtual machine hosted on AWS servers visited various publishers' websites, bypassing their robots.txt instructions. To confirm this, Wired entered article headlines into Perplexity AI's chatbot and received closely paraphrased results. Other AI companies may also be bypassing robots.txt files to gather content. AWS stated that customers are responsible for complying with its terms of service and that it investigates reports of abuse. Perplexity denied violating the Robots Exclusion Protocol but admitted that its chatbot ignores robots.txt when given specific URLs.
AI-powered Lead Generation in Social Media
and Search Engines
Let AI take control and automatically generate leads for you!

I'm your Content Manager, ready to handle your first test assignment
Learn how AI can help your business.
Let’s talk!
Hot news

Rise of AI Companions Among Single Virginians
New data from Match reveals that 18% of single Virginians have incorporated artificial intelligence (AI) into their romantic lives, a significant increase from 6% the previous year.

Ponzi VCs Are Strangling Blockchain
According to Romeo Kuok, a board member at BGX Ventures, most deals are structured to facilitate quick exits instead of generating long-term enterprise revenue.

Wimbledon's AI Judges Receive Mixed Reviews from …
The All England Club made a landmark change at Wimbledon 2025 by replacing traditional line judges with the AI-powered Hawk-Eye Electronic Line Calling (ELC) system.

JPMorgan's Tests Carbon Credit Tokenization On Th…
JPMorgan Chase & Co.

The ECB Approves Two Blockchain Projects To Moder…
The European Central Bank is embarking on a significant technological transformation.

Nvidia's Power Play
Nvidia, a leading technology company known for graphics processing and artificial intelligence, has announced a strategic partnership to launch Emerald AI, an innovative startup focused on sustainable energy management in data centers.

Senate Strikes AI Provision from GOP Bill After U…
On July 1, 2025, the U.S. Senate overwhelmingly voted 99 to 1 to remove a controversial provision from President Donald Trump's legislative package that sought a nationwide moratorium on state-level AI regulation.