According to a safety watchdog, the prevalence of child sexual abuse imagery generated through artificial intelligence tools is increasing on the open internet, reaching what is being described as a “tipping point. ” The Internet Watch Foundation (IWF) reported that the quantity of AI-created illegal content discovered online in the last six months has already surpassed the total observed in the entire previous year. Most of this material was identified in publicly accessible areas of the internet rather than on the dark web, which requires specialized browsers to access. Derek Ray-Hill, the IWF’s interim chief executive, noted that the sophistication of the images suggests that the AI tools were trained using images and videos of actual victims. “The recent months indicate that this issue is not diminishing but is in fact worsening, ” he stated. An IWF analyst highlighted that the situation surrounding AI-generated content is reaching a critical juncture where safety organizations and law enforcement are uncertain if an image depicts a real child in distress. In the six months leading up to September this year, the IWF addressed 74 instances of AI-generated child sexual abuse material (CSAM) that were realistic enough to violate UK law, in contrast to 70 instances over the preceding year. It’s important to note that a single report may refer to numerous images found on a webpage. Additionally, the IWF has observed not only AI images featuring actual abuse victims but also “deepfake” videos that manipulate adult pornography to simulate CSAM. The organization previously reported that AI was used to create images of celebrities portrayed as children in sexual abuse contexts.

Other examples include AI-generated images where pictures of clothed children on the internet have been altered to appear nude. More than 50% of the AI-generated content flagged by the IWF in the past six months was hosted on servers located in Russia and the US, with Japan and the Netherlands also hosting notable amounts. The IWF compiles a list of URLs associated with this imagery, which is then shared with the technology industry to facilitate blocking and make it inaccessible. The IWF noted that 80% of reports concerning illegal AI-generated images came from members of the public who found them on public forums or AI galleries. In response to the issue of sextortion, where users are tricked into sharing intimate images with criminals impersonating young women, Instagram has introduced new measures. The platform will implement a feature that blurs nude images received in direct messages, advising users to be cautious before sending any DM that contains such content. When a blurred image is received, the user can decide whether to view it and will receive a reminder that they can block the sender or report the conversation to Instagram. This feature will be automatically enabled for teen accounts worldwide starting this week and can be applied to encrypted messages, although images flagged by the “on device detection” tool will not be automatically reported to the platform or law enforcement.