Meta announced on Wednesday that it had discovered deceptive content on its Facebook and Instagram platforms, which it believes was likely generated by AI. These AI-generated comments were found below posts from global news organizations and U. S. lawmakers, praising Israel's handling of the Gaza war. The social media company revealed this information in its quarterly security report. The accounts behind this campaign posed as Jewish students, African Americans, and concerned citizens, targeting audiences in the United States and Canada. Meta attributed this campaign to STOIC, a political marketing firm based in Tel Aviv. STOIC has not yet commented on these allegations. This report is significant because it is the first time Meta has disclosed the use of more advanced generative AI technologies since they emerged in late 2022.

Concerns have been raised by researchers that such AI technologies, which can produce human-like text, imagery, and audio quickly and inexpensively, could be used in more effective disinformation campaigns and to influence elections. During a press call, Meta's security executives stated that they believe novel AI technologies have not hindered their ability to disrupt influence networks, which refers to coordinated attempts to push specific messages. They also mentioned that they have not come across AI-generated images of politicians that are realistic enough to be mistaken for authentic photos. The report highlights six covert influence operations that Meta disrupted in the first quarter, including the STOIC network. In addition, Meta shut down an Iran-based network focused on the Israel-Hamas conflict without identifying any use of generative AI in that particular campaign. Meta, along with other tech giants, has been grappling with how to address the potential misuse of new AI technologies, particularly during elections. Instances have been found where image generators from companies like OpenAI and Microsoft have produced photos containing voting-related disinformation, despite these companies having policies against such content. These tech companies have stressed the importance of digital labeling systems to identify AI-generated content at the time of creation. However, these tools are not effective for text and researchers have doubts about their overall effectiveness. Meta's defenses will face significant tests during upcoming elections in the European Union in early June and in the United States in November.