Researchers at Carnegie Mellon University recently conducted a study revealing a vulnerability in advanced AI chatbots including ChatGPT, Google's Bard, and Claude from Anthropic. By adding a specific string of information to the end of a prompt, the attack forces chatbots to generate disallowed and harmful responses. Unfortunately, this vulnerability cannot be easily patched, according to Zico Kolter, an associate professor involved in the study.

OpenAI, Google, and Anthropic were alerted to the exploit prior to the research's release, and although blocks were introduced to prevent the specific exploits described in the study, the companies have not yet found a way to block adversarial attacks more broadly. The study underlines the difficulty of deploying advanced AI and the need for more robust security measures.