A recent report by the Anti-Defamation League (ADL) reveals biases against Jews and Israel in various large language models (LLMs), including OpenAI's GPT-4, Anthropic's Claude 3. 5, Google's Gemini 1. 5 Pro, and Meta's Llama 3-8B. The ADL's study involved evaluating 86 statements across multiple categories, including biases against Jews and Israel, conspiracy theories, and the Israel-Hamas conflict, with LLMs generating 34, 000 responses from 8, 600 prompts. The findings indicated that all LLMs exhibited measurable anti-Jewish and anti-Israel biases, with Meta’s Llama showing the most pronounced and "outright false" answers regarding Jews and Israel. ADL CEO Jonathan Greenblatt emphasized the potential for AI to amplify misinformation and contribute to antisemitism, calling for accountability and better safeguards from AI developers. When responding to questions about the Israel-Hamas war, GPT and Claude displayed significant biases, with a noted reluctance to answer Israel-related questions compared to others.

The ADL criticized the models for their failure to properly reject antisemitic tropes, noting that, aside from GPT, all LLMs were more biased towards Jewish conspiracy theories than those involving non-Jews, while generally showing more bias against Israel. In response, Meta argued that the ADL used an outdated version of their AI, asserting that current models provide different responses, especially for open-ended questions. Google echoed this sentiment, stating the version of Gemini tested was a developer model, not representative of consumer use. ADL representatives stressed the pressing need for companies to rectify these biases, suggesting collaborations with government and academic institutions for pre-deployment testing, adherence to the National Institute of Standards and Technology's (NIST) guidelines, and advocating for a regulatory framework to enhance AI safety.