An artificial intelligence pioneer has launched a non-profit organization dedicated to creating an “honest” AI designed to detect rogue systems attempting to deceive humans. Yoshua Bengio, a distinguished computer scientist often called one of the “godfathers” of AI, will serve as president of LawZero, a group focused on the safe development of advanced technology that has ignited a $1 trillion (£740 billion) arms race. With initial funding of about $30 million and a team of over a dozen researchers, Bengio is working on a system named Scientist AI. This system is intended to act as a safeguard against AI agents—autonomous systems that perform tasks without human involvement—that might exhibit deceptive or self-preserving behavior, such as resisting being shut down. Bengio described current AI agents as “actors” aiming to imitate humans and satisfy users, while he envisions Scientist AI as more akin to a “psychologist” capable of understanding and predicting harmful behavior. “We want to build AIs that will be honest and not deceptive, ” Bengio stated. He added: “It is theoretically possible to imagine machines without a self or personal goals, functioning purely as knowledge holders—like a scientist who possesses extensive information. ” Unlike current generative AI tools, Bengio’s system will not provide definitive answers but instead will offer probabilities indicating the likelihood that a response is correct. “It has humility, acknowledging uncertainty about its answers, ” he explained. When used alongside an AI agent, Bengio’s model would identify potentially harmful behavior by an autonomous system by assessing the probability that its actions might cause harm. Scientist AI is designed to “predict the probability that an agent’s actions will lead to harm, ” and if that probability surpasses a certain threshold, it will block the proposed action. LawZero’s initial supporters include the AI safety organization Future of Life Institute, Jaan Tallinn—a founding engineer of Skype—and Schmidt Sciences, a research entity started by former Google CEO Eric Schmidt. Bengio emphasized that LawZero’s first objective is to prove the concept’s methodology works, then to convince companies or governments to back larger, more powerful implementations.

He noted that open-source AI models, which are freely available for use and modification, will be the foundation for training LawZero’s systems. “The goal is to validate the methodology so we can persuade donors, governments, or AI labs to invest the necessary resources to train this at the same scale as today’s leading AI systems. It’s crucial that the guardrail AI be at least as intelligent as the AI agent it aims to monitor and regulate, ” he said. Bengio, a professor at the University of Montreal, earned the “godfather” nickname after sharing the 2018 Turing Award—considered the computing equivalent of a Nobel Prize—with Geoffrey Hinton, himself later a Nobel laureate, and Yann LeCun, Meta’s chief AI scientist. As a prominent advocate for AI safety, he chaired the recent International AI Safety report, which cautioned that autonomous agents could cause “severe” disruptions if they become capable of executing extended sequences of tasks without human oversight.