The State Bar of California has revealed that some multiple-choice questions in its problematic February bar exam were created using artificial intelligence. In a news release on Monday, the legal licensing authority announced it will request the California Supreme Court to adjust test scores for those who took the exam. Mary Basick, assistant dean of academic skills at UC Irvine School of Law, called the situation “worse than we imagined, ” telling the Los Angeles Times, “I’m almost speechless. Having the questions drafted by non-lawyers using artificial intelligence is just unbelievable. ” The new exam in February sparked complaints after many candidates failed to complete their tests.

The online platforms repeatedly crashed before some even began, and others struggled with finishing and saving essays, experienced screen lags and error messages, and were unable to copy and paste, the Times previously reported. A recent State Bar presentation showed that out of 171 scored multiple-choice questions, 100 were developed by Kaplan, 48 came from a first-year law student exam, and 23 were created by ACS Ventures, the State Bar’s psychometrician, with AI assistance. Leah Wilson, the State Bar’s executive director, told the newspaper, “We have confidence in the validity of the [multiple-choice questions] to accurately and fairly assess the legal competence of test-takers. ” Katie Moran, associate professor at the University of San Francisco School of Law and bar exam preparation expert, called the revelation “a staggering admission. ” She explained, “The State Bar has admitted they employed a company to have a non-lawyer use AI to draft questions given on the actual bar exam, ” and that they then paid the same company to evaluate and ultimately approve those questions. Andrew Perlman, dean of Suffolk University Law School and member of the American Bar Association taskforce on law and AI, said he had not heard of AI being used in developing bar exam questions or of any standards governing such use, but wasn't surprised due to the rapid rise of AI technology. He remarked that AI can be useful for crafting assessment questions but emphasized the importance of expert vetting of all AI-generated content. Perlman expects AI use in this area to grow. Despite current skepticism in the legal field about new technology, he predicted, “we will be worried in the future about the competence of lawyers who don’t use these tools. ”