On Thursday, the California Supreme Court pressed the State Bar of California to clarify how and why it employed artificial intelligence (AI) to draft multiple-choice questions for its problematic February bar exams. The Court revealed on Tuesday that it had not been informed prior to the exam that the State Bar had permitted its independent psychometrician to use AI in creating a minor portion of the questions. Increasing public pressure, the Court demanded detailed explanations about the AI's role in question development and the measures taken to ensure the questions’ reliability. This request coincides with the State Bar’s petition to adjust scores for hundreds of candidates who reported technical issues and irregularities during the February exam. The controversy extends beyond AI use itself, focusing on how the State Bar integrated AI into question development and the thoroughness of its vetting process for an exam pivotal to thousands aspiring to practice law in California each year. It also raises transparency concerns regarding the State Bar’s move away from the widely adopted National Conference of Bar Examiners’ Multistate Bar Examination towards a new hybrid in-person and remote testing format intended to reduce costs. In a Thursday statement, the Supreme Court sought specifics on why and how AI was employed to draft or revise multiple-choice questions, what steps ensured their reliability before administration, whether any AI-derived questions were excluded due to unreliability, and the overall dependability of the scored questions. Last year, the Court approved an $8. 25 million, five-year agreement with Kaplan to produce 200 test questions for the revamped exam, alongside hiring Meazure Learning to administer it. However, the State Bar disclosed only this week, nearly two months post-exam, that it deviated from this plan by not relying solely on Kaplan for question creation. A presentation revealed that out of 171 scored multiple-choice questions, Kaplan authored 100, 48 originated from a first-year law students’ test, and 23 were developed using AI by ACS Ventures, the Bar’s psychometrician. Despite this, Leah Wilson, the State Bar’s executive director, affirmed confidence in the questions’ validity and their fair assessment of candidates’ competence. Alex Chan, attorney and chair of the Committee of Bar Examiners overseeing the California Bar Examination, told The Times that AI was used only for a small set of questions and not necessarily to create them outright.

He noted that in October, the California Supreme Court had encouraged the State Bar to consider new technologies, including AI, to enhance testing reliability and cost-effectiveness, a process which he said would require Court approval. However, Chan revealed Thursday that the Committee was not informed about AI use before the exam and thus could not review or approve it. This lack of prior notification drew criticism from bar exam experts. Katie Moran, an associate law professor specializing in bar prep, questioned who instructed ACS Ventures—an entity without experience in bar question authorship—to create exam questions and what guidelines were provided. Mary Basick, assistant dean of academic skills at UC Irvine Law School, highlighted the significance of this unapproved change, emphasizing that the Committee and Supreme Court had only authorized Kaplan-drafted questions. She noted Kaplan’s expertise as a bar prep company, which was expected to ensure established question quality, and pointed out that substantial exam changes require a two-year notice under California law. Basick added that typical question development spans years to guarantee validity and reliability through multiple review stages, which was not feasible here. She and other academics also expressed concern over a non-legally trained psychometrician crafting questions with AI and judging their validity, suggesting a potential conflict of interest. The State Bar refuted this, stating the validation and reliability processes are objective and consistent regardless of question origin. It explained that prior to the exam, all questions—including those AI-assisted—underwent review by content validation panels and subject matter experts to assess legal accuracy, competence, and bias. Regarding reliability, the State Bar reported that the combined scored multiple-choice questions, irrespective of source, surpassed the psychometric reliability target of 0. 80, affirming performance standards were met despite the controversy.