How well do AI models answer the 210 NCBE sample Multistate Bar Exam (MBE) questions? This research was done at the University of Hawaiʻi, William S. Richardson School of Law. Read the full research paper.
Authors: Matthew Stubenberg, Chloe Berridge, Thomas Fuerst Smith, and Joshua Casey
Percentage of questions each model answered correctly, colored by AI platform. The gray band between the dashed lines is the estimated pass range (58–67%); the solid dark line is the human average (70.9%).
Number of correct answers by each model. The gray band between the dashed lines is the pass range (121–140); the solid dark lines mark the human average (149) and the maximum score (210).
Questions answered correctly by each model, broken down by legal category (30 questions per category).
Average cost per question (x) against percentage correct (y). Each point is a model, colored and shaped by AI platform. Higher and further left is better.
Average seconds to answer a question (x) against percentage correct (y). Higher and further left is better.
Which questions were answered correctly by which models. “% Correct” is the share of the selected models that got the question right.