AI Model Performance on MBE Questions

How well do AI models answer the 210 NCBE sample Multistate Bar Exam (MBE) questions? This research was done at the University of Hawaiʻi, William S. Richardson School of Law. Read the full research paper.

Authors: Matthew Stubenberg, Chloe Berridge, Thomas Fuerst Smith, and Joshua Casey

🎉 These models answered all 210 MBE questions correctly (100%):
  • claude-fable-5 (first)
  • claude-opus-5
  • gpt-6.2-astra
  • gpt-6.2-sol
AI Platforms
Models All models are selected by default.

Percentage of questions each model answered correctly, colored by AI platform. The gray band between the dashed lines is the estimated pass range (58–67%); the solid dark line is the human average (70.9%).

Raw performance data

Number of correct answers by each model. The gray band between the dashed lines is the pass range (121–140); the solid dark lines mark the human average (149) and the maximum score (210).

Raw performance data

Sort models by:

Questions answered correctly by each model, broken down by legal category (30 questions per category).

Detailed category performance

Average cost per question (x) against percentage correct (y). Each point is a model, colored and shaped by AI platform. Higher and further left is better.

Cost and performance data

Average seconds to answer a question (x) against percentage correct (y). Higher and further left is better.

Time and performance data

Question performance analysis

Which questions were answered correctly by which models. “% Correct” is the share of the selected models that got the question right.

Easiest questions

Hardest questions

Copyright © Matthew Stubenberg