MMLU Benchmark: Why Boards Shouldn't Trust the Score

MMLU measures whether a model can answer multiple-choice questions across 57 academic subjects, from elementary maths to professional law and medicine. That's all it measures. A headline MMLU score in the high 80s tells a board very little about whether a model is safe to deploy, because the benchmark is saturated near its ceiling, riddled with label errors, and almost certainly contaminated by training data. If a vendor leads with an MMLU number, treat it as the first question, not the answer.

We sit on the assurance side of these conversations. When a board or a procurement lead asks us "how much does this model actually know?", the honest answer starts with "measured how, on which questions, and can we trust the number?" MMLU is the most quoted knowledge benchmark in the industry, which makes it the most useful case study in why a single score is a weak basis for a deployment decision.

What does the MMLU benchmark actually measure?

MMLU, short for Massive Multitask Language Understanding, is a set of 15,908 multiple-choice questions spanning 57 subjects. It was built by Dan Hendrycks and colleagues and released in September 2020, in the paper Hendrycks et al., "Measuring Massive Multitask Language Understanding". The subjects run from elementary mathematics and US history through to computer science, professional law, clinical medicine, and moral reasoning. Each question gives four answer options, and the model picks one.

The point of the design was breadth. Instead of testing one narrow skill, MMLU samples knowledge a model would only have if it had absorbed a wide slice of human education. Questions are drawn from real exams, textbooks, and practice tests, and models are usually scored in a few-shot setting, where they see a handful of worked examples before answering.

When it launched, MMLU was genuinely hard. The strongest models of 2020 scored barely above random guessing. That's exactly why it became the default. A benchmark earns its place by separating strong models from weak ones. The trouble is what happens next.

Why shouldn't a board trust a headline MMLU score?

Three reasons, and any one of them should make you discount the number.

The benchmark is saturated. By mid-2024, leading models were scoring around 88% on MMLU, close enough to the practical ceiling that the test had stopped doing its job. When everyone clusters near the top, an 88% and an 86% are noise apart, not signal. By 2025 the field had partially phased MMLU out in favour of harder successors, which is the clearest possible admission that the original number no longer separates the models you're choosing between.

The score is likely contaminated. MMLU has been public since 2020. Its questions and answer keys have circulated across the web for years, which means they've almost certainly leaked into model training data. A model that's seen the test isn't reasoning through it. It's recalling it. Independent contamination studies put meaningful chunks of the MMLU suite inside model training sets, and the symptom is consistent: scores fall when models face fresh questions they couldn't have memorised. Microsoft built a contamination-free version, MMLU-CF (accepted at ACL 2025), precisely because, in their words, "the open-source nature of these benchmarks ... has inevitably led to benchmark contamination, resulting in unreliable evaluation results." They kept the test set closed so it can't leak.

The questions themselves are wrong more often than you'd think. This is the part most leaders miss. A team re-annotated MMLU with domain experts and found that 6.49% of questions contain errors, including multiple correct answers, unclear wording, or a flatly incorrect answer key. In the Virology subset, 57% of the questions they checked were faulty. The study, Gema et al., "Are We Done with MMLU?", accepted at NAACL 2025, produced a cleaned subset called MMLU-Redux from 5,700 hand-checked questions, and showed that fixing the labels changes model rankings. Think about what that means. A model can be marked wrong for giving the right answer, or marked right for matching a broken key. The leaderboard you're reading is built partly on noise.

So a strong MMLU number can mean the model is broadly knowledgeable, or that it memorised a benchmark that's been public for six years, or that it happened to match faulty answer keys. From the score alone, you can't tell which. For a board signing off on a deployment, that's not a basis for trust.

What replaced MMLU, and why does it matter?

When a benchmark saturates, the field builds a harder, cleaner one. Three successors are worth knowing by name, because a serious vendor should be quoting these, not the 2020 original.

Benchmark	What changed	Why it exists
MMLU-Pro	10 answer options instead of 4, more reasoning-heavy questions, 12,000+ items	Original MMLU saturated; harder set to re-separate models
MMLU-Redux	5,700 questions re-checked and corrected by experts	Original MMLU has a ~6.5% label-error rate
MMLU-CF	20,000 questions, half kept closed-source	Original MMLU is contaminated; closed test set blocks leakage

MMLU-Pro (Wang et al., NeurIPS 2024) expanded each question from four options to ten and added more reasoning-focused items. The effect was stark: model accuracy dropped by 16% to 33% compared with the original MMLU. Same models, much lower scores, because the easy memorisation route was harder to take. That gap, comfortable on the old test, exposed on the harder one, is the real story a single MMLU number hides.

The pattern repeats across every domain, not just general knowledge. We've written about the same saturation-and-contamination cycle in maths benchmarks in our GSM8K and mathematical-reasoning guide, and in code benchmarks in our HumanEval and MBPP code-generation guide. Different skill, identical failure mode.

What should an assurance or procurement leader do instead?

Don't ban benchmarks. Use them as one weak signal among several, and put the burden of proof on the vendor.

• Ask which version they're quoting, and how old it is. A 2020 MMLU score from a 2026 model has had six years to leak into training data. If a vendor is still leading with original MMLU rather than MMLU-Pro, MMLU-Redux, or a contamination-free set, ask why.
• Ask for performance on contamination-controlled or held-out sets. Results on closed test sets the vendor can't have trained on are worth far more than a public benchmark figure.
• Ask for subject-level breakdowns, not the aggregate. An 88% average can hide a model that's strong on US history and weak on the exact regulated domain you care about. The aggregate is the least useful number on the page.
• Discount the domains you can't verify. If the benchmark itself has a 57% error rate in a subject like Virology, a high score there is meaningless. Treat any high-stakes domain score as unverified until you've checked the questions.
• Test on your own tasks. The only benchmark that maps to your risk is one built from your actual work. Build a small held-out set of real problems from your domain and measure against that, with human review thresholds for anything material.

This is the same discipline we apply across AI assurance work. A benchmark is a marketing claim until someone independent checks it.

Frequently asked questions

What does MMLU stand for?

Massive Multitask Language Understanding. It's a benchmark of 15,908 multiple-choice questions across 57 academic subjects, built by Dan Hendrycks and colleagues and released in 2020, used to measure how broadly a language model has absorbed human knowledge.

Is a high MMLU score good?

It's a weak positive signal at best. Leading models reached around 88% by 2024, so the benchmark no longer separates strong models from weak ones. The score is also likely inflated by training-data contamination, and the underlying questions have a measurable error rate. Treat a high score as a starting question for due diligence, not proof of capability.

Why is MMLU considered unreliable now?

Three reasons. It's saturated, with top models clustered near the ceiling. It's contaminated, because it's been public since 2020 and has leaked into training data. And it has a roughly 6.5% label-error rate, rising to 57% in some subjects, which means some questions are marked against broken answer keys. The field has built MMLU-Pro, MMLU-Redux, and MMLU-CF to address each problem.

What is the difference between MMLU and MMLU-Pro?

MMLU-Pro is a harder, cleaner successor. It expands each question from four answer options to ten and adds more reasoning-heavy items across 12,000+ questions. The same models score 16% to 33% lower on MMLU-Pro than on the original, which makes it far better at separating models near the top.

The bottom line

MMLU is a six-year-old multiple-choice quiz that frontier models have outgrown. The number you see quoted is saturated, probably contaminated, and built partly on faulty answer keys. My view: any general-knowledge benchmark score a vendor leads with should lower your trust until they show you results on a harder, contamination-controlled successor, subject-level breakdowns for the domains you actually care about, and performance on tasks that look like yours. A benchmark is a marketing claim until someone independent checks it. For boards and procurement leaders, that check is the whole job.

Sources: Hendrycks et al. 2020 (MMLU); Wang et al. 2024 (MMLU-Pro); Gema et al. 2024 (Are We Done with MMLU? / MMLU-Redux); Microsoft MMLU-CF (ACL 2025).

For hands-on help, see VerityAI's AI governance.