HumanEval and MBPP: What a Code Benchmark Won't Tell You

A high HumanEval or MBPP score tells you a model can write small, self-contained Python functions. It tells you almost nothing about whether that model can ship safe, maintainable code in your codebase. Both benchmarks are now saturated at the frontier, both are contaminated by training data, and both test a narrow slice of what real engineering demands. If you're assessing or buying an AI coding tool and a vendor leads with a benchmark number, that number is the start of due diligence, not the end of it.

This is the explainer plus the assurance lens: what HumanEval and MBPP actually measure, what they don't, and how to read a code-benchmark claim without being misled.

What is the HumanEval benchmark?

HumanEval is a set of 164 hand-written Python programming problems, released by OpenAI in 2021 alongside the Codex model (Chen et al., 2021). Each problem gives the model a function signature and a docstring describing what the function should do. The model writes the body. A set of hidden unit tests then checks whether the code actually works.

The problems cover language comprehension, basic algorithms, and simple maths. Think string manipulation, list operations, small numeric routines. The kind of thing a junior developer handles in a few minutes.

The headline metric is pass@k. The model generates k attempts per problem, and the problem counts as solved if any one attempt passes every test. Pass@1 is the strict version: one shot, must be correct. Pass@10 or pass@100 measure whether a correct answer appears anywhere in a batch of tries, which flatters the model because real teams don't generate a hundred drafts and pick the right one by magic.

For context, the original Codex model in the 2021 paper solved 28.8% of HumanEval problems at pass@1, rising to 70.2% with 100 samples per problem. Frontier models today score far higher, which is exactly the problem.

What is the MBPP benchmark?

MBPP stands for Mostly Basic Programming Problems. Google Research released it in 2021 (Austin et al., 2021), and the name is honest about the scope. It contains 974 crowd-sourced Python tasks designed to be solvable by entry-level programmers. Each task has a short natural-language description, a reference solution, and three test cases.

Where HumanEval gives the model a function signature to fill in, MBPP gives it a plain-English instruction ("write a function to find the shared elements from two lists") and asks it to produce the whole function. Same idea, slightly different framing, broader coverage of everyday utility code.

The original largest model in the MBPP paper solved 59.6% of problems using few-shot prompting. As with HumanEval, scoring is execution-based: the code runs against the tests, and there's no partial credit.

Feature	HumanEval	MBPP
Released	2021 (OpenAI)	2021 (Google)
Problems	164	974
Source	Hand-written	Crowd-sourced
Prompt style	Function signature + docstring	Plain-English instruction
Tests per problem	Several hidden	3
Difficulty	Junior-level	Entry-level
Metric	pass@k	pass@k

The two are usually quoted together because they measure the same thing at roughly the same level: can a model turn a small, well-specified request into a working Python function?

Why shouldn't I trust a headline HumanEval or MBPP score?

Three reasons. Each one on its own would make me cautious. Together they mean a raw score is close to meaningless for procurement.

The benchmarks are saturated

When a benchmark is new, the spread between models is wide and informative. When the best models all cluster near the ceiling, the benchmark stops discriminating. That's where HumanEval and MBPP are now. Frontier models score in the 90s on both, and the gaps between leading models are within noise. A 2% difference on a saturated benchmark doesn't mean one tool is 2% better at your work. It might mean nothing at all.

The benchmarks are contaminated

HumanEval and MBPP have been public on the internet for years. Their problems and solutions sit in the training data of modern models. Research measuring this directly has found meaningful overlap between these benchmark test sets and model training corpora (Riddell et al., 2024). When a model has effectively seen the answer key during training, a high score measures memorisation, not problem-solving. You can't tell the two apart from the number alone.

The tests are weak

Here's the one that surprised me. The original test suites are thin, often fewer than ten cases per problem, so buggy code slips through. The EvalPlus project rebuilt HumanEval and MBPP with roughly 80 times more test inputs (Liu et al., NeurIPS 2023). Under the tougher tests, top models lost 19% to 29% of their apparent pass rate. The same work found that about 11% of HumanEval's own reference solutions were defective. So part of the "correct" answers the benchmark rewarded were themselves wrong. When you hear a HumanEval number, ask whether it's the original or the HumanEval+ version. They are not the same claim.

What do these benchmarks not measure at all?

This matters more than the score. HumanEval and MBPP test isolated functions. Real engineering is almost none of that.

• Multi-file work. Neither benchmark touches navigating a large codebase, understanding existing architecture, or changing code across many files.
• Security. No checks for injection flaws, unsafe input handling, secrets in code, or vulnerable dependencies. A model can ace HumanEval and still write insecure code.
• Maintainability. Readability, naming, documentation, whether the next engineer can work with the output. Not measured.
• Novel problems. By design these are common, well-trodden tasks. They say little about genuinely new logic.
• Edge cases. The thin original tests reward code that works on the happy path and falls over on awkward inputs.

So a model with a 95% HumanEval score could still produce code that's insecure, unmaintainable, and wrong on the cases that matter. The benchmark wasn't built to catch any of that.

Which benchmarks have superseded HumanEval and MBPP?

The field moved on precisely because of saturation and contamination. Two newer benchmarks are far more useful when you're assessing a coding tool in 2026.

• SWE-bench and its human-checked subset SWE-bench Verified test models on real GitHub issues from open-source projects (Jimenez et al., ICLR 2024). The model must produce a patch that fixes a genuine bug across a multi-file repository, verified by the project's own tests. SWE-bench Verified is a 500-problem set that human engineers confirmed are solvable and well-specified. Far closer to real engineering than filling in one function.
• LiveCodeBench tackles contamination by continuously collecting fresh problems from coding competitions and tagging them by date (Jain et al., 2024). Because the problems are new, a model can't have memorised them, so scores reflect reasoning, not recall.

Neither is perfect. SWE-bench has its own quality and contamination debates, and any benchmark eventually leaks into training data. But both test what HumanEval and MBPP can't: working with real code over time.

How should a leader read a code-benchmark claim?

Treat the number as a marketing artefact and interrogate it. A short checklist for procurement or internal evaluation:

1. Which benchmark, which version? Original HumanEval or HumanEval+? Pass@1 or pass@100? Each changes what the figure means.
2. Is it saturated? If everyone scores in the 90s, the number can't separate good from great. Ask for SWE-bench Verified results.
3. Could it be contamination? A near-perfect score on an old public benchmark is as consistent with memorisation as with skill. Live benchmarks are your tell.
4. Does it match your work? If your engineers maintain a large codebase, a single-function benchmark is the wrong proxy.
5. Where's the security evidence? No public code benchmark covers this well. Ask separately: how does the tool handle vulnerable code, and what's your review layer around it?

Benchmarks tell you a model can do a thing under lab conditions. They never tell you it will do the right thing in your environment, on your code, against your risk tolerance. That gap is where assurance lives.

Frequently asked questions

Is a 90% HumanEval score good?

It's normal, not impressive. Frontier models now cluster in the 90s, so a high score is table stakes rather than a differentiator. Worse, scores on the original benchmark are inflated by weak tests and possible training contamination. A 90% on the original is not the same as 90% on the harder HumanEval+. Always ask which version and which metric.

What's the difference between HumanEval and MBPP?

Both measure whether a model can write small Python functions that pass tests, at a junior to entry level. HumanEval has 164 hand-written problems and gives the model a function signature to complete. MBPP has 974 crowd-sourced problems and gives a plain-English instruction. They're quoted together because they probe the same narrow skill.

Are HumanEval and MBPP still used in 2026?

Yes, but mostly out of habit and for comparability with older results. For meaningful comparison of current models, researchers and serious buyers have shifted to SWE-bench Verified and LiveCodeBench, which resist saturation and contamination and test more realistic engineering. Treat HumanEval and MBPP as historical baselines, not decision-grade evidence.

Can a model pass these benchmarks and still write bad code?

Easily. The benchmarks don't check security, maintainability, multi-file reasoning, or behaviour on novel problems. A model can score near-perfectly and still produce insecure, hard-to-maintain code that breaks on edge cases. That's the core reason a benchmark score can't replace a proper assurance process and human review.

The bottom line

Here's my view: leading with a HumanEval or MBPP number in 2026 is a sign a vendor is either behind the research or hoping you are. The benchmarks did important work, they made functional correctness measurable when match-based scoring was all the field had, and they earned their place in history. But they answer a small question, "can this model write a basic Python function," and the answer for every frontier model is now "yes." That's not the question a board, a buyer, or an engineering leader needs answered.

The real questions are whether the tool writes secure code, whether it works in your actual codebase, and whether your review and testing layer catches what it gets wrong. No public code benchmark answers those for you. Use SWE-bench Verified and LiveCodeBench for sharper model comparison, then build the assurance around the tool, because that's the part the leaderboard will never measure.

For the related view on quantitative reasoning, see our explainer on the GSM8K and MATH benchmarks. For why lab scores diverge from production, see beyond benchmarks: real-world AI testing failures.

More on how we approach it: AI-ready web development.