What GSM8K Really Measures (And Why The Score Lies)

GSM8K measures whether a model can solve grade-school maths word problems that take two to eight steps. That's all it measures. A headline GSM8K score near 100% tells a board almost nothing about whether a model reasons reliably, because the benchmark is saturated, exposed to training-data contamination, and far narrower than the work you're buying the model to do. If a vendor leads with a reasoning-benchmark number, treat it as the first question, not the answer.

We sit on the assurance side of these conversations. When a board or a procurement lead asks us "is this model good at reasoning?", the honest answer starts with "good at what, measured how, and can we trust the number?" GSM8K is a useful case study in why a single benchmark score is a weak basis for a deployment decision.

What does the GSM8K benchmark actually measure?

GSM8K, short for Grade School Math 8K, is a set of 8,500 maths word problems built by OpenAI and released in 2021. It splits into 7,500 training problems and 1,000 test problems. Each problem needs two to eight steps of basic arithmetic to solve, dressed up in plain language.

Here's a representative one:

"A bakery sold 60 muffins on Saturday and 90 on Sunday, at GBP 2.50 each. How much did it take over the weekend?"

To get GBP 375, the model has to read the scenario, pull out the relevant numbers, add the muffins (60 + 90 = 150), then multiply (150 x 2.50). Simple for a person. The test checks whether a model can translate everyday language into the right sequence of arithmetic steps.

The original paper, Cobbe et al., "Training Verifiers to Solve Math Word Problems", built GSM8K to study exactly that: multi-step reasoning on problems a bright 11-year-old could handle. It was never meant to certify a model for financial modelling or scientific computation. It's a grade-school maths test. That framing matters, because the score gets quoted as if it proves general reasoning. It doesn't.

Why shouldn't a board trust a headline GSM8K score?

Three reasons, and any one of them should make you discount the number.

The benchmark is saturated. Frontier models now score in the high 90s on GSM8K. When everyone clusters near the ceiling, the test stops telling you who's better. A 99% and a 97% are noise apart, not signal. Independent trackers in 2026 describe GSM8K as effectively solved, which means it has lost most of its power to separate strong models from weak ones.

The score is likely contaminated. GSM8K has been public since 2021. Its problems, and answers, have circulated across the web for years, which means they've almost certainly leaked into model training data. A model that's seen the test isn't reasoning through it. It's recalling it. Scale AI built a fresh, contamination-free version called GSM1k and found models scored up to 13% worse on the new problems than on GSM8K, with the gap tracking how likely a model was to have memorised the original set. That's the signature of memorisation, not maths. The study, Zhang et al., "A Careful Examination of Large Language Model Performance on Grade School Arithmetic", is worth reading before you accept any GSM8K figure at face value.

The scope is narrow and the reasoning is brittle. Apple's research team built GSM-Symbolic, which generates fresh variants of GSM8K problems by changing names and numbers. Models that ace the original wobble on the variants. When the researchers added a single irrelevant clause that looked relevant but changed nothing, accuracy dropped by up to 65% across every model they tested. Their conclusion: the models are pattern-matching against training examples, not doing genuine logical reasoning. The paper, Mirzadeh et al., "GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models", accepted at ICLR 2025, is the clearest evidence that a high score and real reasoning are not the same thing.

So a near-perfect GSM8K number can mean the model reasons well, or that it memorised the test, or that it pattern-matches and falls over the moment the wording shifts. From the score alone, you can't tell which. For a board signing off on a deployment, that's not a basis for trust.

How does GSM8K relate to MATH and harder benchmarks?

GSM8K sits at the bottom of the difficulty ladder. The MATH dataset, from Hendrycks et al., "Measuring Mathematical Problem Solving With the MATH Dataset", is the next rung: 12,500 competition-level problems from contests like AMC and AIME, split 7,500 train and 5,000 test, across seven subjects and five difficulty levels. Harder, but it's now heading the same way GSM8K did. Independent reviews in 2026 describe MATH as mostly saturated too, with frontier models clearing the competition-maths subset at around 96%.

When a benchmark saturates, the field builds a harder one. The current frontier is sets like FrontierMath, built by Epoch AI with more than 70 mathematicians, using unpublished research-level problems under strict data-protection rules to keep them out of training data. Even the strongest models solve less than half of it. That gap, near-perfect on grade-school maths, under 50% on research maths, is the real story a single GSM8K number hides.

Benchmark	Difficulty	Frontier status (2026)	What a high score proves
GSM8K	Grade-school word problems	Saturated, ~99%	Can do multi-step arithmetic, if not memorised
MATH	Competition maths (AMC, AIME)	Mostly saturated, ~96%	Handles standard contest problems
FrontierMath	Research-level maths	Open, under 50%	Genuine advanced reasoning, hard to fake

The pattern repeats across every domain, not just maths. We've written about the same saturation-and-contamination cycle in code benchmarks in our HumanEval and MBPP code-generation guide, and about the broader gap between benchmark scores and reasoning in our AGI-Eval benchmark explainer.

What should an assurance or procurement leader do instead?

Don't ban benchmarks. Use them as one weak signal among several, and put the burden of proof on the vendor.

• Ask when the benchmark was released and how old it is. A 2021 test scored by a 2026 model has had five years to leak into training data. Treat old public benchmarks as contaminated until proven otherwise.
• Ask for performance on contamination-controlled or held-out sets. GSM1k-style results, or private evaluations the vendor can't have trained on, are worth far more than a GSM8K figure.
• Ask for the variance, not just the headline. GSM-Symbolic showed accuracy swings when you change names and numbers. A vendor who can show stable performance across variants is showing you something real.
• Test on your own tasks. The only benchmark that maps to your risk is one built from your actual work. Build a small held-out set of real problems from your domain and measure against that.
• Match the benchmark to the use case. Grade-school arithmetic accuracy says nothing about whether a model is safe for derivative pricing or clinical calculation. If a model will touch high-stakes maths, it needs domain evidence, human review thresholds, and a second-check on every material calculation, regardless of its GSM8K score.

This is the same discipline we apply across AI assurance work. We've laid out the wider case for independent checks in our note on the validation gap and why independent assessment matters, and the framework for telling a genuine capability gain from marketing in our breakthrough-versus-hype validation guide.

Frequently asked questions

What does GSM8K stand for?

Grade School Math 8K. It's a benchmark of 8,500 grade-school maths word problems built by OpenAI and published in 2021, used to test whether a model can solve multi-step arithmetic problems written in plain language.

Is a high GSM8K score good?

It's a weak positive signal at best. Frontier models now score near 100%, so the benchmark no longer separates strong models from weak ones. The score is also likely inflated by training-data contamination, and research shows the underlying reasoning is brittle. Treat a high score as a starting question for due diligence, not proof of reasoning ability.

Why is GSM8K considered contaminated?

It's been public since 2021, so its problems and answers have circulated online for years and almost certainly leaked into model training data. When Scale AI tested models on a fresh, equivalent set called GSM1k, scores dropped by up to 13%, and the drop tracked how likely each model was to have memorised the original. That points to recall rather than reasoning.

What benchmarks are harder than GSM8K?

MATH covers competition-level problems and is now mostly saturated too. Sets like FrontierMath use unpublished research-level maths kept out of training data, and even the best models solve under half of it. For high-stakes deployment, the most useful test is a private, held-out set built from your own tasks.

The bottom line

GSM8K is a grade-school maths test that frontier models have outgrown. The number you see quoted is saturated, probably contaminated, and brittle under small changes to the wording. My view: any reasoning-benchmark score a vendor leads with should lower your trust until they show you contamination-controlled results, variance across problem variants, and performance on tasks that look like yours. A benchmark is a marketing claim until someone independent checks it. For boards and procurement leaders, that check is the whole job.

Sources: Cobbe et al. 2021 (GSM8K); Hendrycks et al. 2021 (MATH); Zhang et al. 2024 (GSM1k contamination); Mirzadeh et al. 2024 (GSM-Symbolic); Epoch AI (FrontierMath).

For hands-on help, see VerityAI's AI governance.