AI Didn't Make You 10x Faster at Coding. Here's What the Data Actually Shows

If you’ve seen a number like “AI makes developers 10x faster” this year, it didn’t come from the most careful study anyone’s run on the question. That study, run by METR, found the opposite: experienced developers were measurably slower with AI tools, and then a year later the same research group’s follow-up was messy enough that they wouldn’t stand behind it either. The real story isn’t a clean multiplier in either direction. It’s that measuring this well is hard, and the most rigorous attempt anyone’s made keeps landing on “we’re not sure yet.”

What the first study actually found

In 2025, METR ran a randomized controlled trial with 16 experienced open-source developers, each with years of experience contributing to mature codebases, working through 246 real tasks pulled from actual repositories. For each task, the study randomly assigned whether the developer could use AI tools, then measured completion time.

Developers using AI (mostly Cursor Pro with Claude 3.5 or 3.7 Sonnet at the time) took 19% longer to finish tasks than when they worked without AI assistance. That’s not a rounding error. And the gap between reality and perception is the part that should give any team pause: after finishing, those same developers estimated that AI had made them about 20% faster. They were slower, and confident they’d been faster, by roughly 40 percentage points of misperception.

METR was careful about scope. The result doesn’t claim AI tools fail for everyone or in every domain, only that for this group, on real tasks in codebases they knew well, using the AI tooling available at the time, the net effect was a slowdown despite feeling like a speedup.

What changed by 2026, and why METR won’t vouch for it

METR didn’t stop there. By early 2026 they’d run a follow-up experiment, and the topline direction flipped: the data leaned toward developers now being meaningfully faster with AI, somewhere in the neighborhood of 18% faster rather than 19% slower. That’s a real, plausible shift, given how much agentic tooling like Claude Code and GitHub Copilot’s newer agent modes changed between the two studies.

Here’s the part that matters more than the headline number: METR itself said the follow-up couldn’t be trusted as a clean estimate. Developers grew reluctant to be randomly assigned to the no-AI condition, and some avoided submitting tasks to the study that they specifically wanted AI’s help on. Both effects bias the sample toward the people and tasks where AI looks best, which means the “18% faster” figure is closer to a hint than a finding. METR is reworking the study design rather than publishing it as a confident result.

Study	Period	Headline result	Reliability
METR RCT	2025	19% slower with AI	Clean randomized trial, well-controlled
Developer self-report (same study)	2025	Believed 20% faster	Perception, not measured
METR follow-up	Early 2026	~18% faster with AI	Flagged unreliable, selection bias

Source: METR: Early-2025 AI Developer Productivity Study, Rob Bowley: METR’s 2026 update

The honest reading of both studies together isn’t “AI is bad” or “AI got dramatically better.” It’s that the effect is small enough, and confounded enough by who’s motivated to use AI and on what, that measuring it precisely is a genuinely hard research problem, not a settled fact you can cite a single percentage for.

The ‘2x, not 10x’ argument that’s making the rounds

A July 2026 essay that spent time atop Hacker News takes a different angle on the same underlying question: not “how much faster are we today,” but “how much faster are we going to get.” The author’s argument is that LLMs crossed a specific threshold sometime in 2025, becoming reliable enough to run inside automated feedback loops (an agent that edits, runs tests, and retries without a human approving every step) rather than just autocompleting one suggestion at a time. That threshold crossing is what drove the visible jump in usefulness.

The claim that follows is about diminishing returns: once a model is reliable enough to function inside that loop, making it smarter still helps, but a lot less than crossing the threshold did in the first place. The essay’s own analogy is stairs: you need to be tall enough to climb one step at a time before height matters much, but being tall enough to take two or three steps at once doesn’t change your climbing speed nearly as much. Most of the future productivity gains, in this view, won’t come from a smarter model. They’ll come from the industry retooling workflows, review processes, and tooling around the capability that already exists.

That’s consistent with what GitLab’s 2026 AI Accountability survey found on a completely different axis: individual coding speed went up, but overall delivery didn’t keep pace because review and governance didn’t scale with it. Two different research efforts, two different methods, converging on the same practical point: the bottleneck isn’t purely the model’s raw capability anymore.

What this means for planning, not just curiosity

If you’re an engineering lead or an agency owner sizing a project timeline or a hiring plan around an assumed AI productivity multiplier, the data supports being conservative, not aggressive. A defensible planning assumption looks more like “somewhat faster on well-scoped, familiar work, with real variance by task type” than “half the team, twice the output.” If you’re evaluating vendors or contractors who lean on AI tooling as part of their pitch, ask for their own measured before-and-after cycle times on comparable work, not a vendor’s marketing multiplier. Teams we’ve worked with through custom software development engagements that track this rigorously tend to find gains concentrated in specific task types, boilerplate, test scaffolding, familiar refactors, and much smaller or even negative gains on unfamiliar, architecturally sensitive work, which lines up with what METR’s own data suggests once you look past the topline number.

The actual takeaway

Neither “AI barely helps” nor “AI makes you 10x faster” survives contact with the best data anyone has run on the question. What survives is a smaller, messier truth: the effect is real, it’s probably positive for most teams by 2026, it’s smaller than the marketing suggests, and it’s genuinely hard to measure precisely even for a research group built to do exactly that. Plan around your own measured numbers, not someone else’s multiplier.