Does AI make developers slower or faster? What the trials say so far
Nobody can say for sure yet. In a 2025 trial, 16 experienced developers took 19% longer with AI tools, while believing they had been 20% faster. The same group's 2026 follow-up points to a possible speedup, but the researchers call that data unreliable. A separate 2026 trial found that developers who used AI on a new library learned less of it. All three are preprints or blog posts, not peer-reviewed papers.
The three sources in one table
| Source | People | Main result | Status |
|---|---|---|---|
| Becker and others (METR), 2025 | 16 developers, 246 tasks | 19% slower with AI | Preprint |
| METR follow-up, February 2026 | 57 developers, more than 800 tasks | Possible speedup, called unreliable | Blog post |
| Shen and Tamkin, 2026 | 52 developers | Quiz score about 50% with AI, 67% without | Preprint |
The 2025 trial: slower, but it felt faster
The research group METR recruited 16 experienced open-source developers. Each worked on real tasks in code they had known for about five years. For each of the 246 tasks, it was decided at random whether AI tools were allowed.
- Before starting, the developers forecast that AI would make them 24% faster.
- Afterwards, they believed it had made them 20% faster.
- The clock said tasks with AI took 19% longer.
Outside experts had forecast speedups of 38% to 39%. The most useful finding here may not be the 19%. It is the gap between what people felt and what was measured.
The 2026 follow-up: maybe faster now, but hard to tell
In February 2026 METR reported a larger round: 57 developers and more than 800 tasks. For developers returning from the first study, the estimate was that AI made tasks 18% faster, with a range from 38% faster to 9% slower. For new developers it was 4% faster, with a range from 15% faster to 9% slower.
Both ranges include zero. And METR calls the result "an unreliable signal", because more developers chose not to take part, and 30% to 50% held back some tasks, as they did not want to work without AI. When people drop out of one side of a trial, the two sides stop being a fair comparison. METR says it is redesigning the study.
The learning trial: speed is not the only question
Shen and Tamkin, researchers at the AI company Anthropic, ran a randomised trial with 52 mostly junior engineers. All knew Python. None knew the library they were asked to use. They had 35 minutes for a coding task, with or without an AI assistant, and then a 25-minute quiz with no AI.
- Quiz scores. About 50% in the AI group against 67% in the group without, as Anthropic's summary gives it. The paper reports a gap of 4.15 points on a 27-point quiz.
- Time saved. About 2 minutes, which was not statistically significant.
- Biggest gap. Questions about debugging.
How people used the AI mattered. Those who asked it conceptual questions scored about 65%. Those who had it write code and then worked to understand that code scored about 86%. Those who handed the whole task over scored about 39%. These groups were very small, from 2 to 7 people each, and the authors say this part does not show cause.
What these studies did not show
- Not that AI makes developers slower today. The 19% figure is from 16 people using early-2025 tools on code they already knew well.
- Not that AI makes developers faster today. The follow-up's own authors do not trust it.
- Not long-term skill loss. The learning trial tested people right after one 35-minute task. It used a chat assistant, not tools that write and run code by themselves.
A fair one-line summary. "In a 2025 trial, experienced developers were 19% slower with AI but felt faster. Newer data is unclear, and a small 2026 trial found less learning when the AI did the work."
What to do with this
Do not trust the feeling of speed. Time a few tasks if it matters. When you are learning something new, write your own attempt first and ask the AI why, not just what. The routine for coding has the steps.
Questions
Is it true that AI makes developers 19% slower?
It was true for 16 experienced developers in one 2025 trial with the tools of that time. It is not a general fact about AI and coding now.
Why did developers feel faster when they were slower?
The trial measured the gap but cannot fully explain it. What it shows is that a feeling of speed is not a measurement.
Does using AI stop junior developers learning?
In one small trial, the AI group scored lower on a quiz straight afterwards. People who used the AI to understand, and not only to produce code, scored much closer to the group without AI.
Are these studies peer reviewed?
No. Two are preprints and one is a blog post, as of 13 November 2026.