All postsLONG-FORM4 min read
Does AI Make Developers More Productive? The Evidence
What AI coding studies measure, where their findings stop, and what our agent-run video pipeline shows about checking and delivery.
My AI agents generate code every night, and I wanted to know what that tells me about the software people receive. The studies show task gains in some settings, historical slowdowns in another, and substantial uncertainty about how those findings translate into useful, reliable releases.
Choose the outcome before reading the percentage
The Students podcast episode published November 13, 2026 asks the question in its title: “Jonathan Blow — If AI Solved Coding, Why Is My Software Still the Same?” That question prompted this evidence review.
Imagine a password-reset change. An agent produces a patch, the team accepts it, and the person who requested help regains access. Each stage offers a different measure of success. A shorter drafting time leaves acceptance and the user's experience to establish.
METR's May 8, 2026 conceptual analysis distinguishes speed on existing tasks, speed on a changed task mix, and value gained. Its framework helps explain why a coding percentage needs an outcome attached to it.
METR's historical slowdown and uncertain follow-up
METR's July 10, 2025 experiment covered 16 experienced developers and 246 real issues in familiar open-source projects. Tasks averaged roughly two hours. Developers primarily used Cursor Pro with Claude 3.5 and 3.7 Sonnet.
With those early-2025 tools, completion time was 19% longer when AI was allowed. The reported confidence interval ran from +2% to +39% task time. Submitted pull requests had similar quality across conditions. Developers had expected 24% time savings and afterward believed they had saved 20%, illustrating how perceived help can diverge from measured time.
METR now explicitly calls that historical result out of date. Its February 24, 2026 follow-up describes an experiment begun in August 2025, covering 57 developers, 143 repositories and over 800 tasks.
Returning developers had an estimated task-time change of −18%, with a confidence interval from −38% to +9%. New participants had an estimate of −4%, with an interval from −15% to +9%. Both intervals include no change and some slowdown.
METR calls the newer signal unreliable. Between 30% and 50% of surveyed developers withheld some tasks because they wanted AI available. That can exclude work where they expect the greatest help. Participation changed, and concurrent agents complicated timing. These findings leave the magnitude of today's effect unresolved.
Completed-task gains in company experiments
Three randomized company experiments pooled 4,867 developers. The preferred estimate was 26.08% more completed tasks among tool users, with a standard error of 10.3 percentage points. The outcome was completed pull requests per week.
Researchers accounted for incomplete uptake when estimating the effect of using the tool. That estimate answers a different question from the effect of offering access to every engineer. Individual trial results were uncertain.
The trials began in 2022 and 2023 using GitHub Copilot code completion; journal publication followed in February 2026. This supports task-throughput gains in that setting. Product usefulness and reliability still require their own evidence.
Checking, review and later changes
In Stack Overflow's 2025 survey, 66% cited almost-right AI solutions as a frustration. In a separate question, 52% agreed AI tools or agents had helped their productivity. These are respondent reports; the frustration percentage measures neither elapsed task time nor an error rate.
Faros's analysis as of June 2025 associated high AI adoption with 21% more completed tasks, 98% more merged pull requests and 91% longer review time. This vendor telemetry largely concerned autocomplete. Without random assignment, it cannot establish that AI caused those differences.
DORA's March 10, 2026 analysis associated adoption with higher delivery throughput and greater delivery instability. It draws on 2025 observations across mixed tools, leaving causation unresolved.
Echoes of AI, revised February 26, 2026, detected no significant difference in the time or quality of subsequent changes during its randomized second phase. That narrow result leaves longer-term maintenance open; failure to detect a difference does not establish equivalence.
Our pipeline: a small operational example
Our November 14, 2026 snapshot covered nine earlier video-production runs. Every saved final script fact-check said PASS. Five runs were marked released in our production record.
The recorded failures included two render-related failures, a network delivery failure after the episode was completed, and a run that ended unfinished. A script check could pass while production still had work remaining.
Across 25 fact-check rounds, the AI checker recorded 50 required fixes, including wording and attribution changes. These entries were required revisions, with their correctness and uniqueness unverified here. In the five runs with recorded call logs, we counted 64 checking calls and 43 writing calls. Those categories cover broader production work than coding and code review.
This is our workflow's small descriptive sample, without a human-only comparison. It illustrates the distance between a checked draft and completed delivery.
My opinion: follow the change until someone can use it
I'd define the required behavior before asking an agent for a patch. For password reset, I'd require a valid link to work, an expired link to fail safely, and the person to regain access. I'd ask the agent to attach evidence for each requirement and identify anything still unchecked, then inspect that evidence myself.
I'd follow comparable changes from request through acceptance and release. Alongside the tool version, I'd record active human time, elapsed time to acceptance, review waits and rework. After release, I'd track defects and whether people got the behavior they needed.
If cheaper generation makes a previously unbuilt helper worth trying, I'd judge its usefulness through what people accomplish with it. That gives me a practical basis for deciding what to build next.
Daily email
Get each breakdown in your inbox.
One email each evening, right after the Short goes live: the AI tool it covers, the full write-up with sources, and how to get paid for it. Free, and you can unsubscribe anytime.
Sources
- The Students: "If AI Solved Coding, Why Is My Software Still the Same?"
- METR original study
- METR uplift update
- METR task substitution and uplift
- METR paper
- Cui et al., Microsoft Research
- Stack Overflow Developer Survey 2025: AI
- Faros AI report
- DORA: Balancing AI tensions
- Google DORA 2025 summary
- Echoes of AI
Our pipeline record is kept with the episode.
In your own work with AI, which step now takes the longest between asking for a change and getting something usable?