Read an AI headline with the conditions attached
Different results can describe different work.
One study says AI improves productivity. Another says it slows people down.
Before choosing which headline to believe, ask what each study actually measured.
Let's put two research settings beside each other. In the November 2024 version of Generative AI at Work, researchers studied the staggered introduction of an assistant among 5,172 customer-support agents. They reported an average increase of about 15% in issues resolved per hour, with substantial differences across workers.
In METR's early-2025 experiment, 16 experienced developers worked on 246 issues in familiar open-source repositories. Tasks assigned to allow AI took 19% longer on average. That finding concerns those developers, tasks and tools, not every form of programming.
Those percentages are not opposite readings from the same instrument.
Ask what sits under the number
The support study measured completed issues per hour in an ongoing workplace rollout. The coding experiment measured time on assigned development tasks. One outcome is a rate; the other is a duration. More work per hour and less time per task can be related, but their percentage changes are not interchangeable.
As a simple arithmetic example, reducing a task from 60 minutes to 48 minutes is a 20% time reduction. At that pace, the hourly rate rises from one task to 1.25 tasks—a 25% increase. This is why “20% more productive” needs a definition.
The research designs differ too. The support paper analyzes rollout timing and uses statistical controls; its causal interpretation depends on the design's assumptions. METR randomized AI access at the task level. Randomization helps with causal comparison within that experiment. It doesn't make the selected participants representative of everyone who might use the tool.
Keep the date attached
METR's February 2026 update reported that later productivity estimates were difficult to interpret because of selection and measurement problems. It did not provide a clean, reliable new effect size to substitute for the earlier result.
That is a reason to preserve the date and conditions of a finding. An older result remains evidence about its setting. It doesn't automatically describe today's tools.
Our own working-arrangement comparison is much smaller and less controlled than either study. We could inspect useful handoffs and actual corrections. We couldn't establish an efficiency winner from uneven inputs and missing effort measurements. The same discipline applies at different scales: keep the conclusion within what the comparison can support.
When a headline catches your attention, look for five things: the task, the people, the tool and period, the comparison, and the outcome. If you can't find them, you don't yet know what the percentage means for your work.
You can still learn from a study that differs from your situation. It may suggest a task to try or a failure to watch for. Bring that suggestion into a fair local test instead of carrying the headline into your workflow as a promise.