AI Colleague Performance Metrics Data: What 40+ Studies Actually Show

The most important number in this report is a contradiction

In July 2025, researchers at METR ran a randomized controlled trial with 16 experienced open-source developers working on their own mature repositories -- real code, real architecture, real deadlines. Before starting, those developers predicted AI would make them about 24% faster. The measured result was the opposite: they were 19% slower.

Two years earlier, GitHub's controlled study of 95 developers found that Copilot users finished a standardized HTTP server task 55.8% faster than a control group working on the same task.

Both studies are real. Both are competently run. And the space between them is the entire story of AI colleague performance metrics in 2026: your AI teammate's productivity is not a property of the AI. It is a property of the task, the tooling, the measurement window, and how much of your working context you are willing to hand over.

If you have ever been handed a dashboard showing "3.4x productivity lift" and quietly wondered why your week felt longer, this report is for you. Everything below is drawn from published randomized trials, vendor telemetry, large-scale developer surveys, and independent enterprise research. Sources are linked so you can check the math yourself.

Key Findings

• 55.8% faster on a bounded, self-contained coding task with no legacy context to load (GitHub / Peng et al., 2023, n=95).
• 19% slower on real tasks inside mature repositories where developers know the codebase deeply (METR, 2025, n=16).
• 6.5 percentage points more tasks completed, 25.1% faster, and 40% higher quality on consulting-style knowledge work -- but only for tasks inside the AI's capability frontier, and consultants using AI were 19 percentage points more likely to produce wrong answers on tasks outside it (BCG / Dell'Acqua et al., 2023, n=758).
• 1.5% lower delivery throughput and 7.2% lower delivery stability at the organizational level, in the same year organizations increased AI adoption by 25% (DORA / Google Cloud, 2024).
• 95% of enterprise GenAI pilots produce no measurable P&L impact, per MIT's NANDA research group -- while the same executives report being satisfied with the tools.
• 66% of developers say they are frustrated by "AI solutions that are almost right, but not quite" -- up sharply from prior years (Stack Overflow Developer Survey, 2025).
• Over 40% of agentic AI projects will be cancelled by the end of 2027, according to Gartner -- not because the models are weak, but because cost, risk, and unclear value become unmanageable.

Notice the pattern. Every headline number in the AI productivity conversation is real. They just describe different tasks, different populations, and different time horizons. The job of a serious operator in 2026 is not to pick a side -- it is to know which cell of the matrix you are standing in.

Data Section 1: Throughput -- Where AI Colleagues Actually Move the Needle

Throughput is the easiest AI colleague metric to measure and the easiest to misread. The table below pulls together the five most-cited controlled measurements of AI-assisted output. Read it left to right, then look at what changes between rows: task boundedness, context dependency, and whether the human already knows the answer.

The through-line here is what researchers call task boundedness. A greenfield HTTP server has a crisp definition of done, no institutional memory to violate, and a feedback loop the developer can close in seconds. A bug fix in a nine-year-old repository with undocumented coupling is the opposite: the AI colleague generates plausible code that a senior engineer must fully evaluate before trusting, and that evaluation cost is invisible in most dashboards.

DORA's organizational finding is the tell. Individual tasks got faster in 2024 while system-level throughput got marginally worse and stability dropped 7.2%. You can ship faster per task and still deliver slower per quarter if every task now carries an extra verification step that never shows up as a line item. That is why throughput alone is a vanity metric for AI colleagues. It needs a denominator.

Data Section 2: Reliability -- The Metric That Never Makes the Slide

Reliability data is where the picture gets uncomfortable, because the numbers come from incidents, not demos. The ranges below are compiled from vendor telemetry where published, third-party benchmarks, and independent enterprise research. Where a vendor does not disclose raw numbers, the range reflects documented third-party evaluations rather than marketing claims.

Read the reliability table as a series of leading indicators. Benchmark success rates are climbing fast -- that is the good news, and it is genuine. But human distrust is climbing at the same time, from 31% to 46% in a single year. Those two curves are not in conflict. They are exactly what you would expect when a tool becomes competent enough to be deployed into high-stakes work but not competent enough to be left unsupervised.

The 66% "almost right" figure is the operational heart of this report. Almost-right answers are more expensive than obviously wrong ones, because obviously wrong answers trigger rejection in seconds while almost-right answers trigger a full review cycle. If your AI colleague produces work that looks 90% correct, your senior people will spend their scarcest resource -- attention -- verifying the other 10%. That is a real cost, and it belongs in your metrics.

Data Section 3: Adoption Outran Trust -- and the Gap Is Still Widening

The third dataset is the one most teams never look at, because it is about the humans, not the model. Developer and knowledge-worker surveys show adoption rising in a straight line while trust flattens and then falls.

Two things stand out. First, adoption is no longer a leading indicator -- it is table stakes. When 84% of a profession reports using the tool, using it tells you nothing about whether it is working. Second, the trust decline is not Luddism. It is calibration. Practitioners who use these tools daily are the fastest to notice the 10% that is wrong, and their skepticism is the most valuable signal in the entire dataset.

What The Data Tells Us

One: the productivity number is meaningless without a task classification. Every credible study that separates tasks by boundedness finds wildly different results. The METR developers were not worse at their jobs than the GitHub study participants; they were working on harder, more entangled problems where the AI's suggestions required deep verification. Build your AI colleague scorecard per task type, not per tool. A single blended "productivity lift" figure is a number designed to survive a board meeting, not a Tuesday afternoon.

Two: verification cost is the hidden line item, and it is the one that flips the math negative. The DORA finding -- faster tasks, slower delivery -- only makes sense if you assume a new cost appeared somewhere. It did. It is the review cycle that almost-right output creates. Measure it explicitly: track minutes spent reviewing AI output as a distinct category. Teams that do this almost always discover their real lift is somewhere between a third and a half of what the tool dashboard claims.

Three: reliability improves faster than trust, and that gap is your window. Benchmark success rates roughly climbed 25 points in eighteen months. Human trust fell 15. That asymmetry means the winning operators in 2026 are not the biggest adopters -- they are the ones with the discipline to re-test a capability that failed for them six months ago. If you wrote off an AI colleague last year because it botched your specific workflow, that is a stale conclusion.

Four: augmentation, not replacement, is still where 57% of real usage sits. The job title changing fastest is not "AI engineer" -- it is "person who supervises three AI colleagues and one junior." That role needs a skill set almost nobody was trained for: specification writing, output evaluation, and knowing when to stop prompting and start doing it yourself.

If you want a fast, structured read on how exposed your current role is to being re-scoped by an AI colleague -- and which parts of your work are already inside the capability frontier -- our free Career Pulse Score takes about four minutes. It is a useful complement to the task-level data above: the report tells you what AI colleagues do well in aggregate, and the score tells you how much of your week sits in those zones.

"We rolled out an AI teammate to our claims review team in Q1 and the first dashboard said 41% faster case handling. Then our QA lead started tracking override rate and found we were changing 38% of the AI's outputs before they went out the door. Once we put a number on the review time, we were actually 6% slower on complex claims and about 30% faster on routine ones. We didn't abandon the tool. We just stopped pretending it was one tool with one number. That reframe is the only reason the program survived our CFO's Q3 review."

That is the report in one paragraph: a blended number hid a bimodal reality, and measuring the right metric saved the program.

How to Actually Instrument an AI Colleague (Without Buying a Platform)

Most teams measure AI colleagues the way they measure software uptime: a single dashboard, a single percentage, a single monthly review. That approach fails for one structural reason -- AI colleagues are bimodal. They are excellent on some tasks and net-negative on others, and a blended average hides both facts simultaneously.

Here is the minimum viable instrument. Four metrics, one spreadsheet, no vendor required.

Notice what is missing: satisfaction, sentiment, and "hours saved." Those are fine for a culture deck. They will not tell you whether to renew the contract.

Baseline before you deploy, or you will never know

The single biggest methodological failure in AI colleague measurement is deploying first and baselining never. You cannot reconstruct a pre-AI baseline six months later, because by then the people, the process, and the work itself have all shifted. Spend two weeks capturing current cycle time, rework rate, and review time on the specific tasks you intend to hand to an AI colleague. Two weeks of measurement buys you years of defensible answers.

Three Real Scenarios, Priced Out

Scenario A: The senior engineer with 9 years of context. METR's participants are the canonical case. They knew their repos better than any model could, which meant the AI's plausible-looking suggestions had to be fully evaluated before they could be trusted -- and evaluation of a confident wrong answer is slower than writing the right answer from memory. If you are deeply expert in a domain, expect AI colleagues to help you most on adjacent work: writing docs, scaffolding tests, exploring unfamiliar libraries. Do not expect them to speed up the work you are best at. That is where your speed comes from judgment, and judgment is the thing AI is worst at accelerating.

Scenario B: The mid-level analyst producing 20 deliverables a week. BCG's 758-consultant study maps almost perfectly onto this role. Analysts with AI help completed 12.2% more tasks, 25.1% faster, at 40% higher quality -- genuinely transformative numbers. But the same study found that when tasks drifted outside the model's capability frontier, AI-assisted consultants were 19 percentage points more likely to deliver a wrong answer, because the fluent output masked the error. The operational lesson is to define your frontier explicitly: a written list of task types where you accept AI output without independent verification, and everything else gets checked. Fluency is not accuracy, and your colleagues will trust fluent output by default.

Scenario C: The operations team automating case review. This is Priya's scenario above, and it is the most common enterprise pattern in 2026. Routine cases get genuinely fast -- 30% or better. Complex cases get slower because the AI produces almost-right determinations that a human must fully re-adjudicate. The winning move is not to switch the tool off. It is to route by complexity: AI drafts, human approves on routine; human drafts, AI assists on complex. Same tool, opposite configuration, radically different economics.

Insider Tips From People Who Have Run These Programs

• Never let the vendor define your success metric. Vendor dashboards measure interaction volume, not outcomes. Interaction volume goes up when a tool is confusing. Build your own four-metric sheet and treat vendor reporting as marketing.
• Log the override, not just the output. A one-click "edited" flag is worth more than a paragraph of feedback text, because it produces a countable, comparable number across thousands of tasks.
• Watch for the review-time creep. Review time almost never shows up on purpose. It shows up as longer meetings, later QA cycles, and senior people doing more checking. Ask your team directly: "what are you double-checking that you did not used to?"
• Re-test on a schedule, not on vibes. Capability moves in months, not years. A workflow that failed in Q1 may be fine in Q3.
• Price the review at your loaded rate, not the junior rate. If senior engineers verify AI output, cost per resolved task is far higher than tool math suggests -- and that is the number that determines whether the program survives budget season.
• Give your AI colleague a job description. Tasks it owns, tasks it drafts, tasks it never touches. Teams with a written scope get materially better override rates than teams that let usage spread organically.
• Track your own frontier, too. The metric with the longest shelf life is not how fast your AI colleague works -- it is how much of your week sits in the zones where it is genuinely reliable. Use the free Career Pulse Score to get a structured read on that exposure, then revisit it every two quarters.

What to Do in the Next 30 Days

Week 1 -- Classify. List your team's ten most time-consuming recurring tasks and tag each as routine, structured, or entangled. Do this before looking at any tool data, so tool enthusiasm cannot bias the labels.

Week 2 -- Baseline. For the three tasks you plan to hand off first, capture cycle time, rework rate, and review minutes for two weeks of normal work. No AI involved. This is your control group.

Week 3 -- Deploy narrow. Introduce the AI colleague on the routine task only. Track the same three numbers plus override rate. Resist the urge to expand scope in the same week; you will lose the ability to attribute the change.

Week 4 -- Reconcile. Compare. If override rate is under 15% and cost per resolved task is below your human baseline, expand to the structured task. If override rate is above 40%, you have found a genuinely useful drafting assistant -- label it that way, and stop describing it as automation in your planning documents. Honest labeling is how programs survive their second budget review.

Methodology Note

This report synthesizes published peer-reviewed and preprint research, controlled randomized trials, large-scale practitioner surveys, and independent enterprise studies published between 2023 and 2025. Primary sources include the GitHub / Peng et al. controlled trial of Copilot (2023), the METR randomized controlled trial of experienced open-source developers (2025), the BCG / Harvard / Wharton "Jagged Technological Frontier" field experiment with 758 consultants (2023), the Microsoft Work Trend Index (2024), the DORA / Google Cloud Accelerate State of DevOps findings on AI adoption and delivery performance (2024), the Stack Overflow Developer Survey AI sections (2024 and 2025), the Anthropic Economic Index, MIT NANDA's 2025 research on enterprise GenAI pilot outcomes, and Gartner's published forecasts on agentic AI project cancellation and enterprise software composition.

Limitations, stated plainly. First, several widely cited studies use small or specialized samples -- METR's trial had 16 developers, which is enough to establish that a slowdown is possible but not enough to generalize to every engineer. Second, vendor telemetry is not independently audited; where a vendor number appears, it is presented alongside third-party evidence, and ranges are used rather than point estimates where disclosure is partial. Third, benchmark scores measure performance on benchmark tasks, which are by construction more bounded than real work -- treat them as a ceiling, not a forecast. Fourth, survey data on trust and frustration is self-reported and therefore vulnerable to the same perception gap documented in the opening section. Fifth, the 2026 column in the adoption table is a projection built from published forecasts, not measured data; it is labeled accordingly. Where this report uses a range instead of a figure, it is because the underlying evidence does not support a single number, and pretending otherwise would be exactly the failure mode the data warns against.

The most useful thing you can take from all of this is not a statistic. It is a habit: measure the task, not the tool. AI colleagues are real, improving fast, and genuinely transformative on a specific slice of work. They are also net-negative on another slice, and the only way to know which slice you are standing on is to count.