Nobody Independent Is Checking If AI Actually Works
Every ROI figure in the market was commissioned by the vendor being graded. When researchers with no stake measured the same work, developers came out 19% slower, and 95% of enterprise deployments nev
TLDR
The AI cost debate has a villain, and it is the wrong one. Yes, the tools now cost more than the people they replaced. Uber torched its entire 2026 AI coding budget in 4 months. Nvidia admits its compute bill beats its payroll. None of that is the real story, because cost was never the metric that mattered. Productivity was. And the industry already knows it.
This is not an anti-AI piece. The technology is real, the gains in raw output are real, and the shift the market just made, from paying per seat to paying per outcome, is the correct shift. Paying only when the software works is exactly how this should be priced.
The market has not priced this in. The “outcome” is defined, measured, and marketed by the company collecting the check. Every headline ROI figure in the market was commissioned by the vendor being graded. And on the rare occasions someone with no stake in the answer has actually measured the work, a randomized trial, 2 years of telemetry across 22,000 developers, a 300-deployment academic study, the result is worse than either side of the cost fight assumed. Output goes up. Delivered value does not.
The subsidies that hid all of this are unwinding in real time, and the repricing has started. When the bill gets honest, the question stops being what AI costs and becomes whether anyone ever checked that it worked.
A vendor just bet its own revenue on the answer
On June 25, 2026, Salesforce relaunched the pricing for Agentforce, its flagship AI agent. The new model is pay-per-resolution. You are charged roughly $2 only when the agent resolves a customer issue on its own, start to finish. If the customer asks for a human, or walks away unhappy, there is no charge. The cost is tied to the outcome, not the activity.
Read that as what it is: a concession. For 2 years the pitch was that you should buy intelligence and trust that value would follow. This is the vendor quietly admitting that customers stopped believing it, and demanding proof before they pay.
It is also the third pricing model for the same product in 18 months. It launched at $2 per conversation. Customers could not tell what counted as a conversation, so it moved to $0.10 per action. Then per-user licenses at $125 a month. Now, per outcome.
That zigzag is the whole market compressed into one company. The unit of billing has been walking steadily away from “how much AI you consume” and toward “what the AI actually accomplished.” Which is the right direction. It is also an admission that the old way of measuring was selling something nobody could verify.
The cost panic is real. But it is also last quarter’s fight.
The numbers behind the panic are not in dispute. Uber’s CTO disclosed the company burned through its entire 2026 AI coding budget in 4 months. By March 2026, 84% of Uber’s engineers had adopted Claude Code, and roughly 70% of committed code originated with AI. The president conceded that token usage did not seem to correlate with useful features shipped.
It is not an outlier. Nvidia’s VP of applied deep learning said the cost of compute for his team now exceeds what the company spends on the employees using it.
The company that sells the shovels admits the digging costs more than the diggers.
And OpenAI spends close to $2 for every $1 it earns on inference, and loses money on its $200-a-month subscriptions by its own CEO’s admission.
So the tools cost more than the humans. Everyone agrees. That is precisely the problem with the debate: it has already been won, loudly, by both sides, and it is a fight over the wrong number.
Cost tells you what you spent. It tells you nothing about what you got. The only comparison that decides whether any of this was worth it is productivity, and productivity is the thing almost nobody is measuring honestly.
“Costs are falling” is a mirage too
The reflexive rebuttal is that prices are collapsing, so the cost problem solves itself. Per-token prices have genuinely dropped, and Gartner projects that running the largest models could be nearly 90% cheaper by 2030.
The catch is that consumption is scaling faster than the price is falling. Cheaper tokens do not produce a smaller bill when you burn far more of them, and every incentive in the building pushes people to burn more of them.
Amazon built an internal leaderboard to rank AI usage across engineering teams. It came down after staff started burning tokens on meaningless tasks purely to climb it. When you reward consumption, consumption becomes the output.
Falling unit prices do not save you from a culture that treats spending as a scoreboard. They just lower the price of each point.
The newest evidence for the rebuttal landed on July 16, 2026, when Moonshot AI released Kimi K3 in open weights, claiming frontier-competitive performance at a fraction of the price. Rival AI stocks fell the following day. Fortune called it a second DeepSeek shock.
Take that claim at face value, and the arithmetic still does not move. The bill is unit price times consumption, and cheap capability is the strongest invitation to consume more of what this industry has produced.
The industry found the right metric. Then it graded its own homework.
Now go back to that launch and read the coverage.
Bloomberg reported that Kimi K3 beat every rival except Claude Fable 5 and GPT-5.6 on overall capability, citing Moonshot itself. Quartz placed it below GPT-5.6 Sol overall, ahead of the prior generation on coding and agent evaluations. Fortune reported Moonshot claiming K3 substantially outperformed GPT-5.6 Sol, the model the other outlets had it trailing.
Three financial outlets, 24 hours, one vendor’s self-reported benchmark run, and no agreement on what the vendor had even claimed. Markets repriced anyway.
Nothing independent had been checked, because there was nothing independent to check.
This is not a China story. It is a measurement story, and it runs through every lab. On July 8, 2026 and July 9, 2026, Grok 4.5 and GPT-5.6 shipped a day apart, and neither led with “smartest.” Both led with efficiency: tokens per task, performance per dollar, cost per result.
The whole industry repriced its pitch around productivity. That is the strongest confirmation yet that productivity, not raw capability, is now the metric. It is also exactly where the grading problem begins.
Every one of those efficiency figures is the vendor’s own, on benchmarks the vendor picked. When Databricks built its own coding-agent benchmark in July 2026, it refused the public ones outright because the tasks are public and the solutions leak into training data over time. It also refused to grade with an LLM judge, on the grounds that a judge rewards sounding right over being right.
Then it went further. Mid-run, Databricks had to seal off its own git history, after finding that agents with shell access were walking through past commits to dig up the merged solution. That is what it costs to get an honest number, and almost nobody pays it.
OpenAI’s own footnote concedes its cost and latency estimates come from offline simulation and could vary substantially in practice. This applies to every lab, Anthropic included.
There is no neutral referee timing the race.
The self-grading does not stop at benchmarks. Once you accept that outcomes are what matters, the question is who decides what counts as an outcome. Today, it is the vendor being paid for it.
Take that pay-per-resolution model. A “resolution” is defined by the seller. Salesforce’s own CRMArena-Pro benchmark found that an out-of-the-box agent hit an accuracy of roughly 35% before heavy customization.
Yet the customer-facing resolution numbers that it and its peers advertise run anywhere from 25% to 95%, and no two of them mean the same thing. Some count ticket deflection. Some count customer-confirmed resolution. Some count raw response accuracy.
When one word stretches from 25 to 95 depending on who is defining it, the word is not a measurement. It is marketing wearing a percentage sign. And the customer, who is now paying per resolution, is paying against a number that the vendor gets to draw.
Every ROI figure you have seen was ordered by the company being graded
Widen the lens from customer service to the whole enterprise-AI ROI industry and the pattern holds. The wall of returns at the top of this piece, from 293% to 465%, comes from Forrester’s “Total Economic Impact” studies. Every one of them was commissioned by the vendor whose product is being evaluated.
This is not an accusation of fraud. Forrester’s methodology is disclosed, the studies are labeled, and the underlying shape they describe is often real. It is a point about who is holding the ruler. A vendor does not commission and publish a study that makes its own product look like a waste of money. These figures are the ceiling of what the seller wants you to believe, not an independent read on what buyers actually got.
The question is not whether the returns are real. It is who is holding the ruler.
And there is at least one number floating around the discourse that traces to nobody at all. A widely reshared claim of “540% ROI across 287 audited enterprise deployments” appears on a single vendor content page with no findable underlying report. It is the kind of stat that spreads precisely because no one checks. This piece does not use it.
When someone with no stake actually measured
Now put the self-graded numbers next to the independent ones. The gap is the entire story.
METR, an AI research nonprofit, ran a randomized controlled trial with experienced open-source developers on real tasks in codebases they knew well. Economists predicted AI would make them 39% faster. Machine-learning experts said 38%. After using the tools, the developers themselves estimated they had been sped up 20%. Measured, they were 19% slower. The people doing the work could not tell that the tool was costing them time.
That is one study on a hard setting, so hold it loosely on its own. It does not stand alone.
Faros AI analyzed 2 years of engineering telemetry across 22,000 developers and more than 4,000 teams, comparing each team’s lowest-AI quarters against its highest. Output rose: epics completed per developer up 66%, tasks up 34%.
Then the invoice arrived downstream. Bugs per developer are up 54%. Incidents per pull request are up 243%. Median review time is up more than 5x. Code churn, work thrown away shortly after it was written, is up 861%.
More code got written. More of it also got reverted, reviewed, and repaired. That is not a productivity gain. That is motion. And when MIT’s NANDA initiative reviewed more than 300 real enterprise GenAI deployments, it found that roughly 5% produced measurable impact on the bottom line. The other 95% never crossed from pilot to profit.
The same MIT research found the other half of the story, and it is the part that isolates the problem. While 95% of sanctioned deployments stalled, employees were not waiting. Roughly 90% reported using personal AI tools for work, often daily, even inside companies whose official pilots had failed. And that unsanctioned “shadow AI” frequently delivered more real value than the six-figure deployments sitting next to it.
Sit with what that means. The technology works best exactly where nobody is selling you an outcome, a person picking a tool for a task they understand. It stalls where someone is, a vendor booking revenue against a number they defined. The failure is not the AI. It is the sold outcome wrapped around it.
Self-graded, the story is 300% to 465% returns and 95% resolution. Independently measured, it is 19% slower, 861% more churn, and 5% with real P&L impact. Both cannot be true. Only one of them was checked by someone who did not get paid for the answer.
The obvious objection, taken head on
The strongest rebuttal is that the independent studies are already stale. METR tested early-2025 tools, and the models have improved sharply since. Fair. If the slowdown were a one-time artifact of early-2025 tooling, the effect should be shrinking as the tools get better.
It is not shrinking. It is widening. Faros’s own bug-rate finding went from a 9% increase in its 2025 study to a 54% increase in its 2026 study, on newer tools. GitClear, tracking more than 211 million changed lines, found code churn roughly doubled from its pre-AI baseline as adoption climbed, an independent second source for the same rework problem.
The tools got better and the repair bill got bigger, because better tools generate more code faster, and the bottleneck was never generation. It was review, testing, and integration, the parts a human still has to own.
Faster generation into an unchanged review pipeline does not speed delivery. It floods it. The counter-argument predicts the gap should be closing. The data says it is opening.
Where this leaves the money
This is the connection to a thesis this newsletter has made before: after the AI repricing, value does not accrue to whoever runs the biggest model. It accrues to whoever owns the workflow, the domain logic, and the governance, the layer where “did it work” can actually be measured, audited, and stood behind.
This piece is about that measurement gap, the layer where AI vendors grade their own outcomes. It is not about application-layer software serving real customers at real enterprise scale, which runs on completely different unit economics.
As a solution consultant at Adobe I have watched augmentation deployments work when they are scoped correctly. Forrester’s own Total Economic Impact studies point at the same shape once you look past the headline number: triple-digit returns when the use case is tight and a human stays in the loop, and near-zero when a company tries to pull the human out of the workflow entirely.
That is the tell. The deployments that work are the ones where someone can prove they worked. The ones drowning in churn and stalled pilots are the ones sold on a number the buyer could never verify.
The durable position in AI is not the most intelligence. It is the most accountability.
Databricks is what that looks like in practice, and it is the reason this piece is not a case against the technology. It built its benchmark out of its own engineers’ merged pull requests, graded against the tests those engineers wrote, on code no model had trained on. The agents genuinely helped. The company got real efficiency out of them.
What it also got was a number nobody sold it. Token price turned out to be a poor predictor of task cost: one model roughly 1.7x cheaper per token finished the work more expensively, at $2.09 per task against $1.94, while scoring 6 points lower, because it read more and burned 1.9x the tokens getting there. Running the same model through a different harness swung cost per task by more than 2x at identical quality.
None of that is visible from a price list, and none of it appears in a vendor deck. It is only visible if you measure your own work yourself.
Every company with a history of merged pull requests already owns an evaluation set, written by its own engineers, that no vendor can game.
That is the option almost nobody exercises, and it is free.
The parallel is Cisco in 1999. The internet was real. Cisco’s routers were real. The company still lost most of its market value, because a real technology and a sustainable business are two different bets, and the market had confused them.
AI is a real technology. Whether it pays for itself before the subsidies run out is a separate question, and right now the industry is answering it with numbers it wrote itself.
Nobody is checking if AI actually works. They are checking whether the company selling it says it worked.
The tools cost more than the people. The vendors moved the goalposts to outcomes. Then they appointed themselves referee.
AI’s productivity is mispriced.