Vals AI benchmark: Agent teams cost 1.8–5.1 times more, and delivered better quality in only one of four comparisons
New benchmark results from the evaluation firm Vals AI suggest that teams of AI agents mainly buy speed, not better outcomes. According to a report published by The Decoder on October 11, 2026, multi-agent solutions with agent teams cost between 1.8 and 5.1 times more than single agents, while only one in four comparisons showed a statistically significant quality improvement. The conclusion is corroborated by Anthropic's own scaling tests and warnings of a "coordination tax" from OpenAI researchers — but the evidence base has clear limitations that organizations should know about before reworking their own agent deployments.
What Vals AI found
Vals AI tested the models GPT-6 Sol and Claude Opus 5.5 on the "Vibe Code Bench" benchmark, both as single agents and as teams, at two reasoning levels: medium and maximum reasoning effort. According to The Decoder's reporting, the agent teams cost between 1.8x and 5.1x more than the single agents (The Decoder).
The quality gains were far more modest. Of four comparisons between teams and single agents, only one showed a statistically significant improvement: GPT-6 Sol at medium reasoning effort, where the team scored 7.3 points higher. At maximum reasoning effort, the team setup gave neither Sol nor Opus 5.5 any real advantage. In other words: the more expensive configuration paid off in practice in only one of four cases — and then under conditions where the model was not running at full compute.
The Decoder summarizes the results this way: the extra cost of agent teams is not worth it in most cases, especially when the models are already running at full compute power.
Anthropic's own scaling tests point in the same direction
The picture is partly corroborated by Anthropic, which, according to The Decoder, has run two of its own tests with Opus 5.5. The company observed that quality gains flatten out as more agents are added: larger teams did indeed reach a given performance level faster, but going from ten to 100 agents moved scores only marginally after 24 hours. On a knowledge-base task, the score rose from 0.70 to 0.74, and on Lean theorem proving from 0.66 to 0.68 — a tenfold increase in the number of agents for a few hundredths on the scale.
One interesting exception is the model Fable 5.1, which showed strong quality gains on Lean theorem proving above ten agents. But Fable 5.1 nonetheless scored below Opus 5.5 in all tests, and on the knowledge-base task the score actually fell slightly when scaling from 30 to 100 agents. That last point is worth noting: more agents can therefore produce not just diminishing returns — in some cases the results appear to get worse.
It is worth keeping in mind that this is secondary-source reporting. Anthropic's test data is presented without a publication date or methodology in the source, and it is unclear whether the company has published them independently or exclusively via The Decoder. Read them as indications, not fully documented results.
OpenAI: speed, not quality — and a "coordination tax"
OpenAI researcher Noam Brown has, according to The Decoder, confirmed the picture on the Dwarkesh Podcast: multi-agent systems mainly buy speed, not better quality. Four agents solved tasks twice as fast, but also cost twice as much. With 16 agents the pattern held, though efficiency became somewhat worse. Brown emphasized, according to the reporting, that the effect is strongly task-dependent — it is not a universal law.
OpenAI developer Eric Provencher has recently warned against using agent swarms for the same reason. They are most likely wasted money, he argued, because the coordination between agents breaks down. He called the phenomenon the "coordination tax": the more agents that have to work together, the more of the effort goes into keeping them synchronized instead of solving the actual task.
What the findings actually mean
Taken together, three independent sources point in the same direction: agent teams appear mainly to reduce waiting time, not raise the ceiling on quality. That is a useful distinction for anyone planning agent architecture.
Concretely, the data suggest two situations where teams can be justified:
- When speed is the whole point. Brown describes four agents solving tasks twice as fast at double the price. If time-to-result is worth more than token cost — for example in interactive workflows — the math can work out. Anthropic's finding that larger teams reach performance levels faster points in the same direction.
- When the model is otherwise running below maximum reasoning effort. The only statistically significant quality gain in Vals AI's data came at medium reasoning effort. At maximum effort, the teams gave no advantage. That suggests extra agents partly act as an expensive substitute for compute that the single model could already use better on its own.
And one situation where they seemingly cannot: when the goal is the highest possible final result and the model is already running at full capacity. There, you pay up to 5.1 times more for roughly the same result.
This is supported analysis, not established consensus. Vals AI is a company that sells evaluation services — no conflict of interest has been proven, but it is a position that should be made visible, and the source does not address the question.
The open questions
There are significant limitations in the evidence base:
- Everything rests on one secondary source. Neither Vals AI's own report, Anthropic's internal test documents, nor the podcast transcript are available here. All figures are subject to The Decoder's reporting.
- The benchmark is undocumented in the source. The scope of Vibe Code Bench, the task types, and what the scores actually measure are not described in detail. The fact that the results concern coding and agent tasks does not mean they generalize to all coding or agent workloads.
- The methodology behind Anthropic's tests is missing. Without dates, setups, and error margins, we cannot assess how robust the figures 0.70→0.74 and 0.66→0.68 are.
- The task dependency is unresolved. Brown himself characterizes the effect as strongly task-dependent, but the source does not say which task types benefit most or least from teams.
The most defensible conclusion is therefore not that agent teams are a "fad." It is that the cost calculus has changed: while agent-team frameworks are being scaled out across organizations now, three independent sources suggest the standard assumption — more agents, better results — often does not hold. For most organizations, that should trigger a concrete question: are we even measuring whether our agent teams are delivering quality gains that justify the multiplier they add to the bill?