Only One of Four Agent-Team Configurations Showed a Measurable Gain — at Up to Five Times the Price
A new benchmark suggests that swarms of identical AI agents rarely write better code than a single agent, yet cost anywhere from 1.8 to 5.1 times as much. At the same time, other documentation points to role-differentiated pipelines that can exploit multiple models profitably. The choice should therefore not be "team or no team," but which kind of team.
What Vals AI actually measured
Everything known about the numbers comes from The Decoder's coverage of Vals AI's benchmark study — the company's own report is not among the documents supplied, so all information about methodology or scores must be read as secondhand reporting, not independently verified (source: The Decoder, Matthias Bastian, October 11, 2026, ID b56a5322).
The test was Vibe Code Bench with two frontier models: OpenAI's GPT-6 Sol and Anthropic's Claude Opus 5.5. Each was run solo and as an agent team, at two reasoning levels — medium and maximum effort. The cost result was unambiguously negative for the teams: across the configurations tested, teams cost between 1.8 and 5.1 times as much as the single-agent variant.
The quality result, by contrast, was roughly absent. Of four team-versus-solo comparisons, only one was statistically significant: GPT-6 Sol at medium reasoning, where the team scored 7.3 points higher. The other three showed no measurable gain. The coverage does not give the absolute baseline scores, so it is impossible to assess how large the 7.3-point gain is in practice — or what it concretely costs in the three non-significant cases. What can be said is that for three of the four configurations, there was no statistically measurable quality gain at all, while the cost increase lay somewhere in the reported range of 1.8 to 5.1 times.
Why teams fail: the coordination tax
Two OpenAI employees have explained the mechanism, both as relayed through The Decoder.
Eric Provencher, a developer at the company, warned against agent swarms for precisely this reason. The coordination between the agents breaks down, he argued, and the money is "most likely wasted." He called the phenomenon the "coordination tax."
The researcher Noam Brown made the same point from another angle, via the Dwarkesh Podcast: multi-agent systems primarily buy speed, not quality. Four agents solve tasks twice as fast — but also cost twice as much. In that framing, the "team" is simply a parallelization investment, not an intelligence booster.
Brown also noted that the effect depends on the type of task. Web research and math parallelize well, because they can be divided into independent sub-problems. Writing a novel does not: throwing 10,000 agents at a novel is as useless as throwing 10,000 humans at it, Brown said, according to The Decoder. The same logic presumably applies to any coherent codebase: parallelize divisible work, not coherent output.
Anthropic's own scaling tests confirm the pattern
It is worth noting that Anthropic itself — the vendor behind Opus 5.5 — has also observed diminishing returns, according to The Decoder's account of two tests the company ran with Opus 5.5. Larger teams reached a given performance level faster. But going from ten to one hundred agents yielded only a marginal improvement after 24 hours.
This is a company that actively sells agent infrastructure, and yet it is reported that the coordination tax exists in its own systems. That strengthens the credibility of the pattern — although these figures, too, are available only via secondary sources, not the primary publication.
The complicating evidence: role-differentiated pipelines
This is where the picture becomes less clean than the "teams are wasted money" narrative suggests.
Anthropic recently launched Claude Haiku 5.5 as a deliberate "sub-agent" model: 72.4% on OSWorld 2.1 — the standard benchmark for AI computer use, which the publisher places at level with the 72 percent human reference — at $0.10 per million tokens for "the vast majority" of real requests. That is a 90% drop from the predecessor's $1.00 (Anthropic claim, as relayed in coverage material, ID 5cacaf4a).
Cognition, the company behind Devin, has built a configuration — Devin Fusion — that explicitly exploits this price space: Haiku 5.5 as a "sidekick" for execution tasks, with Opus 5.5 as the lead planner. According to Cognition's own figures, the combination scored 66.2 on FrontierCode 1.1 — higher than either model alone — while both cost per task and latency fell compared with a single-model setup.
This is a critical distinction: Vibe Code Bench did not test this type of heterogeneous pipeline. The Devin Fusion figure is a self-reported benchmark, not an independent verification. But it points toward a nuanced conclusion: it is not the "multi-agent" idea itself that fails — it is swarms of identical models that duplicate work and compete over coordination. When the agents have different roles, different price points, and different strengths, the math can flip.
Practical guidance: when is it worth it?
Based on the available documentation, the boundaries look like this:
Pay for teams when:
- The task is parallelizable — web research across many sources, mathematical search, exploration of large codebases. Brown's observation supports this directly.
- You can afford double the price for double the speed, and speed is actually the bottleneck.
- The team is heterogeneous (cheap executor + strong planner), not a swarm of identical models.
Don't pay for teams when:
- The task requires coherent, unified output — refactoring across files, architecture design, writing.
- The coordination cost (Provencher's "coordination tax") likely exceeds the gain, which Vals AI's data suggests happens often.
- You are considering scaling from ten to one hundred agents — Anthropic's own data suggests the gain there is marginal relative to the cost.
Open questions
Several things are still missing: Vals AI's full report, transcripts from Brown or Provencher, and Anthropic's and Cognition's primary documents are not available in this evidence base. The absolute baselines behind the 7.3-point gain are not given in the coverage material, so we cannot assess how large the gain is in practice. And the FrontierCode and OSWorld results have not been independently reproduced.
But the direction is clear enough to act on: don't build swarms. Build pipelines.