From 75 to 56 percent: Fewer companies let AI agents control production changes on automated tests alone
The share of enterprises that allow — or are building toward — autonomous AI agents changing production systems based solely on automated evaluations fell from 75 percent in July to 56 percent in August, according to VentureBeat Intelligence's VB Pulse survey. Over the same period, the share requiring human review of agent changes doubled, from 20 to 42 percent. The numbers suggest that the automated evaluation pipeline — the industry's answer to the question "can we trust the agent?" — is losing its status as the final guarantee in enterprise deployment practice.
The Shift in the Numbers
The survey was conducted among respondents at organizations with at least 100 employees. Among those reporting that their organization runs autonomous AI agents, the bases were 96 respondents in July and 118 in August, with one response each.
In July, 75 percent of this group said they either already let an agent push changes to production on automated evaluation results alone, or are building toward that. By August, the same response share had fallen to 56 percent. At the same time, the share expecting to have a human review of agents' production changes "for the foreseeable future" rose from 20 to 42 percent in a single month, according to VentureBeat Intelligence.
The August figure of 56 percent is not a picture of a full stop. It splits into two groups: 32 percent already allow it for specific low-risk agents or changes, and 24 percent are building pipelines to allow it within 12 months. The caution, in other words, often takes the form of narrower exceptions and tighter risk classes — not a brake on the entire agent program.
Why Trust in the Evaluations Is Faltering
The survey points to a concrete problem: evaluations let through errors that customers notice. Among August respondents running pre-deployment evaluations — setting aside the 5 percent that do not run such evaluations at all — 61 percent reported at least one customer-facing error in the past twelve months that had passed the evaluations.
Confidence in the very tool meant to catch such errors is modest. When respondents were asked which limitation most reduces their trust in automated evaluation, only 9 percent chose the response that they trust automated evaluation as it works today. The rest pointed to a limitation.
Another number highlights the gap between pre-deployment testing and production reality: only 29 percent say their production monitoring is built around real-time automated quality checks — mechanisms that verify whether the agent's work is actually correct as it happens. For 53 percent, the main approach is trace-logging or gateway tracing, which, per VentureBeat's phrasing, is not framed as a correctness check. In other words: most can see what the agent did, but few have a system that can reveal it was wrong.
An Important Caveat: No Established Cause
The shift from July to August is tempting to explain by the error experiences — "61 percent saw evaluation-passing errors, so they doubled the requirement for human review." But the survey itself does not support that link. Among the organizations running autonomous agents, the share allowing unreviewed pushes — or building toward them — was "too close to call" between respondents who had reported an evaluation-passing error and those who had not.
That means error experience does not distinguish allowers from non-allowers in the data. Why trust fell in exactly one month has not been established. Any causal explanation — market news, incidents in their own operations, changed risk understanding — is analysis, not documented fact.
It is also worth keeping in mind what the data foundation is: 96 and 118 self-reported respondents in two monthly surveys, with no published statistical basis for the month-to-month comparison, and no independent verification of whether the organizations actually let agents through or not.
The Pattern in Practice: The Human as Write Gate
An illustration of what enterprises are actually putting in place comes from a different kind of source. Brian King and Richard Mendis describe in a first-person VentureBeat post — a vendor account of their own product — how their ServiceNow-based digital worker handles risk: a human reviews and approves every write operation before it is executed (VentureBeat). That is exactly the pattern the doubled 42 percent share in the survey points toward: the agent gets to work, but anything that changes state goes through a human first.
The claim is thus the company's own description of its own product, not documentation of the survey's findings. But it shows what human review looks like as concrete architecture, not as general skepticism.
Why Automated Checks Don't Catch Everything
Why do the evaluations fail to catch the errors? The survey reports that they slip through, but not the mechanism. A secondhand anecdote from Android Police points, however, toward a well-known problem: a developer examined 20 AI-generated apps in the Play Store and confirmed security issues in seven of them. Four of the apps contained secrets in the release package, including a private Google Cloud key and .env files packed with AI vendor keys that had "accidentally" ended up there (Android Police).
This too is an anecdote, not proof of the survey's findings — one developer's sample, relayed secondhand. But it illustrates the category of errors automated tests easily miss: not broken functionality that tests can detect, but things that appear to work and only go wrong in encounter with the real world — precisely where the customers are. The survey alone cannot establish that this is the cause of enterprises' experiences; it only suggests why "the evaluation said yes" and "the customer experienced errors" can coexist in the same organization.
The Platform Picture: More Evaluation, More Review at the Same Time
Paradoxically, enterprises' investment in evaluation is rising even as trust in letting evaluations stand alone falls. Among respondents who answered the platform question, 59 percent in August reported that their organization runs OpenAI Developer Platform's built-in evaluation and tracing tools, up from 31 percent in July. And in total, 62 percent of August respondents planned to introduce a new, additional, or replacing evaluation platform within 12 months.
The reading is not that evaluation is disappearing, but that its role is changing: from final gatekeeper to one of several signals. The trend points toward a model in which automated tests filter out the obvious, production monitoring should catch what slips through — and a human retains the final word on changes with consequences.
What This Means for Teams Building Agent Pipelines Now
For teams building agent pipelines today, the numbers point to three concrete moves. First: expect narrow risk classification to become the standard, not fully automatic deployment — the 32 percent who allow automated pushes do so explicitly for "specific low-risk agents or changes." Second: plan for the skew between testing and monitoring — with only 29 percent on real-time quality checks, evaluation-passing errors (which 61 percent have experienced) are mainly covered by trace-logging that detects, but does not assess quality. Third: build human review into the architecture as a write gate, as the ServiceNow example shows, rather than as an after-the-fact process formality.
The Open Questions
There are several things the survey cannot answer. Why trust fell in exactly one month — July to August — is unknown, and as noted, error experience does not distinguish the groups from each other. The statistical basis for calling some differences real and others "too close to call" has not been published. Everything rests on self-reporting from 96 and 118 respondents respectively at organizations with 100+ employees, without verification of actual deployment practice. And whether the August trend holds in September, or was one month's fluctuation in a small sample, no one yet knows.
What the current evidence supports is what VentureBeat Intelligence's own numbers show: in one month, enterprises' answers went from treating the automated evaluation as the final check to treating it as an intermediate step — and the share wanting a human as the last check doubled.