Half of enterprises have shipped an AI agent that passed every internal evaluation and then failed in front of a customer. One in four have done it more than once. The problem isn't that AI agent evaluation got skipped — it's that the tests were the wrong shape.
The failure that isn't a bug
In June 2026, VentureBeat ran a Pulse survey of 157 qualified enterprise respondents at companies with 100 or more employees. Half of them had deployed an AI agent or LLM feature that passed internal evaluations and still caused a customer-facing failure. A quarter had done it more than once.
The sample was self-selected rather than a probability sample, and VentureBeat says plainly that the numbers should be read as directional rather than precise. Fine. The direction is still not ambiguous, and it points somewhere uncomfortable.
Because this is not a QA failure in the ordinary sense. Nobody skipped a step. The suite ran, the score came back green, the release went out on the strength of it. What failed was the assumption underneath the suite.
Deterministic testing asks a question agents can't answer
Software testing has one basic shape: fixed input, expected output, assert. It works because the system under test is deterministic. Run it a thousand times and you get the same answer a thousand times, so one run is a complete answer.
An agent chooses its own route. It decides which tools to call, in what order, with what parameters, and it can reach the same destination by several different paths — or reach a different destination entirely on the second attempt at an identical request.
Which means the interesting failures aren't crashes. They're the runs where nothing errors. The agent retrieves the correct account and updates the wrong field. It drafts a valid refund and sends it without approval. It calls five tools cleanly and the sixth leaves the workflow half-finished. Each individual decision looks defensible in the transcript. The outcome is still wrong, and no exception was ever raised to tell you so.
A green test result on a system like that isn't a verdict. It's a sample of one.
The arithmetic nobody runs before signing
Anthropic's engineering guidance on evaluating agents draws a distinction that belongs on the first page of every AI procurement document, and almost never appears there.
pass@k measures whether the agent gets at least one correct result across k attempts. Raise k and the number goes up — more shots on goal, better odds one goes in.
pass^k measures whether all k attempts succeed. Raise k and the number goes down, because demanding the same result every time is a harder bar.
Anthropic's own worked example: an agent with a 75% per-trial success rate, run three times, has a (0.75)³ ≈ 42% chance of getting all three right.
Nothing about the agent changed between those two figures. The model is the same, the prompt is the same, the tools are the same. The only thing that changed is which question you asked it.
Capability and consistency start at the same number and separate immediately. Both are true statements about the same agent.
Demos answer the first question. So do most vendor decks, most pilot write-ups, and most internal proof-of-concept reports — not dishonestly, just by default, because a demo is a single run and a single run can only ever measure capability.
Every customer-facing workflow you would actually pay for lives in the second question.
And the human is being taken out at the same moment
The same VentureBeat survey found that 66% of respondents already permit some production deployment without human review, or are building systems intended to do so within the next twelve months. Only 5% say they fully trust the automated evaluations that would make those release decisions.
Sit with the gap between those two numbers for a second. Two-thirds are removing the reviewer. One in twenty trusts the thing replacing them.
The person in the loop was never there purely for governance theatre. They were absorbing variance. They were the one who noticed the field was wrong before the customer did, who queried the refund that looked odd, who quietly re-ran the thing that came back strange. Remove them and the variance does not go anywhere. It simply stops being caught, and an uncertain outcome becomes an automatic one.
The complaint isn't that AI agent evaluation is slow
When the survey asked why enterprises distrust automated evaluation, the most common answer — 29% — was poor alignment with real-world outcomes. Bias or inconsistency followed at 21%, lack of explainability at 18%, and data leakage or privacy concerns at 17%.
Read the shape of that list. Nobody's leading objection is that evaluation costs too much or takes too long. The leading objection is that the score does not predict what happens when a real customer meets the system.
That is a validity problem, not a throughput problem. And you cannot fix a validity problem by running the invalid test more often.
Five things that change the answer
Ask for consistency, not capability. Before you accept any agent — built in-house or bought — run the same scenario ten times with varied phrasing and count how many times the business outcome was correct. Not the response. The outcome: the record that got written, the message that got sent, the state the system was left in. If a vendor can't give you that number, the number they did give you was pass@1.
Grade the outcome, not the route. There's a strong instinct to assert on the exact sequence of tool calls. Anthropic's guidance advises against it: agents regularly find valid approaches the test author never anticipated, so path-based assertions produce brittle tests that punish good behaviour. Check where the system ended up.
Let consequence set the threshold, not ambition. Drafting an internal summary or categorising a document can tolerate a wide autonomy band. Moving money, writing to a customer, changing access, deleting data — those need repeated consistency tests, a rollback path, and a named human escalation route. Autonomy should be earned per action, not granted per system.
Turn every incident into a permanent test. The failure that reached a customer is the single most valuable test case you will ever get, because it is drawn from the real distribution rather than the imagined one. It should end up in the pre-deployment suite, not just in the ticket queue.
Start smaller than you think. Teams postpone building evaluations because they assume they need hundreds of cases. Anthropic puts the useful starting point at twenty to fifty tasks drawn from real failures — and notes that evaluations get harder to build the longer you wait, because you end up reverse-engineering success criteria from a live system instead of writing them down while they're still fresh.
The acceptance question
Most acceptance conversations we walk into are still built around a yes-or-no: does it work?
For a deterministic system that question is complete. For an agent it is barely a question at all, because the honest answer is "sometimes," and everything that matters is hidden inside how often, on which tasks, and what happens on the runs where it doesn't.
So change the question. Not does it work — how many times in ten, on which tasks, and what happens the other times?
An agent that succeeds 75% of the time is a genuinely useful system if you know that's the number and you've designed the workflow around it. The same agent is a liability if you believed it was 100% because you saw it work once.
Symentic Technologies builds and integrates AI systems for businesses that need them to work on the second run as well as the first. If you're evaluating an agent — yours or a vendor's — and want a second pair of eyes on what the numbers actually mean, get in touch.
