The common objection is that wrong answers are just noise.

In the skeptic's own voice.

There is an objection I get almost every time I explain what we do. We validate AI agents from the outside, and we check whether they stay consistent, in-policy, and stable, without needing to know the right answer in advance. And someone sharp always says: "But an agent can be wrong and still be consistent. So what is your stamp actually worth?"

They are right. And the answer to that objection is the most important thing to understand about evaluating AI agents in production, because it turns out there are two completely different ways for an agent to be wrong, and they are not caught the same way, by anyone.

The first way an agent is wrong is because it is unstable.

Most wrong agents give themselves away.

Most agents that are wrong in production are wrong because they are unstable.

The agent tells one caller the refund window is 30 days and another caller it is 60. It quotes one price in the morning and a different one that afternoon to the same question. It is fine at the first turn of a conversation and incoherent by the twelfth. It follows a policy for one user and violates it for the next.

In every one of these cases, the agent is wrong, and you can prove it is wrong without ever knowing the correct answer. If it says 30 days to one user and 60 to another for the same question, at least one is wrong. You do not need to know which. The contradiction is the proof.

This is not a niche category. It is most of what goes wrong with deployed agents, because the underlying models are non-deterministic and their behavior shifts with phrasing, context, sequence, and the world around them. The wrongness shows up as the agent disagreeing with itself.

Metamorphic testing, older than the models it now tests

There is a mature, decades-old discipline for catching exactly this. It is called metamorphic testing, first proposed by Chen and colleagues in 1998, and its entire premise is that you can verify a program's correctness by checking relationships between multiple related runs, rather than checking any single output against a known right answer. Instead of asking "is this output correct," it asks "do the outputs for related inputs satisfy a property that any correct system must have."

The formal problem it solves has a name: the oracle problem. In software testing, an oracle is the mechanism that tells you whether a given output is correct. For most conventional software you have one. For machine learning systems, and especially for LLMs, you frequently do not, the expected output is unknown, ambiguous, or too expensive to obtain at scale. Metamorphic testing was built precisely for that situation, and it has been applied to machine learning systems specifically because it lets you test them without ground-truth labels.

The relations come in a few standard shapes. Some are invariance relations: change something about the input that should not matter, rephrase the question, reorder independent details, and the output should not change. Some are directional: change the input in a way that should move the output in a known direction, raise a transaction amount and a fraud-risk score should not fall, and check the direction held. In every case, you never need the right answer. You need only the properties the right answer must obey.

An agent that contradicts itself under rephrasing fails an invariance relation. An agent whose risk score moves the wrong way fails a directional one. Both are provable failures with no oracle at all. This is the large class, and it is the class your dashboards cannot see, because the failure does not live in any single output, it lives in the relationship between two of them, and a tool watching one output at a time is structurally blind to it.

Central mechanism

If an agent contradicts itself across two related inputs, at least one answer is wrong. The contradiction is the proof. No oracle required.

The second way an agent is wrong is stable and consistent.

The hard case. The one the objection is really about.

An agent can be wrong and completely stable about it. It tells everyone, every time, in every phrasing, the same incorrect thing. It never contradicts itself. It never drifts. It breaks no rule anyone wrote down. It is confidently, consistently, rule-abidingly wrong.

This one you cannot catch from the outside without knowing the right answer. Not with metamorphic testing, because it is stable, so every relation holds. Not with rule-checking, because no rule was written for it. And here is the part the industry does not like to say out loud: nobody catches this from the outside without a source of truth. Not us. Not the eval vendors. Not the operator's own monitoring, unless their judge happens to genuinely know the domain, which is exactly what fails in specialized, high-stakes domains.

The honest thing is to name it plainly. There is a residue of failure, consistently wrong in a way nobody anticipated, that no outside-in method resolves on its own. Anyone who tells you their tool catches everything is either not thinking clearly about this or hoping you are not.

Honest disclosure

A stably-wrong agent passes every invariance relation and violates no rule. Nobody catches this from outside without ground truth. The industry rarely says this out loud.

The industry keeps missing the stably wrong failure mode.

Calibration is tractable. Competence is not.

If the second kind of wrong is the hard one, you would expect the field to be pouring its energy there. It is not. It is pouring energy into the first-order problem of making the judge better.

The dominant way to evaluate open-ended agent output today is LLM-as-a-judge: use a capable language model to grade whether an output is acceptable, an approach popularized by Zheng and colleagues in 2023 as a scalable substitute for human annotators. It works well enough to have become the default. And it has a well-documented set of biases that the research community has spent years characterizing: position bias, where the judge favors an answer based on its order rather than its content; verbosity bias, where longer answers are rated higher regardless of quality; and self-preference bias, where a judge rates outputs from its own model family more favorably. The consensus in that literature is worth stating directly: LLM judges should be treated as fallible measurement instruments, not neutral arbiters.

Nearly all of that work is aimed at making the judge less biased, better calibrated. Swap judges, randomize order, penalize length, use panels. These are real improvements, and they address what I would call the calibration problem: the judge is skewed, and you can de-skew it.

But calibration is not the deep problem. The deep problem is competence: whether the judge knows what correct even looks like in the domain it is grading. A judge can be perfectly unbiased and still have no reliable model of correctness for HVAC diagnostics, or insurance claims, or a drug interaction. And no amount of de-biasing repairs a judge that lacks the underlying knowledge. Two judges that both lack the competence and agree do not produce competence between them, they produce confidently shared ignorance, which reads as signal and is worse than an honest "I do not know."

So the distinction that matters, and the one I do not think the field has named: calibration problems shrink with better technique. Competence problems do not. The industry is largely working the calibration axis, which is the tractable one, and mostly leaving the competence ceiling, and the stably-wrong residue that sits above it, untouched.

Honest evaluation names both ways an answer can be wrong.

Name it. Shrink it. Bound what remains.

If you cannot catch the stably-wrong residue from outside, the answer is not to pretend you can. It is to handle it in the open.

Name it precisely: the class of failure your method cannot catch. Stably wrong. Consistent across rephrasings. In-policy against every written rule. Not detectable from outside without ground truth.

Shrink it over time by converting each stably-wrong failure into a rule the moment it is discovered, so what fooled you once becomes a check that fires forever after. Every escape becomes a new invariant.

Bound what remains with sampled expert review, a small, domain-expert-graded sample that gives you a correctness rate with a confidence interval, rather than a false claim of total coverage.

That posture, catch every failure that betrays itself through instability or rule-violation, and precisely scope the small residue that does not, is both the honest description of what is possible and, I would argue, the stronger commercial position. A technical buyer can tell the difference between a vendor who claims to catch everything and one who says: here is exactly the class of failure that still needs your experts, here is how small we have made it, and here is how we make it smaller every quarter.

Our position

We catch every failure that gives itself away. We are honest about the residue that does not. And we shrink the residue every time an expert catches one.

Closing: oracle-free checks and expert review both still matter.

So when someone tells me an agent can be wrong but stable, I agree. That is the one kind of wrong we do not catch from outside, and neither does anyone else without ground truth. What we catch is the other kind, the far larger kind, the agent that betrays its own wrongness by contradicting itself, drifting, or breaking its rules. That is most of it. And the rest, we name instead of hide.