Two questions hide inside the phrase “is it working?”.

They have almost nothing to do with each other.

Ask a simple question about a deployed AI agent — is it working — and you’ll find there are actually two questions hiding inside it.

The first is: can you even observe the agent the way your users do?

The second is: once you can, does it behave consistently and within its rules, or does it quietly come apart?

These are different problems. They require different things to solve. And the reason no one can tell you whether your agent is actually reliable is that solving one of them does nothing for the other, and almost every tool on the market solves at most one.

Problem one is reaching the real agent your users hit.

Validity: is your measurement pointed at the right thing?

Most testing is run from a datacenter. Your CI, your synthetic monitors, most eval platforms — they all originate from cloud IPs, because that’s where compute lives.

Your users don’t.

This matters more than it sounds, because the infrastructure sitting in front of a modern agent — the CDN, the WAF, the bot detection, the rate limiter — reads the origin of every request and treats it differently based on reputation. Datacenter ranges are where scrapers and abuse come from, so they get blocked, throttled, or challenged. Residential ranges are where humans are, so they get served.

We measured this across a large set of production agents. Datacenter-origin probes were blocked at a rate several times higher than residential ones. A large fraction of agents served residential traffic cleanly while turning datacenter traffic away. (The Anti-Synthetic cohort: mean per-agent 4xx of 74.2% from datacenter origins versus 22.6% from residential, across 6,228 matched agents.)

Sit with what that means for anyone testing from the cloud. Their monitoring isn’t measuring the agent their users hit. It’s either getting blocked, and reporting a healthy agent as broken, or it’s been allowlisted through the front door, in which case it’s testing a VIP path no real user travels. Either way, the number on the dashboard describes a version of the agent that doesn’t match the one in production.

This is the reachability problem. It is a question of validity: is your measurement even pointed at the right thing. And it is not solved by better scoring, smarter judges, or more test cases. It’s solved by one thing — testing from where your users actually are. For us that means a residential network across dozens of countries. There’s no clever software substitute, because the bouncer at the door is reading a physical fact about your origin.

Hard limit — inbound only

One thing this does not do, and we’re precise about it because the limit is real: reaching an agent from a residential IP changes how we reach the agent. It does not change how the agent’s own tools reach their upstreams. When an agent calls Google or an API or an MCP server, that traffic leaves from the agent’s infrastructure, not ours, and nothing outside the agent can change that. Reachability is about the inbound path — getting to the agent as a user would. It is honest about stopping there.

Problem two is trusting what you reach once you get there.

Depth: once you can see it, can you tell whether it’s behaving?

Say you’ve solved reachability. You’re reliably reaching the real, production agent from the same conditions your users do. You still don’t know if it’s any good.

Because now you have to decide whether its answers are right, and that turns out to be the harder problem, for a reason most tooling papers over: for open-ended agent output, you usually don’t have the right answer to compare against. The correct response is unknown, ambiguous, or too expensive to label at scale. This is a known, named problem in software testing — the oracle problem — and it’s exactly what makes evaluating agents hard.

So the naive approach — ask a bigger model to judge whether the answer was good — runs into a ceiling. A judge can be fooled, and worse, a judge doesn’t necessarily know what correct looks like in a specialized domain. You can’t fully test correctness from the outside without a source of truth, and anyone claiming otherwise is overselling.

But here’s what you can do without ever knowing the right answer: catch the agent contradicting itself.

If an agent tells one user the refund window is 30 days and another 60, at least one is wrong, and you proved it with no oracle at all. If you rephrase a question that shouldn’t change the answer and the reply flips — same refund policy, different numbers — that’s a failure you can see without knowing the correct policy. If it’s coherent at the first turn and falling apart by the twelfth, that’s drift you can measure directly. If it followed its policy for one user and broke it for the next, that’s a rule violation, no ground truth required.

This is the discipline of checking relationships between outputs rather than grading outputs one at a time, and it’s decades old; it exists precisely to test systems where you don’t have a reference answer. Rephrase an input that shouldn’t change the answer and the answer should hold. Change an input in a direction that should move the output and it should move the right way. When those relationships break, the agent is unreliable, and you caught it from the outside, without an oracle.

That’s the reliability problem. It’s a question of depth: once you can see the agent, can you tell whether it’s behaving. And notice it’s a completely different kind of work than reachability — it’s a measurement method, not an infrastructure asset. Being residential doesn’t grade answers. Grading answers doesn’t get you past the bouncer.

What we also run — gold and judges, scoped

We do not pretend metamorphic checks are the only evaluation we ship. Where we have known-correct prompts (gold / contract checks), we use them. Where we need semantic scoring on open answers, we run dual LLM-as-judge evaluation — knowing its ceiling, and knowing it does not replace a domain expert for stably wrong but consistent answers. Oracle-free relations (rephrase invariance, multi-turn drift, policy stress) catch self-betrayal without ground truth. Gold and judges catch more when truth is available or approximate. Name both, and you don’t oversell either.

Almost nobody solves reachability and reliability together.

Different muscles. Most tools train one.

Reachability is an infrastructure problem. Solving it means operating a real distributed residential network, which is a hard, unglamorous business most measurement companies have no interest in running.

Reliability is a measurement problem. Solving it means building consistency checks, invariant relations, drift baselines, and the honest accounting for what still needs a human expert — which is research work most infrastructure players don’t do.

They’re different muscles, and the tools in this space almost all have one. The eval platforms have scoring and run from a datacenter, so they measure the wrong agent well. The synthetic monitors reach an endpoint but only check that it responded, so they reach a real agent and learn nothing about whether it’s right. The tracing tools sit inside your pipeline, so they never traverse the front door at all and never see what a user sees.

A valid measurement needs both. You have to reach the real agent, and you have to be able to tell whether it’s behaving once you’re there. Miss the first and your numbers describe a phantom. Miss the second and you’ve confirmed the agent is reachable and nothing else.

We built both because you can’t honestly claim to validate an agent with either one alone. And we’re careful about the edges of both: residential fixes inbound reach, not the agent’s outbound tools; consistency-checking catches self-betraying failures, not the stably-wrong answer that needs a domain expert. Name the limits, and what’s left is something you can actually trust: measurement that’s pointed at the real thing, and deep enough to mean something.