Test in the Wild

To everyone watching agents go to work -

On February 14, 2024, a chatbot at Air Canada told a grieving passenger about a bereavement fare. The fare did not exist. The dashboard was green the whole time. The agent had been wrong for months. A judge later told Air Canada it had to honour the policy it had invented, and the headlines moved on. The thing the headlines missed is that nothing in Air Canada's monitoring stack would have caught the failure even if every engineer had been awake. The agent was up. The agent was running. The agent was wrong.

We are writing this letter because we have come to believe one thing about the agent era, and we are betting the company on it.

Everything else, uptime, traces, latency, eval scores, is a proxy. In the old world, the proxies were close enough that nobody got hurt. In the world of agents, they aren't, and people are getting hurt. This letter is about the gap between what the dashboard reports and what the user actually receives, and what we are doing about it.

I.The end of seen-equals-true

For thirty years, software either worked or it didn't. The line was up or down. The page loaded or it didn't. Monitoring was a binary affair, and a whole industry, Datadog, New Relic, Pingdom, Grafana, Catchpoint, grew rich on a simple assumption: if I can see the system, I know what users see. It was a good assumption. It was true.

It is no longer true.

What is coming online now is not better software. It is a different kind of software: probabilistic, tool-dependent, behavior-conditional on the network it is invoked from. Agents are no longer demos. They are answering customers, generating code, approving loans, triaging patients. The world is wiring them into workflows that used to require humans. The assumption every previous monitoring tool was built on, seen equals true, has stopped holding.

II.The inside-out stack

We owe the existing stack the respect of taking it seriously. Datadog and its peers are not lazy products. They were built, carefully, for a world where systems were deterministic and observable from inside. Instrument the request. Trace the call. Capture the log. The dashboard, in that world, is the gospel.

The trouble is that inside-out monitoring sees what the agent reports about itself. If the agent reports scenario=hallucinate, http_status=200, response_length=30, the inside-out tool dutifully captures all of that. It trusts the agent's self-report. It is designed to trust the self-report. That is what observability is.

In a deterministic system, trusting the self-report is fine. In an agent system, the self-report is the thing you most cannot trust. The dashboard says 200 OK; the agent has just told a customer their flight is confirmed when it isn't.

III.The 200 OK lie

Three things changed at once.

Agents are non-deterministic. The same prompt to the same agent can return two different answers in two minutes. The dashboard sees a 200; it cannot see correctness.

Agents are tool-dependent. They call APIs, fetch documents, hit search engines. Every external dependency is a place the dashboard cannot follow. A travel API silently changing its policy in São Paulo lands in a customer's chat as confidently wrong English.

Agents are origin-dependent. The same agent answers differently in New York and Lagos because the network it is invoked from is different. Different ISPs, different IPs, different rate limits, different filters, different last-mile congestion. We have known this about the internet for thirty years. We forgot the lesson the moment we put a model behind an API.

Of those three, origin-dependence is the one the industry has not absorbed. Non-determinism shows up in every blog post. Tool dependency is on every product roadmap. Where the test runs from gets ignored. We think it is the most important of the three.

So you get the 200 OK lie. The dashboard says green. The user files a ticket. The court rules against the airline. The agent had been wrong for months.

IV.Test in the wild

The category that needs to exist is not better observability and not bigger evals. It is something else: user-side validation.

User-side validation is the layer that does not trust the self-report. It tests the agent the way the user encounters it, from the user's network, on the user's device, scored the way a human would score the answer, and reports what is actually true. It is not synthetic monitoring; cloud validations do not see what users see. It is not observability; observability trusts the agent's self-report, and the self-report is exactly the thing that has stopped being trustworthy.

We had reasons to believe this before we had proof of it. We have proof now.

In our most recent study, we ran 1.55 million datacenter validations and 4.85 million residential validations against the same 6,228 production agents. Same prompts. Same endpoints. The two validate networks did not agree. The gap was not a calibration error, it was structural. Agents that pass cleanly from a datacenter fail in real users' hands, and the other way around. You cannot extrapolate one to the other. The internet does not let you. The packets do not travel the same path.

This is not new. Akamai and Cloudflare were built because the page on the server was not the page in the browser. Cellular carriers drove signal-coverage cars because a tower in Manhattan could not predict a call in the Berkshires. RIPE Atlas deployed twelve thousand validations in volunteers' homes because the only honest way to measure the internet was from inside houses. Every previous chapter learned the same lesson: truth at the edge. The validate must execute where the node is. There is no shortcut.

We built Agent Status on this lesson. Fabric is a network of consumer devices, laptops, workstations, servers, real machines on real residential connections, running validation workloads from where users actually are. Today it spans more than 400 nodes across 70 countries. World State is the capability that turns Fabric into a planetary sensor: point it at any URL, any agent endpoint, and it tells you what the world sees, from the world. We call the act Test in the Wild, because that is what it is.

V.From the field

User-side validation has three places to live, and we are building for all three.

In pre-production, the test runs on every PR. AgentDiff is a GitHub App that fires multi-region residential validations against your agent on every code change, and posts pass/fail per region in the review. Catch the regression before merge.

In production, the tests run continuously. Agent Status is the platform: scheduled validation against your live agents from the wild, with LLM-as-judge evaluation, drift attribution, and alerting that fires before the support ticket does.

And in the longitudinal record, the data compounds. The behavioral dataset that user-side validation produces is something insurance underwriters, certification bodies, governance teams, and standards organizations need and cannot collect for themselves. Our Data Partners program licenses the record under signed contracts. Every observation Fabric makes sharpens the next one.

Three motions on one engine. The engine is user-side validation. The motions are where the world pays for it.

We are starting now because the window is open now. Agents are moving from pilot to production faster than the tooling around them is maturing. Enterprise blind spots are surfacing in headlines. The downstream categories, agent insurance, agent governance, agent certification, are forming, and they need a record. The thing that gets built in this window is the thing the field uses for the next decade.

We intend it to be ours.

If you ship agents, this is for you. The user is where the truth is. We will be there.