Every team running an AI agent in production has a version of this conversation on Slack, usually at an inconvenient hour:
"Is something wrong with the agent?"
"Let me check the dashboard."
"Looks fine."
"Then why did the support queue just light up?"
That conversation happens because your dashboards are telling you what your code saw, and your users are telling you what they saw. Those are not the same thing. This post is a walkthrough of an outside-in monitoring pattern we recommend for every production AI agent - the same pattern Agent Status applies by default. The three failure classes below are the ones this pattern reliably catches before a customer files a ticket. The setup is portable; you can apply it to any agent you run.
The setup in one diagram
Users hit your agent API, which calls the model provider. Outside-in validations hit the same agent API from three or more geographic regions, report to a dashboard, and page on failure.
The components:
- A dedicated
/validateendpoint on the agent that exercises the same code path as a user request. - Validations firing from at least three regions at the tightest cadence your plan supports (paid tiers on Agent Status run as short as 5 minutes).
- Four assertions on every validate response.
- Alerts to the same Slack channel as your other paging.
The validate endpoint
A mistake we see teams make: validating a /health route that only returns 200. That tells you the process is up. It tells you nothing about whether the agent can answer a question. Build a validate endpoint that does what a user would do - loads a model, makes a tool call, returns a structured response. Rate-limit it to yourself and skip the auth for that one route.
Why four assertions and not one
Each assertion catches a different failure mode. HTTP 200 catches process down. JSON parseable catches "the model returned garbage." .ok === true catches "the agent's own logic decided something went wrong." .answer contains catches "the model is alive but confused." Any single one of these misses the other three.
The three failure classes it catches
Class 1 - silent model degradation
A model provider's p99 latency doubles for a 40-minute window. Inside-out tracing shows the same p50 it always sees; the p99 spike is masked by the average. An outside-in validate, which measures every single call rather than aggregating, flags the third consecutive 6-second response and pages the on-call team. You're on a call with the provider's support team minutes later - typically well before a customer complaint lands.
Class 2 - a regional routing bug
A deploy routes all Singapore traffic to a US PoP through a misconfigured CDN rule. Server side, everything looks fine; there's no Singapore alerting because there are no servers there. An outside-in validate from ap-southeast-1 starts returning 9-second latencies and fires minutes after the deploy. The rollback lands before users notice.
Class 3 - a tool-call regression
A refactor swaps the argument order on a tool. The tool itself still works; the agent just starts passing bad arguments to it, and the model handles the resulting error by silently dropping the tool call from the chain. Inside-out tracing shows success because no exception is thrown. The outside-in assertion that the response contains the expected substring fails, and you catch it in the first validate after deploy.
Lessons from those three
Measure every call, not aggregates
Class 1 only shows up because you check every validate against its own assertion, not against an average. Averages hide incidents; percentiles smooth them over minutes; per-request assertions catch them in the first failed validate.
Validation from where your users are
Class 2 is invisible to anyone validating only from the cloud region where the app runs. If your users are global, your validations need to be global too.
Assertions are contracts, not tests
Class 3 is caught by an assertion that looks trivial - does the answer contain the expected substring - but that assertion is effectively a contract the agent has with its users. Write assertions as contracts: what a user needs to be true about every response for the agent to be useful. Then every time one fails, you know a user would have been affected.
What we're not doing (and why)
We don't replace inside-out tracing with outside-in. They answer different questions. When the outside-in validate pages us, we immediately go to inside-out traces to figure out why. Without the traces, we'd know something broke but not what. Without the validations, we'd know what was slow in the server but not whether users were affected. See our outside-in primer for the conceptual framing, and TTFB vs latency for why TTFB is the right metric to alert on.
We also don't try to assert on response quality using another model in the validate path. It's tempting - "grade the answer's correctness with GPT-4" - but the grader's latency becomes your validate's latency, and the grader's failure modes become yours. Keep validations deterministic.
How to copy this for your agent
- Add a
/validateendpoint that exercises the same code path as a user call. - Define 3–5 assertions on its response - include at least one about content, not just shape.
- Set up outside-in monitoring from three regions covering your user base.
- Route alerts to the same channel as your existing paging.
- When a validate fails, look at inside-out traces to find the root cause - don't try to debug from the validate alone.
If you want the shortcut, Agent Status runs this exact setup out of the box. Free tier covers 30 tests/month on 1 agent from 3 regions. See per-framework setup guides for LangChain, LlamaIndex, and CrewAI, or jump to the platform overview.
