AgentStatus
We tested every public MCP server we could find. Calling is solved. Delivery is not.
Read the reportEvery team running an AI agent in production has a version of this conversation on Slack, usually at an inconvenient hour:
"Is something wrong with the agent?"
"Let me check the dashboard."
"Looks fine."
"Then why did the support queue just light up?"
That conversation happens because your dashboards are telling you what your code saw, and your users are telling you what they saw. Those are not the same thing. This post is a walkthrough of a user-side monitoring pattern we recommend for every production AI agent - the same pattern Agent Status applies by default. The three failure classes below are the ones this pattern reliably catches before a customer files a ticket. The setup is portable; you can apply it to any agent you run.
Users hit your agent API, which calls the model provider. Independent third-party checks hit the same agent API from three or more geographic regions, report to a dashboard, and page on failure.
The components:
/validate endpoint on the agent that exercises the same code path as a user request.A mistake we see teams make: validating a /health route that only returns 200. That tells you the process is up. It tells you nothing about whether the agent can answer a question. Build a validate endpoint that does what a user would do - loads a model, makes a tool call, returns a structured response. Rate-limit it to yourself and skip the auth for that one route.
Each assertion catches a different failure mode. HTTP 200 catches process down. JSON parseable catches "the model returned garbage." .ok === true catches "the agent's own logic decided something went wrong." .answer contains catches "the model is alive but confused." Any single one of these misses the other three.
A model provider's p99 latency doubles for a 40-minute window. Inside-out tracing shows the same p50 it always sees; the p99 spike is masked by the average. A user-side validate, which measures every single call rather than aggregating, flags the third consecutive 6-second response and pages the on-call team. You're on a call with the provider's support team minutes later - typically well before a customer complaint lands.
A deploy routes all Singapore traffic to a US PoP through a misconfigured CDN rule. Server side, everything looks fine; there's no Singapore alerting because there are no servers there. A user-side validate from ap-southeast-1 starts returning 9-second latencies and fires minutes after the deploy. The rollback lands before users notice.
A refactor swaps the argument order on a tool. The tool itself still works; the agent just starts passing bad arguments to it, and the model handles the resulting error by silently dropping the tool call from the chain. Inside-out tracing shows success because no exception is thrown. The user-side assertion that the response contains the expected substring fails, and you catch it in the first validate after deploy.
Class 1 only shows up because you check every validate against its own assertion, not against an average. Averages hide incidents; percentiles smooth them over minutes; per-request assertions catch them in the first failed validate.
Class 2 is invisible to anyone validating only from the cloud region where the app runs. If your users are global, your validations need to be global too.
Class 3 is caught by an assertion that looks trivial - does the answer contain the expected substring - but that assertion is effectively a contract the agent has with its users. Write assertions as contracts: what a user needs to be true about every response for the agent to be useful. Then every time one fails, you know a user would have been affected.
We don't replace inside-out tracing with user-side. They answer different questions. When the user-side validate pages us, we immediately go to inside-out traces to figure out why. Without the traces, we'd know something broke but not what. Without the validations, we'd know what was slow in the server but not whether users were affected. See our user-side primer for the conceptual framing, and TTFB vs latency for why TTFB is the right metric to alert on.
We also don't try to assert on response quality using another model in the validate path. It's tempting - "grade the answer's correctness with GPT-4" - but the grader's latency becomes your validate's latency, and the grader's failure modes become yours. Keep validations deterministic.
/validate endpoint that exercises the same code path as a user call.If you want the shortcut, Agent Status runs this exact setup out of the box. Free tier covers 30 tests/month on 1 agent from 3 regions. See per-framework setup guides for LangChain, LlamaIndex, and CrewAI, or jump to the platform overview.