Validate chat agents
the way users hit them.
Every layer of your agent, checked from where your users actually are. Scroll to assemble the stack.
Can people reach it?
Hit from real home networks in the regions you care about — not a 200 OK from your office.
Your dashboard can be completely fine.Your users can still get wrong answer.
The most expensive failures in production AI today are the ones every existing monitoring tool would mark green. The server responded, the latency was good, the error rate was zero - and the user was lied to.
- HTTP status
- 200 OK
- p95 response latency
- 284 ms
- Error rate (5xx)
- 0.02 %
- Token throughput
- 432 / s
- Model availability
- 99.99 %
- Eval set pass@1
- 94.2 %
Hey, I bought the wrong size - can I return it for store credit?
The observability era sees a successful request. The customer sees a fabricated refund. Both are correct.
What we validate
Uptime is table stakes. We watch the rest too.
Reliability, consistency, and robustness. From home networks, on a schedule. Plain verdicts you can explain to your boss.
Reliability
Does it keep working, stay fast, and behave the same over time?
Can people reach it?
Not just 200 OK from your office. We hit it from home networks where your users actually are.
time to first byte
Fast enough to feel alive
Slow first replies feel broken even when the answer eventually shows up. We watch time-to-first-byte, not just total wait.
From home networks
Same agent, different country, different result. Geo blocks and CDN quirks show up here first.
Keeps working, run after run
One green check means nothing. We track whether it stays healthy across dozens of scheduled runs.
week 6 · -9pts vs baseline
When behavior starts to drift
Slow leaks matter. Pass rates, latency, and answer patterns shifting week over week get flagged before users notice.
Answer quality
Not just reachable. Actually helpful, shaped right, and checked more than once.
Are the answers good?
We read replies like a human would: pass, degraded, or inconclusive, with a short explanation you can act on.
Two reviewers had to agree
When there is no single right string, two independent reviewers weigh in. Disagreement means we say inconclusive, not a false alarm.
Your example questions
Bring your own scenarios: refunds, escalations, edge cases you already worry about.
Questions for what it actually does
We watch how the agent behaves in the wild and keep testing with fresh example questions that match its real job.
Exact shape, every time
JSON APIs and structured replies should look like you promised. We check the shape, not just the vibe.
Extra eyes when it looks fishy
Automated checks miss nuance. Flag samples for a human look, then turn repeat mistakes into a rule.
Conversations
Does it finish the job? Real goals, pursued over several messages, judged on the outcome.
A user with a real goal
We give a simulated user something real to want, like a refund, a booking, or a fix, and let them pursue it over several messages, clarifying and pushing back like a real person.
Did they get what they came for?
When the conversation ends, an independent reviewer reads the whole transcript and makes one call: goal achieved or not.
We don't take its word for it
If the agent claims success, we double-check the claim against outside evidence where it exists, such as a cited page, a structured field, or a second reviewer, before counting it.
Consistency
Same situation, same story. No flip-flopping when wording, mood, or follow-ups change.
Same question, different words
Rephrased prompts should not flip the answer. We flag replies that swing for no good reason.
Calm ask vs panicked ask
A billing FAQ and a locked-out account are not the same urgency. Good agents step up when stakes rise.
Follow-ups still line up
Ask A, then B, then C. Later answers should not contradict what the agent said two turns ago.
Robustness
Tools, streams, rules, and people trying to break it.
Do tools get called?
Send questions that should trigger an action. Catch broken integrations before customers hit them.
MCP servers
Discover tools and resources, run calls, and check the outputs for agents wired through MCP.
4/4 chunks · closed cleanly
Streaming that finishes
Streaming endpoints can hang, stall, or never complete. We catch that, not just the final text blob.
Your safety rules
Tell us what the agent must never do, and what it should still do. We check both sides.
When someone tries to trick it
Pushy, weird, or adversarial prompts. Does the agent stay on policy or fold?
When it breaks, you know
Alerts, deploy gates, and one place to see it all.
Alerts when it breaks
Slack, webhooks, PagerDuty, with enough context to fix it, not just a red dot.
agent failed outside-in check
Block bad deploys before ship
Wire us into CI so a broken agent does not merge just because the unit tests passed.
One dashboard
Every check rolls up into a simple grid: what is OK, what is still collecting, and what needs attention.
FAQ
Common questions
| Question | Answer |
|---|---|
| I'm already using LangSmith or Datadog. Why do I need you? | They watch from inside your stack or from cloud servers. We hit your agent the way a user would, from a home network, and tell you if the answer is actually good. |
| Our uptime monitor says green. Why would the agent still be broken? | 200 OK doesn't mean the reply makes sense. Your agent can be reachable and still hallucinate, call the wrong tool, or give a different answer every time someone asks. |
| We run evals in CI. Isn't that enough? | CI evals usually run once, from one place, on perfect internet, with prompts you wrote. We keep testing from home networks in the regions you care about, after deploy too. |
| We test from our office and it works. Why would users see something different? | Your office isn't your user. Geo blocks, CDN quirks, and residential routing change what people actually get. We test from where they are. |
| How is this different from checking for specific keywords? | Good answers don't always use the same words. We judge whether the reply makes sense, so you catch real failures without crying wolf on every rephrase. |
| Do you test tool calls and MCP servers? | Tool calls in chat, yes — send questions that should trigger an action and we verify the integration. For MCP servers as a primary surface, see MCP validation — discovery, tools/call, and workflow chains probed directly. |
| Do I need to install anything? | No. Point us at your endpoint. Everything runs from the outside. Nothing to embed in your agent. |
| What do we actually get back? | A plain verdict on every run: pass, degraded, or inconclusive. Plus a dashboard grid showing what's healthy, what's still collecting, and what needs attention. |
| Can we monitor chat and voice in one place? | Yes. Chat and voice live on the same agent in one workspace. Same plain verdicts, different transport. |
