Validate chat agents
the way users hit them.
Every layer of your agent, checked from where your users actually are. Scroll to assemble the stack.
Can people reach it?
Hit from real home networks in the regions you care about — not a 200 OK from your office.
Your dashboard can be completely fine.Your users can still get wrong answer.
The most expensive failures in production AI today are the ones every existing monitoring tool would mark green. The server responded, the latency was good, the error rate was zero - and the user was lied to.
- HTTP status
- 200 OK
- p95 response latency
- 284 ms
- Error rate (5xx)
- 0.02 %
- Token throughput
- 432 / s
- Model availability
- 99.99 %
- Eval set pass@1
- 94.2 %
Hey, I bought the wrong size - can I return it for store credit?
The observability era sees a successful request. The customer sees a fabricated refund. Both are correct.
What we validate
Uptime is table stakes. We watch the rest too.
Two pillars: reachability from home networks, then outcome verification — did it finish the job? Plain verdicts you can explain to your boss.
Reachability
Did the user path open from real home networks — transport, auth, uptime — not AWS synthetic alone?
Can people reach it?
Not just 200 OK from your office. We hit it from home networks where your users actually are.
time to first byte
Fast enough to feel alive
Slow first replies feel broken even when the answer eventually shows up. We watch time-to-first-byte, not just total wait.
From home networks
Same agent, different country, different result. Geo blocks and CDN quirks show up here first.
Keeps working, run after run
One green check means nothing. We track whether it stays reachable across dozens of scheduled runs.
week 6 · -9pts vs baseline
When the path starts to fail
Slow leaks matter. Connect success, latency, and auth failures shifting week over week get flagged before users notice.
Outcome verification
Did it finish the job? Not whether it reused the same words.
Did it finish the job?
We score outcomes — pass, degraded, or inconclusive — with a short explanation you can act on. Fluent wrong still fails.
Optional corroboration
When there is no single right string, independent reviewers can weigh in as a second opinion. Disagreement means inconclusive, not a false alarm.
Your example jobs
Bring your own scenarios: refunds, escalations, cart paths, sales goals you already worry about.
Domain job anchors
Where configured: must_stay / topic / progress anchors for chat sales, side-effect checks for cart agents — job truth, not paraphrase match.
Exact shape, every time
JSON APIs and structured replies should look like you promised. We check the shape, not just the vibe.
Extra eyes when it looks fishy
Automated checks miss nuance. Flag samples for a human look, then turn repeat mistakes into a rule.
Conversations
Does it finish the job? Real goals, pursued over several messages, judged on the outcome.
A user with a real goal
We give a simulated user something real to want, like a refund, a booking, or a fix, and let them pursue it over several messages, clarifying and pushing back like a real person.
Did they get what they came for?
When the conversation ends, an independent reviewer reads the whole transcript and makes one call: goal achieved or not.
We don't take its word for it
If the agent claims success, we double-check the claim against outside evidence where it exists, such as a cited page, a structured field, or a second reviewer, before counting it.
Job stability
Same job under rephrase, stakes, and follow-ups. We score whether the outcome holds — not whether the prose matched.
Same job, different ask
Rephrased prompts should not flip the job outcome. We flag when the substance swings for no good reason — not when the wording differs.
Calm ask vs panicked ask
A billing FAQ and a locked-out account are not the same urgency. Good agents step up when stakes rise.
Follow-ups still line up
Ask A, then B, then C. Later turns should not undo the job the agent already committed to.
Robustness
Tools, streams, rules, and people trying to break it.
Do tools get called?
Send questions that should trigger an action. Catch broken integrations before customers hit them.
MCP servers
Discover tools and resources, run calls, and check the outputs for agents wired through MCP.
4/4 chunks · closed cleanly
Streaming that finishes
Streaming endpoints can hang, stall, or never complete. We catch that, not just the final text blob.
Your safety rules
Tell us what the agent must never do, and what it should still do. We check both sides.
When someone tries to trick it
Pushy, weird, or adversarial prompts. Does the agent stay on policy or fold?
When it breaks, you know
Alerts, deploy gates, and one place to see it all.
Alerts when it breaks
Slack, webhooks, PagerDuty, with enough context to fix it, not just a red dot.
agent failed outside-in check
Block bad deploys before ship
Wire us into CI so a broken agent does not merge just because the unit tests passed.
One dashboard
Every check rolls up into a simple grid: what is OK, what is still collecting, and what needs attention.
FAQ
Common questions
| Question | Answer |
|---|---|
| I'm already using LangSmith or Datadog. Why do I need you? | They watch from inside your stack or from cloud servers. We hit your agent the way a user would, from a home network, and tell you if the answer is actually good. |
| Our uptime monitor says green. Why would the agent still be broken? | 200 OK doesn't mean the reply makes sense. Your agent can be reachable and still hallucinate, call the wrong tool, or give a different answer every time someone asks. |
| We run evals in CI. Isn't that enough? | CI evals usually run once, from one place, on perfect internet, with prompts you wrote. We keep testing from home networks in the regions you care about, after deploy too. |
| We test from our office and it works. Why would users see something different? | Your office isn't your user. Geo blocks, CDN quirks, and residential routing change what people actually get. We test from where they are. |
| How is this different from checking for specific keywords? | We don't score whether the chatbot reused the same words. Outcome verification means did it finish the job — scenario, compositional, safety, and stability checks; job anchors and side-effects where they exist. Paraphrase of a correct job is fine; a fluent wrong job is not. |
| Do you test tool calls and MCP servers? | Tool calls in chat, yes — send questions that should trigger an action and we verify the integration. For MCP servers as a primary surface, see MCP validation — discovery, tools/call, and workflow chains probed directly. |
| Do I need to install anything? | No. Point us at your endpoint. Everything runs from the outside. Nothing to embed in your agent. |
| What do we actually get back? | A plain verdict on every run: pass, degraded, or inconclusive. Plus a dashboard grid showing what's healthy, what's still collecting, and what needs attention. |
| Can we monitor chat and voice in one place? | Yes. Chat and voice live on the same agent in one workspace. Same plain verdicts, different transport. |
