Validate chat agents
the way users hit them.

Every layer of your agent, checked from where your users actually are. Scroll to assemble the stack.

Reachability
can users reach it
Speed
time to first byte
829ms
Outcome verification
did it finish the job
PASS
Consistency
same story every time
Drift
holding over time
Layer 1 / 5

Can people reach it?

Hit from real home networks in the regions you care about — not a 200 OK from your office.

Five layers, one verdict — reachable, fast, correct, consistent, and holding steady over time.

Your dashboard can be completely fine.Your users can still get wrong answer.

The most expensive failures in production AI today are the ones every existing monitoring tool would mark green. The server responded, the latency was good, the error rate was zero - and the user was lied to.

The inside-out view
all systems healthy
HTTP status
200 OK
p95 response latency
284 ms
Error rate (5xx)
0.02 %
Token throughput
432 / s
Model availability
99.99 %
Eval set pass@1
94.2 %
J
Jess

Hey, I bought the wrong size - can I return it for store credit?

The observability era sees a successful request. The customer sees a fabricated refund. Both are correct.

What we validate

Uptime is table stakes. We watch the rest too.

Two pillars: reachability from home networks, then outcome verification — did it finish the job? Plain verdicts you can explain to your boss.

Reachability

Did the user path open from real home networks — transport, auth, uptime — not AWS synthetic alone?

useragent

Can people reach it?

Not just 200 OK from your office. We hit it from home networks where your users actually are.

LatencyFAST
829ms

time to first byte

Fast enough to feel alive

Slow first replies feel broken even when the answer eventually shows up. We watch time-to-first-byte, not just total wait.

Jurisdictions100+
AUAustralia
SESweden
PLPoland
DKDenmark

From home networks

Same agent, different country, different result. Geo blocks and CDN quirks show up here first.

Scheduled runs28/30
ABC-009Running
ABC-008Pass
ABC-007Pass
ABC-006Fail
ABC-005Pass

Keeps working, run after run

One green check means nothing. We track whether it stays reachable across dozens of scheduled runs.

Pass rate · 6 wkDRIFT

week 6 · -9pts vs baseline

When the path starts to fail

Slow leaks matter. Connect success, latency, and auth failures shifting week over week get flagged before users notice.

Outcome verification

Did it finish the job? Not whether it reused the same words.

Answer reviewPASS
"How do I reset my password?"

Did it finish the job?

We score outcomes — pass, degraded, or inconclusive — with a short explanation you can act on. Fluent wrong still fails.

A
B
BOTH AGREE

Optional corroboration

When there is no single right string, independent reviewers can weigh in as a second opinion. Disagreement means inconclusive, not a false alarm.

Refund my last order, please.
Reschedule for next Tuesday.
Talk to a human, now.

Your example jobs

Bring your own scenarios: refunds, escalations, cart paths, sales goals you already worry about.

Generated testsFRESH
cancel flowADDED
refund edge caseADDED
angry customerQUEUED

Domain job anchors

Where configured: must_stay / topic / progress anchors for chat sales, side-effect checks for cart agents — job truth, not paraphrase match.

Response shapeSHAPE OK
{
"status": "ok",
"eta_days": 3
}

Exact shape, every time

JSON APIs and structured replies should look like you promised. We check the shape, not just the vibe.

Flagged sampleREVIEW
AGREENEW RULE

Extra eyes when it looks fishy

Automated checks miss nuance. Flag samples for a human look, then turn repeat mistakes into a rule.

Conversations

Does it finish the job? Real goals, pursued over several messages, judged on the outcome.

Simulated userGOAL: REFUND
I need a refund.Sure — order number?#1042-BDone. 3–5 days.

A user with a real goal

We give a simulated user something real to want, like a refund, a booking, or a fix, and let them pursue it over several messages, clarifying and pushing back like a real person.

Full transcriptREVIEWED
GOAL ACHIEVED

Did they get what they came for?

When the conversation ends, an independent reviewer reads the whole transcript and makes one call: goal achieved or not.

claim
evidence
HELD UP

We don't take its word for it

If the agent claims success, we double-check the claim against outside evidence where it exists, such as a cited page, a structured field, or a second reviewer, before counting it.

Job stability

Same job under rephrase, stakes, and follow-ups. We score whether the outcome holds — not whether the prose matched.

Same intentNO SWING
"Cancel my plan"SAME
"I want out, please"SAME

Same job, different ask

Rephrased prompts should not flip the job outcome. We flag when the substance swings for no good reason — not when the wording differs.

Stakes checkSTEPPED UP
Just curious about billingFAQ
I'm locked out — help!ESCALATE

Calm ask vs panicked ask

A billing FAQ and a locked-out account are not the same urgency. Good agents step up when stakes rise.

1
2
3
ask A → then B → then CALIGNED

Follow-ups still line up

Ask A, then B, then C. Later turns should not undo the job the agent already committed to.

Robustness

Tools, streams, rules, and people trying to break it.

Tool trigger
user: any flights to SFO?
tool_call → search_flights ✓

Do tools get called?

Send questions that should trigger an action. Catch broken integrations before customers hit them.

listcallreadfetch

MCP servers

Discover tools and resources, run calls, and check the outputs for agents wired through MCP.

StreamCOMPLETE

4/4 chunks · closed cleanly

Streaming that finishes

Streaming endpoints can hang, stall, or never complete. We catch that, not just the final text blob.

schedule callbackALLOW
medical adviceBLOCK

Your safety rules

Tell us what the agent must never do, and what it should still do. We check both sides.

ignore your rules and…
HELD THE LINE

When someone tries to trick it

Pushy, weird, or adversarial prompts. Does the agent stay on policy or fold?

When it breaks, you know

Alerts, deploy gates, and one place to see it all.

AlertsFIRED
SlackSENT
WebhookSENT
PagerDutyPAGED

Alerts when it breaks

Slack, webhooks, PagerDuty, with enough context to fix it, not just a red dot.

Pull requestCI
featuremain
MERGE BLOCKED

agent failed outside-in check

Block bad deploys before ship

Wire us into CI so a broken agent does not merge just because the unit tests passed.

All checksLIVE

One dashboard

Every check rolls up into a simple grid: what is OK, what is still collecting, and what needs attention.

FAQ

Common questions

QuestionAnswer
I'm already using LangSmith or Datadog. Why do I need you?They watch from inside your stack or from cloud servers. We hit your agent the way a user would, from a home network, and tell you if the answer is actually good.
Our uptime monitor says green. Why would the agent still be broken?200 OK doesn't mean the reply makes sense. Your agent can be reachable and still hallucinate, call the wrong tool, or give a different answer every time someone asks.
We run evals in CI. Isn't that enough?CI evals usually run once, from one place, on perfect internet, with prompts you wrote. We keep testing from home networks in the regions you care about, after deploy too.
We test from our office and it works. Why would users see something different?Your office isn't your user. Geo blocks, CDN quirks, and residential routing change what people actually get. We test from where they are.
How is this different from checking for specific keywords?We don't score whether the chatbot reused the same words. Outcome verification means did it finish the job — scenario, compositional, safety, and stability checks; job anchors and side-effects where they exist. Paraphrase of a correct job is fine; a fluent wrong job is not.
Do you test tool calls and MCP servers?Tool calls in chat, yes — send questions that should trigger an action and we verify the integration. For MCP servers as a primary surface, see MCP validation — discovery, tools/call, and workflow chains probed directly.
Do I need to install anything?No. Point us at your endpoint. Everything runs from the outside. Nothing to embed in your agent.
What do we actually get back?A plain verdict on every run: pass, degraded, or inconclusive. Plus a dashboard grid showing what's healthy, what's still collecting, and what needs attention.
Can we monitor chat and voice in one place?Yes. Chat and voice live on the same agent in one workspace. Same plain verdicts, different transport.

See your agent the way the world sees it.

Start free