AgentStatus × Cognigy partner brief
User-side validation for Cognigy AI agents.
Seven checks from real home networks, status through alerts, alongside Cognigy Insights, OData, and Live Agent. We complement inside-out visibility; we do not replace it. No instrumentation on your stack. Synthetic validations only, no production transcripts.
What we understand about Cognigy
Inside-out visibility
Cognigy.AI spans build, deploy, analyze, and optimize across voice, chat, and messaging. Cognigy Insights delivers conversation analytics and NLU performance. OData API feeds BI teams raw conversation data. Live Agent handles escalation; Agent Copilot supports agents in real time.
For user-side validation, Cognigy's REST Endpoint is the natural surface: POST with userId, sessionId, and Bearer token, the same path a channel integration would use.
What we validate
Uptime is table stakes. We watch the rest too.
Reachability from home networks, then outcome verification - did it finish the job? Plain verdicts you can explain to your boss.
Reachability
Did the user path open from real home networks — transport, auth, uptime — not AWS synthetic alone?
Can people reach it?
Not just 200 OK from your office. We hit it from home networks where your users actually are.
time to first byte
Fast enough to feel alive
Slow first replies feel broken even when the answer eventually shows up. We watch time-to-first-byte, not just total wait.
From home networks
Same agent, different country, different result. Geo blocks and CDN quirks show up here first.
Keeps working, run after run
One green check means nothing. We track whether it stays reachable across dozens of scheduled runs.
week 6 · -9pts vs baseline
When the path starts to fail
Slow leaks matter. Connect success, latency, and auth failures shifting week over week get flagged before users notice.
Outcome verification
Did it finish the job? Not whether it reused the same words.
Did it finish the job?
We score outcomes — pass, degraded, or inconclusive — with a short explanation you can act on. Fluent wrong still fails.
Optional corroboration
When there is no single right string, independent reviewers can weigh in as a second opinion. Disagreement means inconclusive, not a false alarm.
Your example jobs
Bring your own scenarios: refunds, escalations, cart paths, sales goals you already worry about.
Domain job anchors
Where configured: must_stay / topic / progress anchors for chat sales, side-effect checks for cart agents — job truth, not paraphrase match.
Exact shape, every time
JSON APIs and structured replies should look like you promised. We check the shape, not just the vibe.
Extra eyes when it looks fishy
Automated checks miss nuance. Flag samples for a human look, then turn repeat mistakes into a rule.
Conversations
Does it finish the job? Real goals, pursued over several messages, judged on the outcome.
A user with a real goal
We give a simulated user something real to want, like a refund, a booking, or a fix, and let them pursue it over several messages, clarifying and pushing back like a real person.
Did they get what they came for?
When the conversation ends, an independent reviewer reads the whole transcript and makes one call: goal achieved or not.
We don't take its word for it
If the agent claims success, we double-check the claim against outside evidence where it exists, such as a cited page, a structured field, or a second reviewer, before counting it.
Job stability
Same job under rephrase, stakes, and follow-ups. We score whether the outcome holds — not whether the prose matched.
Same job, different ask
Rephrased prompts should not flip the job outcome. We flag when the substance swings for no good reason — not when the wording differs.
Calm ask vs panicked ask
A billing FAQ and a locked-out account are not the same urgency. Good agents step up when stakes rise.
Follow-ups still line up
Ask A, then B, then C. Later turns should not undo the job the agent already committed to.
Robustness
Tools, streams, rules, and people trying to break it.
Do tools get called?
Send questions that should trigger an action. Catch broken integrations before customers hit them.
MCP servers
Discover tools and resources, run calls, and check the outputs for agents wired through MCP.
4/4 chunks · closed cleanly
Streaming that finishes
Streaming endpoints can hang, stall, or never complete. We catch that, not just the final text blob.
Your safety rules
Tell us what the agent must never do, and what it should still do. We check both sides.
When someone tries to trick it
Pushy, weird, or adversarial prompts. Does the agent stay on policy or fold?
When it breaks, you know
Alerts, deploy gates, and one place to see it all.
Alerts when it breaks
Slack, webhooks, PagerDuty, with enough context to fix it, not just a red dot.
agent failed user-side check
Block bad deploys before ship
Wire us into CI so a broken agent does not merge just because the unit tests passed.
One dashboard
Every check rolls up into a simple grid: what is OK, what is still collecting, and what needs attention.
Architecture
How validation runs
Fabric nodes execute transport validations: reachability, node-side gold checks, streaming latency, and inbound MCP/tools discovery conformance against your REST endpoint (reachability of the MCP surface — not a claim that residential probes change agent tool egress). Raw results return to the AgentStatus backend, which schedules cycles, runs portfolio validations and adaptive scenarios, and applies the instrument layer on top of node evidence.
That instrument layer is not a single semantic pass. It includes Scenario, Compositional, Safety, and Stability checks; Consistency and Drift; outcome verification where configured; and statistical rollups: pass^k stability, drift baselines, alerts, and explain traces in the dashboard. Your Cognigy platform (Insights, OData, transcripts) stays inside your boundary. We do not pull from it.
What runs on nodes vs. the backend
Nodes execute transport. The backend runs instruments and rollups. Fabric nodes sit on residential networks in 70 countries and hit your REST Endpoint the way a real channel would. The backend receives those raw results and runs the multi-instrument scoring layer plus portfolio statistics. Semantic review is one instrument among several, not the whole system.
Results are verdict metadata, latency, region, and evidence snippets, not production conversation exports.
Data privacy
Data boundaries
Most teams ask whether they must pass customer conversation data to AgentStatus. You do not. Pilot default is Cognigy sandbox only: synthetic prompts you define, validation credentials you provision, no end-customer PII, no production transcript pipeline.
Data boundaries for a sandbox pilot
What we need
- • REST Endpoint URL + auth for validation traffic
- • Synthetic prompts / scenarios you approve
- • Agent responses to those validation prompts
- • Verdict metadata (pass/fail, latency, region)
Outcome
Least privilege
Credential-based, customer-approved monitoring is the right model for enterprise trust.
What we do not need
- • Cognigy Insights exports or OData bulk feeds
- • Production conversation transcripts
- • End-customer PII
- • Access inside your VPC beyond the validation endpoint
Outcome
Your exports
Insights, OData, and transcripts remain Cognigy's surface. We do not ask for them to run user-side checks.
Credential-based, customer-approved monitoring is the right model for enterprise trust. We can share data-handling detail, retention, and audit evidence under NDA for security review.
User-side validation is two separate jobs
Reachability
Outcome
Can we talk to it?
Residential observers take the inbound path customers take — past CDN, WAF, and bot walls that treat datacenter synthetics differently. Monitoring asks: is it healthy right now? Reliability asks: does it keep working over time? “Up” means reachable from home networks, not from AWS.
Outcome verification
Outcome
Did it do the right thing?
Reachable and wrong is still broken. Scenario — did it finish the job? Compositional — do the pieces hold together? Safety — must-not-say, policy, attack probes. Stability — same ask, same story? Scores: Consistency and Drift. Not prose matching. Optional sample review is corroboration only.
Where we fit
Beside Cognigy Insights
Cognigy sees inside the platform; we see what the channel actually delivered to a user.
Cognigy sees what happens inside the platform and what you export to your stack. AgentStatus answers a different question: what did the real channel actually deliver from a residential observer vantage (inbound reachability past CDN/WAF — not answer localization by probe IP)?
Insights aggregates conversations; we independently run seven validation groups from outside your stack.
Insights gives you conversation truth. We verify reachability, answer correctness, multi-turn task completion, consistency under rephrasing, and robustness under stress, with evidence procurement can audit.
We execute from eight hundred residential nodes across thirty countries, not a single cloud region.
Issues that only reproduce from specific locations or ISPs surface here first, common in Cognigy's enterprise and telco customer base.
We only connect with credentials and scenarios your team approves.
No automatic discovery of Cognigy customers. Sandbox REST Endpoint, agreed synthetic scenarios, customer-approved credentials, aligned with procurement and data governance.
The split
How the work divides
Your platform
- • Build, deploy, analyze, optimize
- • Cognigy Insights dashboards
- • OData API event exports
- • Transcripts and session-level data
- • Live Agent escalation and Copilot
Outcome
System of record
Dashboards, exports, lifecycle tools, and orchestration remain yours. We do not replace that surface.
AgentStatus
- • We check whether the agent is reachable from real home networks right now.
- • We track whether it keeps working, stays fast, and holds its pass rate over time.
- • We grade whether answers are actually good, not just HTTP 200.
- • We run goal-driven conversations and verify whether the user got what they came for.
- • We test whether replies stay consistent when wording, stakes, or follow-ups change.
- • We stress tools, streams, safety rules, and adversarial inputs before customers do.
- • We alert your team when something breaks, with evidence attached.
Outcome
User-side layer
Reachability (Monitoring / Reliability) from residential networks, then outcome verification once reached — Scenario, Compositional, Safety, Stability; Consistency and Drift scores. Not prose matching.
Proof of scale
Auditable scale metrics
On the order of 20 million validation runs across the network. On the order of 8,000 agent records in our system, configurations we track, including evaluation and pipeline agents, not "8,000 paying customers."
Stricter production-only definitions available under NDA.
What we are not claiming
Independent evidence layer
We are not a replacement for Cognigy Insights, OData exports, or the Live Agent transcript layer. We help teams correlate user-side validation outcomes with inside-out conversation truth when both matter to the buyer.
Suggested next steps
Pilot checklist
Start with a two-week sandbox pilot using synthetic scenarios only.
Sandbox REST Endpoint (URL, userId, sessionId, Bearer token), agreed synthetic scenarios, no production traffic, no end-customer data. Written report: what we validated, what passed, what drifted.
Walk through security and data governance before anything touches production.
How AgentStatus connects under enterprise privacy requirements: validation-traffic boundaries, least privilege, retention, audit evidence. We expect this before any production-adjacent work.
Decide whether internal QA, joint customer proof, or both is the right first use case.
Cognigy-internal QA, a joint customer who wants third-party evidence alongside Insights, or both.
Cognigy helps enterprises build and operate serious AI agents at scale.
AgentStatus helps them prove continuously those agents behave the way policy and customers require, globally, with evidence that holds up under scrutiny.
Contact·dulra@carmel.so·roman@carmel.so
Metrics use explicit definitions: validation runs are scheduled executions; agent records are database rows, not revenue customers. Cognigy product references reflect public documentation as of this note.