AgentStatus × Cognigy partner brief
Outside-in validation for Cognigy AI agents.
Seven checks from real home networks, status through alerts, alongside Cognigy Insights, OData, and Live Agent. We complement inside-out visibility; we do not replace it. No instrumentation on your stack. Synthetic validations only, no production transcripts.
What we understand about Cognigy
Inside-out visibility
Cognigy.AI spans build, deploy, analyze, and optimize across voice, chat, and messaging. Cognigy Insights delivers conversation analytics and NLU performance. OData API feeds BI teams raw conversation data. Live Agent handles escalation; Agent Copilot supports agents in real time.
For outside-in validation, Cognigy's REST Endpoint is the natural surface: POST with userId, sessionId, and Bearer token, the same path a channel integration would use.
What we validate
Uptime is table stakes. We watch the rest too.
Reliability, consistency, and robustness. From home networks, on a schedule. Plain verdicts you can explain to your boss.
Reliability
Does it keep working, stay fast, and behave the same over time?
Can people reach it?
Not just 200 OK from your office. We hit it from home networks where your users actually are.
time to first byte
Fast enough to feel alive
Slow first replies feel broken even when the answer eventually shows up. We watch time-to-first-byte, not just total wait.
From home networks
Same agent, different country, different result. Geo blocks and CDN quirks show up here first.
Keeps working, run after run
One green check means nothing. We track whether it stays healthy across dozens of scheduled runs.
week 6 · -9pts vs baseline
When behavior starts to drift
Slow leaks matter. Pass rates, latency, and answer patterns shifting week over week get flagged before users notice.
Answer quality
Not just reachable. Actually helpful, shaped right, and checked more than once.
Are the answers good?
We read replies like a human would: pass, degraded, or inconclusive, with a short explanation you can act on.
Two reviewers had to agree
When there is no single right string, two independent reviewers weigh in. Disagreement means we say inconclusive, not a false alarm.
Your example questions
Bring your own scenarios: refunds, escalations, edge cases you already worry about.
Questions for what it actually does
We watch how the agent behaves in the wild and keep testing with fresh example questions that match its real job.
Exact shape, every time
JSON APIs and structured replies should look like you promised. We check the shape, not just the vibe.
Extra eyes when it looks fishy
Automated checks miss nuance. Flag samples for a human look, then turn repeat mistakes into a rule.
Conversations
Does it finish the job? Real goals, pursued over several messages, judged on the outcome.
A user with a real goal
We give a simulated user something real to want, like a refund, a booking, or a fix, and let them pursue it over several messages, clarifying and pushing back like a real person.
Did they get what they came for?
When the conversation ends, an independent reviewer reads the whole transcript and makes one call: goal achieved or not.
We don't take its word for it
If the agent claims success, we double-check the claim against outside evidence where it exists, such as a cited page, a structured field, or a second reviewer, before counting it.
Consistency
Same situation, same story. No flip-flopping when wording, mood, or follow-ups change.
Same question, different words
Rephrased prompts should not flip the answer. We flag replies that swing for no good reason.
Calm ask vs panicked ask
A billing FAQ and a locked-out account are not the same urgency. Good agents step up when stakes rise.
Follow-ups still line up
Ask A, then B, then C. Later answers should not contradict what the agent said two turns ago.
Robustness
Tools, streams, rules, and people trying to break it.
Do tools get called?
Send questions that should trigger an action. Catch broken integrations before customers hit them.
MCP servers
Discover tools and resources, run calls, and check the outputs for agents wired through MCP.
4/4 chunks · closed cleanly
Streaming that finishes
Streaming endpoints can hang, stall, or never complete. We catch that, not just the final text blob.
Your safety rules
Tell us what the agent must never do, and what it should still do. We check both sides.
When someone tries to trick it
Pushy, weird, or adversarial prompts. Does the agent stay on policy or fold?
When it breaks, you know
Alerts, deploy gates, and one place to see it all.
Alerts when it breaks
Slack, webhooks, PagerDuty, with enough context to fix it, not just a red dot.
agent failed outside-in check
Block bad deploys before ship
Wire us into CI so a broken agent does not merge just because the unit tests passed.
One dashboard
Every check rolls up into a simple grid: what is OK, what is still collecting, and what needs attention.
Architecture
How validation runs
Fabric nodes execute transport validations: reachability, node-side gold checks, streaming latency, and inbound MCP/tools discovery conformance against your REST endpoint (reachability of the MCP surface — not a claim that residential probes change agent tool egress). Raw results return to the AgentStatus backend, which schedules cycles, runs portfolio validations and adaptive scenarios, and applies the instrument layer on top of node evidence.
That instrument layer is not a single semantic pass. It includes format contracts, dual-review semantic scoring, metamorphic and task checks, outcome verification where configured, and statistical rollups: pass^k stability, drift baselines, alerts, and explain traces in the dashboard. Your Cognigy platform (Insights, OData, transcripts) stays inside your boundary. We do not pull from it.
What runs on nodes vs. the backend
Nodes execute transport. The backend runs instruments and rollups. Fabric nodes sit on residential networks in 70 countries and hit your REST Endpoint the way a real channel would. The backend receives those raw results and runs the multi-instrument scoring layer plus portfolio statistics. Semantic review is one instrument among several, not the whole system.
Results are verdict metadata, latency, region, and evidence snippets, not production conversation exports.
Data privacy
Data boundaries
Most teams ask whether they must pass customer conversation data to AgentStatus. You do not. Pilot default is Cognigy sandbox only: synthetic prompts you define, validation credentials you provision, no end-customer PII, no production transcript pipeline.
Data boundaries for a sandbox pilot
What we need
- • REST Endpoint URL + auth for validation traffic
- • Synthetic prompts / scenarios you approve
- • Agent responses to those validation prompts
- • Verdict metadata (pass/fail, latency, region)
Outcome
Least privilege
Credential-based, customer-approved monitoring is the right model for enterprise trust.
What we do not need
- • Cognigy Insights exports or OData bulk feeds
- • Production conversation transcripts
- • End-customer PII
- • Access inside your VPC beyond the validation endpoint
Outcome
Your exports
Insights, OData, and transcripts remain Cognigy's surface. We do not ask for them to run outside-in checks.
Credential-based, customer-approved monitoring is the right model for enterprise trust. We can share data-handling detail, retention, and audit evidence under NDA for security review.
Outside-in validation is two separate jobs
Reachability
Outcome
Residential path
Your monitor hits the VIP lane. Users hit the WAF. Datacenter checks get blocked, throttled, or allowlisted. Residential observers take the inbound path customers take — so “up” means reachable from home networks, not from AWS.
Reliability
Outcome
Answer quality
Reachable and self-contradicting is still broken. Rephrase flips, drift, and policy breaks need no ground truth. Gold and dual judges cover the rest when truth exists. Uptime grades none of that.
Where we fit
Beside Cognigy Insights
Cognigy sees inside the platform; we see what the channel actually delivered to a user.
Cognigy sees what happens inside the platform and what you export to your stack. AgentStatus answers a different question: what did the real channel actually deliver from a residential observer vantage (inbound reachability past CDN/WAF — not answer localization by probe IP)?
Insights aggregates conversations; we independently run seven validation groups from outside your stack.
Insights gives you conversation truth. We verify reachability, answer correctness, multi-turn task completion, consistency under rephrasing, and robustness under stress, with evidence procurement can audit.
We execute from eight hundred residential nodes across thirty countries, not a single cloud region.
Issues that only reproduce from specific locations or ISPs surface here first, common in Cognigy's enterprise and telco customer base.
We only connect with credentials and scenarios your team approves.
No automatic discovery of Cognigy customers. Sandbox REST Endpoint, agreed synthetic scenarios, customer-approved credentials, aligned with procurement and data governance.
The split
How the work divides
How the work divides
Their platform
- • Build, deploy, analyze, optimize
- • Cognigy Insights dashboards
- • OData API event exports
- • Transcripts and session-level data
- • Live Agent escalation and Copilot
Outcome
System of record
Dashboards, exports, lifecycle tools, and orchestration remain theirs. We do not replace that surface.
AgentStatus
- • We check whether the agent is reachable from real home networks right now.
- • We track whether it keeps working, stays fast, and holds its pass rate over time.
- • We grade whether answers are actually good, not just HTTP 200.
- • We run goal-driven conversations and verify whether the user got what they came for.
- • We test whether replies stay consistent when wording, stakes, or follow-ups change.
- • We stress tools, streams, safety rules, and adversarial inputs before customers do.
- • We alert your team when something breaks, with evidence attached.
Outcome
Outside-in layer
Residential inbound path past CDN/WAF, then gold, consistency, and scoped judges once the agent is reachable.
Proof of scale
Auditable scale metrics
On the order of 20 million validation runs across the network. On the order of 8,000 agent records in our system, configurations we track, including evaluation and pipeline agents, not "8,000 paying customers."
Stricter production-only definitions available under NDA.
What we are not claiming
Independent evidence layer
We are not a replacement for Cognigy Insights, OData exports, or the Live Agent transcript layer. We help teams correlate outside-in validation outcomes with inside-out conversation truth when both matter to the buyer.
Suggested next steps
Pilot checklist
Start with a two-week sandbox pilot using synthetic scenarios only.
Sandbox REST Endpoint (URL, userId, sessionId, Bearer token), agreed synthetic scenarios, no production traffic, no end-customer data. Written report: what we validated, what passed, what drifted.
Walk through security and data governance before anything touches production.
How AgentStatus connects under enterprise privacy requirements: validation-traffic boundaries, least privilege, retention, audit evidence. We expect this before any production-adjacent work.
Decide whether internal QA, joint customer proof, or both is the right first use case.
Cognigy-internal QA, a joint customer who wants third-party evidence alongside Insights, or both.
Cognigy helps enterprises build and operate serious AI agents at scale.
AgentStatus helps them prove continuously those agents behave the way policy and customers require, globally, with evidence that holds up under scrutiny.
Contact·dulra@carmel.so·roman@carmel.so
Metrics use explicit definitions: validation runs are scheduled executions; agent records are database rows, not revenue customers. Cognigy product references reflect public documentation as of this note.