AgentStatus
We tested every public MCP server we could find. Calling is solved. Delivery is not.
Read the reportPicture the worst version of an outage:
Your AI agent goes down at 2:14 AM. Your monitoring catches it instantly. The alert fires. Your incident channel is silent. PagerDuty stays quiet. No one wakes up. By the time engineering arrives at 9 AM, customers have been impacted for nearly seven hours.
Root cause: the webhook delivering the alert returned a 502 from a misconfigured reverse proxy. The alert was created. It was just never delivered.
This isn't a hypothetical. It's the most under-monitored failure mode in modern observability stacks: the alert pipeline itself.
You instrument your agent. You instrument your model providers. You instrument your dependencies. You almost never instrument the path that tells you any of those things broke.
Every alerting system in production assumes the same chain works:
event then rule then alert then channel adapter then delivery then human acknowledgementMost teams instrument the first three steps and the last one. The middle two, channel adapter and delivery, are treated as plumbing. They're not. They're the part most likely to silently fail.
Slack is down. Discord rate-limits your bot. PagerDuty's Events API returns a 502 for 11 minutes. Your customer's webhook URL was rotated last week and you haven't redeployed. Microsoft Teams' incoming webhook connector was deprecated and you didn't notice. Each of these is a real outage we've seen, and in every case, the alert fired, the alert was created, and the alert was not delivered.
A monitoring system that can't prove its own alerts arrived isn't a monitoring system. It's a logging system with optimism.
What happens: Your webhook URL returns a 400, 401, 403, or 404. The alert dispatcher logs the failure and moves on. Nothing surfaces in the channel because the destination rejected the payload before delivery.
Why it happens:
Real example: A team's Slack incoming webhook URL was rotated by an admin during quarterly key rotation. Their alert system kept POSTing to the old URL. Slack returned 404 invalid_token. The dispatcher logged it. No one read the dispatcher logs. For 23 days, every status-change alert silently 404'd. The team only discovered it during an actual production incident, when no one in the channel reacted because no one received it.
How to prevent it:
last_successful_delivery_at and alert when it stales.The minimum bar: every delivery attempt should produce a row you can query.
attempt | channel_type | http_status | ok | error | sent_atIf you can't SELECT count(*) FROM alert_deliveries WHERE ok = false AND created_at > now() - interval '1 hour', you can't see your own alert failures.
What happens: Your webhook returns 503 because the receiver is briefly under load. Your dispatcher gives up on the first attempt. Three minutes later the receiver is healthy. The alert is gone.
Why it happens:
try/except around httpx.post() with no retry logicReal example: A team's PagerDuty integration POSTed to events.pagerduty.com. During a 90-second AWS Direct Connect blip, three alerts hit network errors. The dispatcher logged "delivery failed" and the alerts disappeared. PagerDuty was healthy 60 seconds later. The team only learned about the missed alerts when reconciling Slack threads against PagerDuty incidents the next week.
How to prevent it:
A reasonable default looks like this:
_MAX_RETRY_ATTEMPTS = 3
_RETRY_DELAYS = [2, 5] # seconds between attempt 1 to 2 and 2 to 3
# Retry only on network-level errors or 5xx
is_retryable = result.http_status is None or (result.http_status >= 500)Three attempts, staged delays, retryable-only, and a row in a deliveries table per attempt. That's the floor. Anything less is hope.
What happens: Your customer's webhook receiver verifies an HMAC signature on every POST. Your dispatcher rotates the signing secret on one side without coordinating the other. Every alert now arrives but is rejected for a bad signature. You see 200s in your logs (the receiver swallows and returns 200 to avoid retries) but nothing makes it past the verifier.
Why it happens:
key_id header to support overlapping secretsReal example: A B2B AI platform offered alert webhooks to enterprise customers. They rotated the platform-wide signing secret as part of a security audit. Customers' verifiers, using the previous secret, silently dropped every alert for nine days. The platform team only learned during a customer escalation: "Why didn't you alert us about the regression?" They had. The customer's verifier had thrown every alert away.
How to prevent it:
X-Signature: sha256=...; key_id=v3).The hardest part of HMAC isn't the algorithm. It's the rotation choreography.
What happens: A channel that worked perfectly last quarter no longer accepts the payload format you send. Microsoft Teams deprecated MessageCard connectors. Discord changed embed validation. Slack tightened block-kit rules. Your dispatcher hasn't been updated. Alerts stop arriving on one channel while every other channel works.
Why it happens:
Real example: A team's Teams alerts stopped working in Q1 2026 when Microsoft began phasing out the legacy MessageCard format. The connector still accepted POSTs. It returned 200. It just stopped rendering. The alerts were technically delivered. They just weren't visible. Engineering spent two weeks chasing "phantom alerts" before someone opened Teams on a fresh device.
How to prevent it:
A delivered-but-invisible alert is the worst kind of failure. The only defense is human-in-the-loop verification.
What happens: You configure alerts to fan out across email, Slack, PagerDuty, and a customer webhook. Three of four work. The dashboard says "alert sent." Coverage feels redundant. It isn't, because the one channel the on-call engineer relies on tonight is the broken one.
Why it happens:
Real example: A startup configured PagerDuty + Slack + email for every alert. PagerDuty was their primary on-call surface. PagerDuty's routing_key was misconfigured after a re-org. For three months, every alert went to Slack and email, both of which were noisy and ignored after-hours. PagerDuty silently rejected every payload. They missed two production incidents before catching it.
How to prevent it:
Redundancy without measurement is just wishful thinking with extra steps.
What happens: Your webhook delivered. The receiver accepted. The payload rendered. No human acknowledged it. The alert achieves all of its technical objectives and zero of its business objectives.
Why it happens:
Real example: A team's monitoring delivered an alert about model drift at 4:17 AM. It landed in #alerts. By morning, six other unrelated alerts had pushed it off-screen. The drift continued for two more days because no one scrolled back. The system did everything right. The humans never engaged.
How to prevent it:
Five questions a healthy alert system answers continuously:
The minimum data model to answer those:
alert_deliveries (
alert_id, channel_type, attempt, http_status, ok,
provider_id, error_message, sent_at, created_at
)If your alerting infrastructure can't tell you which alert went to which channel, succeeded or failed, on which attempt, with what response code, you don't have an alerting system. You have a print() statement with delusions of grandeur.
key_id so rotation doesn't break verifiers.Monitoring you can't prove arrived isn't monitoring.
The alert was real. The detection worked. The dispatcher did its job. The human never knew. The fix isn't more channels — it's measuring the channels you already have.