Thoughts

Your Alert Webhooks Are Failing Silently (And You Have No Idea)

Picture the worst version of an outage:

Your AI agent goes down at 2:14 AM. Your monitoring catches it instantly. The alert fires. Your incident channel is silent. PagerDuty stays quiet. No one wakes up. By the time engineering arrives at 9 AM, customers have been impacted for nearly seven hours.

Root cause: the webhook delivering the alert returned a 502 from a misconfigured reverse proxy. The alert was created. It was just never delivered.

This isn't a hypothetical. It's the most under-monitored failure mode in modern observability stacks: the alert pipeline itself.

You instrument your agent. You instrument your model providers. You instrument your dependencies. You almost never instrument the path that tells you any of those things broke.


The Quiet Assumption Underneath Every Alert

Every alerting system in production assumes the same chain works:

event then rule then alert then channel adapter then delivery then human acknowledgement

Most teams instrument the first three steps and the last one. The middle two, channel adapter and delivery, are treated as plumbing. They're not. They're the part most likely to silently fail.

Slack is down. Discord rate-limits your bot. PagerDuty's Events API returns a 502 for 11 minutes. Your customer's webhook URL was rotated last week and you haven't redeployed. Microsoft Teams' incoming webhook connector was deprecated and you didn't notice. Each of these is a real outage we've seen, and in every case, the alert fired, the alert was created, and the alert was not delivered.

A monitoring system that can't prove its own alerts arrived isn't a monitoring system. It's a logging system with optimism.


Failure Mode 1: The Silent 4xx

What happens: Your webhook URL returns a 400, 401, 403, or 404. The alert dispatcher logs the failure and moves on. Nothing surfaces in the channel because the destination rejected the payload before delivery.

Why it happens:

  • The destination URL was rotated and your alert config is stale
  • The signing secret changed and the receiver now rejects unsigned payloads
  • The receiving service deprecated a payload format
  • Bot tokens expired (Slack, Discord, Teams all do this differently)

Real example: A team's Slack incoming webhook URL was rotated by an admin during quarterly key rotation. Their alert system kept POSTing to the old URL. Slack returned 404 invalid_token. The dispatcher logged it. No one read the dispatcher logs. For 23 days, every status-change alert silently 404'd. The team only discovered it during an actual production incident, when no one in the channel reacted because no one received it.

How to prevent it:

  • Treat 4xx and 5xx on alert delivery as a first-class incident, not a log line.
  • Surface delivery health on the same dashboard as the agent itself.
  • Track per-channel last_successful_delivery_at and alert when it stales.

The minimum bar: every delivery attempt should produce a row you can query.

attempt | channel_type | http_status | ok | error | sent_at

If you can't SELECT count(*) FROM alert_deliveries WHERE ok = false AND created_at > now() - interval '1 hour', you can't see your own alert failures.


Failure Mode 2: The Retry That Never Fires

What happens: Your webhook returns 503 because the receiver is briefly under load. Your dispatcher gives up on the first attempt. Three minutes later the receiver is healthy. The alert is gone.

Why it happens:

  • Naive try/except around httpx.post() with no retry logic
  • Retries exist but treat 5xx as fatal
  • Retries exist but back off too aggressively (e.g. 5-minute initial delay)
  • No distinction between retryable (5xx, network errors) and non-retryable (4xx)

Real example: A team's PagerDuty integration POSTed to events.pagerduty.com. During a 90-second AWS Direct Connect blip, three alerts hit network errors. The dispatcher logged "delivery failed" and the alerts disappeared. PagerDuty was healthy 60 seconds later. The team only learned about the missed alerts when reconciling Slack threads against PagerDuty incidents the next week.

How to prevent it:

  • Bounded retries with exponential or staged backoff.
  • Distinguish retryable from non-retryable status codes.
  • Persist retry attempts so you can prove what happened.

A reasonable default looks like this:

_MAX_RETRY_ATTEMPTS = 3
_RETRY_DELAYS = [2, 5]  # seconds between attempt 1 to 2 and 2 to 3

# Retry only on network-level errors or 5xx
is_retryable = result.http_status is None or (result.http_status >= 500)

Three attempts, staged delays, retryable-only, and a row in a deliveries table per attempt. That's the floor. Anything less is hope.


Failure Mode 3: The Signature Mismatch

What happens: Your customer's webhook receiver verifies an HMAC signature on every POST. Your dispatcher rotates the signing secret on one side without coordinating the other. Every alert now arrives but is rejected for a bad signature. You see 200s in your logs (the receiver swallows and returns 200 to avoid retries) but nothing makes it past the verifier.

Why it happens:

  • Signing secret rotation isn't atomic across systems
  • Receivers swallow signature failures to avoid amplification attacks
  • Senders don't include a key_id header to support overlapping secrets
  • Rotation happens during off-hours with no smoke test

Real example: A B2B AI platform offered alert webhooks to enterprise customers. They rotated the platform-wide signing secret as part of a security audit. Customers' verifiers, using the previous secret, silently dropped every alert for nine days. The platform team only learned during a customer escalation: "Why didn't you alert us about the regression?" They had. The customer's verifier had thrown every alert away.

How to prevent it:

  • Include a key/secret identifier in the signed header (e.g. X-Signature: sha256=...; key_id=v3).
  • Run a synthetic alert post-rotation to validate end-to-end signature acceptance.
  • Track receiver-side rejection separately from sender-side delivery.

The hardest part of HMAC isn't the algorithm. It's the rotation choreography.


Failure Mode 4: The Channel Drift

What happens: A channel that worked perfectly last quarter no longer accepts the payload format you send. Microsoft Teams deprecated MessageCard connectors. Discord changed embed validation. Slack tightened block-kit rules. Your dispatcher hasn't been updated. Alerts stop arriving on one channel while every other channel works.

Why it happens:

  • Vendor channel APIs evolve faster than your dispatcher
  • Deprecation notices arrive in admin email, not engineering email
  • Schema validation tightens in minor releases (Slack does this routinely)
  • Adaptive Card format changes break Teams while keeping the URL alive

Real example: A team's Teams alerts stopped working in Q1 2026 when Microsoft began phasing out the legacy MessageCard format. The connector still accepted POSTs. It returned 200. It just stopped rendering. The alerts were technically delivered. They just weren't visible. Engineering spent two weeks chasing "phantom alerts" before someone opened Teams on a fresh device.

How to prevent it:

  • Validation each alert channel periodically with a synthetic test alert that includes a known marker.
  • Have a human ack the synthetic from each channel monthly.
  • Watch deprecation calendars for every alert vendor in your stack.

A delivered-but-invisible alert is the worst kind of failure. The only defense is human-in-the-loop verification.


Failure Mode 5: The Multi-Channel Illusion

What happens: You configure alerts to fan out across email, Slack, PagerDuty, and a customer webhook. Three of four work. The dashboard says "alert sent." Coverage feels redundant. It isn't, because the one channel the on-call engineer relies on tonight is the broken one.

Why it happens:

  • "Alert sent" gets aggregated as a single boolean across multiple channels
  • Per-channel delivery state isn't surfaced
  • Failures get logged but not graded by criticality
  • Redundancy creates false confidence rather than actual coverage

Real example: A startup configured PagerDuty + Slack + email for every alert. PagerDuty was their primary on-call surface. PagerDuty's routing_key was misconfigured after a re-org. For three months, every alert went to Slack and email, both of which were noisy and ignored after-hours. PagerDuty silently rejected every payload. They missed two production incidents before catching it.

How to prevent it:

  • Track delivery success per channel, not as a single aggregate.
  • Tier channels by criticality. The primary on-call channel must succeed; secondary channels are nice-to-have.
  • Page when the primary channel hasn't successfully delivered in N minutes, even if no alert was due.

Redundancy without measurement is just wishful thinking with extra steps.


Failure Mode 6: The Acknowledgement Gap

What happens: Your webhook delivered. The receiver accepted. The payload rendered. No human acknowledged it. The alert achieves all of its technical objectives and zero of its business objectives.

Why it happens:

  • Channel volume too high; alerts get lost in scroll
  • Severity not visually distinguishable (everything is red)
  • After-hours delivery to a daytime channel
  • Alert arrives during a different incident's war room

Real example: A team's monitoring delivered an alert about model drift at 4:17 AM. It landed in #alerts. By morning, six other unrelated alerts had pushed it off-screen. The drift continued for two more days because no one scrolled back. The system did everything right. The humans never engaged.

How to prevent it:

  • Require explicit ack on critical-severity alerts. Repeat until acknowledged.
  • Tier severity visually, critical alerts should look different from informational ones.
  • Route by time-of-day (PagerDuty during off-hours, Slack during business hours).
  • Track time-to-ack alongside time-to-deliver. Both matter.

What "Real" Alert Pipeline Monitoring Looks Like

Five questions a healthy alert system answers continuously:

  1. Did the alert get created? (Detection layer is working.)
  2. Did the dispatcher attempt every configured channel? (No channel was silently skipped.)
  3. Did each channel return success? (Per-channel HTTP status, persisted.)
  4. Did the receiver actually render the alert? (Synthetic alerts with markers.)
  5. Did a human acknowledge it within the SLA? (Ack-to-fire latency tracked.)

The minimum data model to answer those:

alert_deliveries (
  alert_id, channel_type, attempt, http_status, ok,
  provider_id, error_message, sent_at, created_at
)

If your alerting infrastructure can't tell you which alert went to which channel, succeeded or failed, on which attempt, with what response code, you don't have an alerting system. You have a print() statement with delusions of grandeur.


Quick Wins

This week

  1. Add a deliveries table. Persist per-channel attempt + status + error for every alert. Even SQLite is fine. The point is you can query it.
  2. Implement bounded retries. Three attempts, staged backoff, retryable-only (5xx + network), one row per attempt.
  3. Surface "last successful delivery" per channel. A channel that hasn't delivered in 24 hours when you fire alerts hourly is not "quiet", it's broken.

This month

  1. Run a weekly synthetic alert through every channel. Have a human ack each one. Track ack rate per channel.
  2. HMAC-sign customer webhooks. Include a key_id so rotation doesn't break verifiers.
  3. Tier severity then routing. Critical alerts page; informational alerts post; never both equally.

Monitoring you can't prove arrived isn't monitoring.

The alert was real. The detection worked. The dispatcher did its job. The human never knew. The fix isn't more channels — it's measuring the channels you already have.