Thoughts

The Failure Mode Nobody Watches: Agent Loops That Never Reach HTTP

Every dashboard you have was designed for a world where failures look like 5xx responses.

That world doesn't include AI agents.

Modern agents fail in places where no HTTP code is ever emitted. The model provider hangs mid-stream. A tool call enters a retry loop nobody bounded. A background worker picks up a job and dies between heartbeats. An async planner gets stuck on a task it can't decompose. The agent isn't returning errors, it isn't returning at all. And because nothing was ever sent to your edge, your edge has nothing to report.

This is the most expensive failure mode in production agents and the one with the least monitoring coverage. Because the failure happens inside the agent loop, traditional observability, built for request/response systems with clean status codes, sees nothing.

Let's go through what actually breaks, why your stack misses it, and what monitoring it correctly looks like.


Why Agents Break Outside HTTP

Traditional web services are stateless. A request comes in, work happens synchronously, a response goes out. Every interaction emits a status code. Failures are observable by definition.

Agents are not that.

A modern production agent typically runs:

  • A request handler that enqueues a job rather than executes it
  • A worker that picks up the job and plans a response (multiple LLM calls)
  • Tool invocations that may call other agents, MCP servers, or long-running APIs
  • A streaming response back to the user via SSE or WebSocket
  • Background reflection, memory writes, and follow-up workers triggered post-response

A failure can happen at any layer, and most layers don't speak HTTP. Some don't even have a request/response shape. They have steps that are supposed to complete and sometimes don't.

When something in this graph breaks, the user sees one of three things:

  1. Nothing. The connection sits open, gets killed by an upstream proxy minutes later, and a generic timeout page eventually loads.
  2. A truncated stream. The first 40 tokens arrive. The rest don't. The user assumes the response was complete.
  3. A confident lie. The agent replies with whatever it had before the loop went sideways. The result reads correct and isn't.

None of these produce a 5xx. None trigger your error-rate alert. All three were customer-impacting and invisible.


Failure Mode 1: The Stuck Worker

What happens: A worker process pulls a job, marks it executing, then dies, OOM kill, segfault, network partition, container restart, or just a thread blocked on a syscall that never returns. The job sits in executing state forever. The orchestrator assumes work is in flight. The user waits.

Why it happens:

  • Kubernetes pod evicted mid-execution
  • Worker holds a connection to a model provider that hangs at the TLS layer
  • Memory pressure crashes the process before it can update job state
  • Long-running tool call exceeds the worker's wall-clock and gets SIGKILL'd

Real example: A team's agent worker pulled a job, called a third-party search API that hung at the TLS handshake, and was evicted by the orchestrator after 10 minutes. The job stayed in executing state. The orchestrator never finalized the parent run. The user's request hung until the browser timed out at 5 minutes. From the worker's logs: nothing. From the API's logs: nothing. From the user's perspective: a blank page after a long wait.

How to prevent it:

  • Implement a stuck-job watchdog at the orchestrator level.
  • Define a STUCK_JOB_THRESHOLD (e.g. 5 minutes) past which a queued or executing job is presumed dead.
  • Mark stuck jobs failed so the parent run can finalize on whatever did come back.
  • Make the watchdog write race-safe: only update if the job is still queued or executing. A real worker write must always win.

A reasonable watchdog query looks like this:

UPDATE jobs
SET status        = 'failed',
    error_message = COALESCE(error_message, 'watchdog_timeout'),
    completed_at  = NOW()
WHERE id = $1
  AND status IN ('queued', 'executing');

The AND status IN (...) clause is the part that matters. It means a real worker that finished half a second after the watchdog ran still wins. Without it, you're racing yourself.


Failure Mode 2: The Provider Stream That Never Ends

What happens: The model provider opens an SSE stream, sends a few tokens, and then sits there. No close. No error. No final chunk. Your agent reads forever, blocked on next(stream). The user watches a cursor blink.

Why it happens:

  • Provider's edge proxy stalls on backpressure
  • Connection migrates between provider POPs mid-stream
  • A token-counting bug on the provider's side stops emitting but doesn't close
  • Network path develops asymmetric loss; ACKs flow, data doesn't

Real example: A team using a major LLM provider had streams hang indefinitely about 0.4% of the time during peak hours. Their agent worker had no read timeout on the SSE iterator. Workers accumulated. Pods OOM'd. Cascading failures took down 30% of capacity for nine minutes. The fix was a four-line change: read_timeout = 30s on the streaming client.

How to prevent it:

  • Always set explicit read timeouts on streaming model calls. Never trust the SDK default.
  • Track time-to-first-token (TTFT) and time-between-tokens (TBT) as separate metrics.
  • Alert when TBT exceeds threshold for N seconds even if the connection is technically open.

Streaming is a special case. A connection that's "alive" but not delivering bytes is the most insidious form of broken. The only defense is treating time-since-last-byte as a vital sign, not just connection state.


Failure Mode 3: The Recursive Planner

What happens: An agent's planner decides the task requires another planner step, which decides it requires another, which decides it requires another. The agent never returns because it never finishes thinking.

Why it happens:

  • No max-depth bound on planning recursion
  • Reflection loops don't have a "give up" condition
  • Tool call returns ambiguous output that triggers re-planning
  • Multi-agent handoffs form an unintended cycle

Real example: A research agent designed to "decompose tasks until each step is atomic" got handed an under-specified question. It decomposed it into 14 sub-tasks. Each sub-task got decomposed into 14 more. By the third level the agent was running 2,744 LLM calls and generating tokens at $40/minute. The user gave up after 90 seconds. The agent kept running for 31 minutes before a daily cost cap finally killed it.

How to prevent it:

  • Hard cap planning depth at the agent code level. Never trust the model to self-bound.
  • Hard cap total token spend per request. Halt and return a partial result.
  • Hard cap wall-clock per request, per step, and per outer loop.
  • Treat "agent didn't finish in N seconds" as a customer-impacting verdict.

The default for any agent loop should be: fail loudly when bounded. The worst outcome is silent runaway, the second-worst is a successful response that arrived too late to matter.


Failure Mode 4: The Background Job That Wasn't

What happens: Your agent posts a successful response, then enqueues a background task to write to memory, log feedback, or trigger a follow-up. The background task fails. The user response was correct. The next interaction has missing context. Quality silently degrades over sessions.

Why it happens:

  • Background job queue is best-effort, not durable
  • Worker that consumes background tasks crashed yesterday and hasn't been restarted
  • Background task itself errors on a corner case nobody tests
  • Job exists but the worker pool is starved by other priorities

Real example: A customer-support agent's "save conversation summary" background task started failing on a specific edge case (very long conversations with code blocks). 4% of conversations weren't being summarized. New conversations with the same customer no longer had context. Customers got frustrated repeating themselves. The agent looked broken to users, and was actually being broken by a background job nobody had instrumented.

How to prevent it:

  • Treat background tasks as first-class jobs with status, retries, and a watchdog.
  • Track per-job-type success rate and alert on drift.
  • Run synthetic conversations that depend on background-task side effects to detect missing follow-through.

The asymmetry of foreground vs. background reliability is where most "the agent feels worse over time" complaints originate. The foreground response works. The background pipeline that maintains state quietly broke.


Failure Mode 5: The Job Lifecycle You Don't Own

What happens: Your agent's loop depends on a worker pool you don't directly control, a serverless platform, a third-party agent runtime, a managed orchestrator. A job enters queued, gets accepted, and then disappears. Your audit trail ends at "we sent it."

Why it happens:

  • Provider's queue evicted under load
  • Cold start exceeded the queue's TTL
  • A multi-tenant noisy neighbor starved your scheduling slot
  • Provider scaled in mid-execution

Real example: A team using a serverless agent runtime was seeing about 2% of their requests "succeed", the runtime returned 202 Accepted, but never produce output. The runtime had a queue eviction policy that kicked in under load and never notified the caller. The team only found out when they cross-referenced their submitted-job log against their received-result log and noticed the gap.

How to prevent it:

  • Reconcile submitted jobs against completed jobs continuously.
  • Treat any job with no terminal state past your wall-clock SLA as failed.
  • Surface gap rate as a top-level reliability metric, separate from error rate.

If you only count failures the provider tells you about, you're underreporting. The most honest reliability number is 1 - (terminal_jobs / submitted_jobs).


Failure Mode 6: The Tool Loop

What happens: The agent calls tool_a. tool_a returns. The agent re-evaluates and decides to call tool_a again with slightly different inputs. Then again. Then again. The agent isn't broken, it's just convinced the next call is the one that will work. Twelve seconds in, the user gives up.

Why it happens:

  • No deduplication on identical tool calls
  • Tool returns {"result": null} instead of an explicit "no data" signal
  • The model can't disambiguate "no result" from "wrong query"
  • Reflection prompt encourages "try again with different inputs" without bound

Real example: A travel agent designed to "search until you find availability" called the flights API 18 times in a single user turn after the first call returned no results. Each call took 1.5s. The user waited 27 seconds for a response that simply said "I couldn't find any flights matching your criteria." The agent's loop was technically working. The user's experience was completely broken.

How to prevent it:

  • Hard cap tool calls per agent turn.
  • Dedupe identical tool calls within a turn, return the cached result.
  • Treat repeated identical tool calls as a behavioral anomaly worth alerting on.
  • Track tool calls per turn as a quality metric, not just a debugging detail.

What "Real" Agent-Loop Monitoring Looks Like

Six questions every healthy agent observability stack answers:

  1. Did the request enter the queue? (Submission tracked.)
  2. Did a worker pick it up within SLA? (Queue latency bounded.)
  3. Did execution complete in any state, success or failure? (Watchdog catches stuck.)
  4. Did the streaming response deliver bytes consistently? (TTFT + TBT tracked.)
  5. Did all background tasks triggered by the response succeed? (Per-job-type success rate.)
  6. Did the agent respect its loop bounds? (Depth, token spend, tool calls per turn.)

The minimum infrastructure to answer those:

  • A jobs table with explicit status (queued, executing, completed, failed) and timestamps.
  • A watchdog process that sweeps stuck jobs on an interval shorter than your SLA.
  • Per-step latency tracking (queue wait, execution time, time-to-first-token).
  • Reconciliation between submitted and terminated jobs.

Agent reliability isn't an HTTP status problem. It's a job-lifecycle problem. The teams that get this right treat agent runs the way payment systems treat transactions: every state transition logged, every terminal state reachable, every stuck record reconciled.


Quick Wins

Today

  1. Add an explicit job state machine. queued then executing then completed | failed. Persist transitions. Don't pass jobs around in memory hoping for the best.
  2. Set a read timeout on every streaming model call. If the SDK doesn't expose one, wrap it. The default is "wait forever" and that's not a default, that's a bug waiting.

This week

  1. Run a stuck-job watchdog. Sweep jobs older than your SLA. Mark them failed. Make the update race-safe (AND status IN ('queued','executing')). Run it every 30 seconds.
  2. Cap planning depth, tool calls per turn, and per-request wall clock. Pick numbers. Enforce them. A bounded failure beats an unbounded one every time.

This month

  1. Reconcile submitted vs. terminated. Daily report: how many jobs entered, how many reached a terminal state, what's the gap. If the gap isn't zero, you have invisible failures.
  2. Track TTFT and TBT separately. Stream health isn't a single number. Latency to first token is a UX metric. Time between tokens is a liveness metric. Different alerts, different thresholds.

Failures that never reach HTTP are still failures.

Agent loops have steps and lifecycles, not status codes. Reliable teams treat them like distributed systems: explicit state, durable transitions, watchdog reconciliation, bounded loops. The question is whether your monitoring acknowledges that yet.