AgentStatus
We tested every public MCP server we could find. Calling is solved. Delivery is not.
Read the reportEvery dashboard you have was designed for a world where failures look like 5xx responses.
That world doesn't include AI agents.
Modern agents fail in places where no HTTP code is ever emitted. The model provider hangs mid-stream. A tool call enters a retry loop nobody bounded. A background worker picks up a job and dies between heartbeats. An async planner gets stuck on a task it can't decompose. The agent isn't returning errors, it isn't returning at all. And because nothing was ever sent to your edge, your edge has nothing to report.
This is the most expensive failure mode in production agents and the one with the least monitoring coverage. Because the failure happens inside the agent loop, traditional observability, built for request/response systems with clean status codes, sees nothing.
Let's go through what actually breaks, why your stack misses it, and what monitoring it correctly looks like.
Traditional web services are stateless. A request comes in, work happens synchronously, a response goes out. Every interaction emits a status code. Failures are observable by definition.
Agents are not that.
A modern production agent typically runs:
A failure can happen at any layer, and most layers don't speak HTTP. Some don't even have a request/response shape. They have steps that are supposed to complete and sometimes don't.
When something in this graph breaks, the user sees one of three things:
None of these produce a 5xx. None trigger your error-rate alert. All three were customer-impacting and invisible.
What happens: A worker process pulls a job, marks it executing, then dies, OOM kill, segfault, network partition, container restart, or just a thread blocked on a syscall that never returns. The job sits in executing state forever. The orchestrator assumes work is in flight. The user waits.
Why it happens:
Real example: A team's agent worker pulled a job, called a third-party search API that hung at the TLS handshake, and was evicted by the orchestrator after 10 minutes. The job stayed in executing state. The orchestrator never finalized the parent run. The user's request hung until the browser timed out at 5 minutes. From the worker's logs: nothing. From the API's logs: nothing. From the user's perspective: a blank page after a long wait.
How to prevent it:
STUCK_JOB_THRESHOLD (e.g. 5 minutes) past which a queued or executing job is presumed dead.failed so the parent run can finalize on whatever did come back.queued or executing. A real worker write must always win.A reasonable watchdog query looks like this:
UPDATE jobs
SET status = 'failed',
error_message = COALESCE(error_message, 'watchdog_timeout'),
completed_at = NOW()
WHERE id = $1
AND status IN ('queued', 'executing');The AND status IN (...) clause is the part that matters. It means a real worker that finished half a second after the watchdog ran still wins. Without it, you're racing yourself.
What happens: The model provider opens an SSE stream, sends a few tokens, and then sits there. No close. No error. No final chunk. Your agent reads forever, blocked on next(stream). The user watches a cursor blink.
Why it happens:
Real example: A team using a major LLM provider had streams hang indefinitely about 0.4% of the time during peak hours. Their agent worker had no read timeout on the SSE iterator. Workers accumulated. Pods OOM'd. Cascading failures took down 30% of capacity for nine minutes. The fix was a four-line change: read_timeout = 30s on the streaming client.
How to prevent it:
Streaming is a special case. A connection that's "alive" but not delivering bytes is the most insidious form of broken. The only defense is treating time-since-last-byte as a vital sign, not just connection state.
What happens: An agent's planner decides the task requires another planner step, which decides it requires another, which decides it requires another. The agent never returns because it never finishes thinking.
Why it happens:
Real example: A research agent designed to "decompose tasks until each step is atomic" got handed an under-specified question. It decomposed it into 14 sub-tasks. Each sub-task got decomposed into 14 more. By the third level the agent was running 2,744 LLM calls and generating tokens at $40/minute. The user gave up after 90 seconds. The agent kept running for 31 minutes before a daily cost cap finally killed it.
How to prevent it:
The default for any agent loop should be: fail loudly when bounded. The worst outcome is silent runaway, the second-worst is a successful response that arrived too late to matter.
What happens: Your agent posts a successful response, then enqueues a background task to write to memory, log feedback, or trigger a follow-up. The background task fails. The user response was correct. The next interaction has missing context. Quality silently degrades over sessions.
Why it happens:
Real example: A customer-support agent's "save conversation summary" background task started failing on a specific edge case (very long conversations with code blocks). 4% of conversations weren't being summarized. New conversations with the same customer no longer had context. Customers got frustrated repeating themselves. The agent looked broken to users, and was actually being broken by a background job nobody had instrumented.
How to prevent it:
The asymmetry of foreground vs. background reliability is where most "the agent feels worse over time" complaints originate. The foreground response works. The background pipeline that maintains state quietly broke.
What happens: Your agent's loop depends on a worker pool you don't directly control, a serverless platform, a third-party agent runtime, a managed orchestrator. A job enters queued, gets accepted, and then disappears. Your audit trail ends at "we sent it."
Why it happens:
Real example: A team using a serverless agent runtime was seeing about 2% of their requests "succeed", the runtime returned 202 Accepted, but never produce output. The runtime had a queue eviction policy that kicked in under load and never notified the caller. The team only found out when they cross-referenced their submitted-job log against their received-result log and noticed the gap.
How to prevent it:
failed.If you only count failures the provider tells you about, you're underreporting. The most honest reliability number is 1 - (terminal_jobs / submitted_jobs).
What happens: The agent calls tool_a. tool_a returns. The agent re-evaluates and decides to call tool_a again with slightly different inputs. Then again. Then again. The agent isn't broken, it's just convinced the next call is the one that will work. Twelve seconds in, the user gives up.
Why it happens:
{"result": null} instead of an explicit "no data" signalReal example: A travel agent designed to "search until you find availability" called the flights API 18 times in a single user turn after the first call returned no results. Each call took 1.5s. The user waited 27 seconds for a response that simply said "I couldn't find any flights matching your criteria." The agent's loop was technically working. The user's experience was completely broken.
How to prevent it:
Six questions every healthy agent observability stack answers:
The minimum infrastructure to answer those:
queued, executing, completed, failed) and timestamps.Agent reliability isn't an HTTP status problem. It's a job-lifecycle problem. The teams that get this right treat agent runs the way payment systems treat transactions: every state transition logged, every terminal state reachable, every stuck record reconciled.
queued then executing then completed | failed. Persist transitions. Don't pass jobs around in memory hoping for the best.AND status IN ('queued','executing')). Run it every 30 seconds.Failures that never reach HTTP are still failures.
Agent loops have steps and lifecycles, not status codes. Reliable teams treat them like distributed systems: explicit state, durable transitions, watchdog reconciliation, bounded loops. The question is whether your monitoring acknowledges that yet.