Beyond Uptime: What Real Agent Monitoring Looks Like

"What's your agent's uptime?"

It's the first question everyone asks. And it's the wrong question.

Uptime measures whether your agent responds. It doesn't measure whether your agent works.

For AI agents, that distinction is everything.

The Uptime Illusion

Traditional uptime monitoring answers: "Did the endpoint return a response?"

For most software, this is sufficient. A payments API either processes the payment or returns an error. A database query either returns data or fails.

AI agents don't work that way.

An agent can return a perfect 200 response that is:

• Completely wrong
• Uselessly generic
• A polite refusal to help
• Correct but delivered in 45 seconds

All of these are failures. None trigger uptime alerts.

What Real Monitoring Measures

Layer 1: Reachability

"Can we reach the agent at all?"

• DNS resolution
• TCP connection
• TLS handshake
• HTTP response

This is traditional uptime. It's necessary but not sufficient.

Catches: Complete outages, infrastructure failures

Layer 2: Responsiveness

"Does the agent respond in acceptable time?"

• Time to First Byte (TTFB)
• Total response time
• Latency percentiles (P50, P95, P99)

For streaming agents, TTFB is critical-it's when the user sees "typing..."

Catches: Performance degradation, cold starts, upstream slowdowns

Layer 3: Schema Validity

"Is the response correctly formatted?"

• Valid JSON
• Expected fields present
• Correct data types
• Proper structure

Catches: Broken response formatting, API changes

Layer 4: Semantic Correctness

"Does the agent actually work?"

This is where AI-specific monitoring begins.

Evaluation prompts are queries with known-correct answers:

• "What is 2+2?" → Must contain "4"
• "Return JSON: {x:1}" → Must be valid JSON with x:1
• "Echo: hello" → Must contain "hello"

If the agent fails these, it's broken-regardless of HTTP status.

Catches: "Up but broken" states, model degradation, auth failures

Layer 5: Geographic Consistency

"Does the agent work everywhere?"

Test from multiple locations:

• Different continents
• Residential networks (not just data centers)
• Various network conditions

Catches: Regional outages, CDN issues, geographic rate limiting

The Gold Prompt Framework

Evaluation prompts are the core of semantic validation. Here's how to design them:

Health Tier (Basic Functionality)

Purpose: Verify the agent can think and respond

Characteristics:

• Any reasonable LLM can answer
• Answer can appear anywhere in response
• Doesn't depend on specific knowledge

Examples:

• "What is 2+2?" (basic reasoning)
• "What color is the sky?" (common knowledge)
• "Repeat: hello world" (basic instruction following)

Pass criteria: Answer must be present somewhere in response

Contract Tier (Format Compliance)

Purpose: Verify the agent follows expected format

Characteristics:

• Tests structured output capability
• Format must be exact or schema-compliant
• Tests what API consumers actually need

Examples:

• "Return JSON: {"status": "ok"}" (JSON compliance)
• "List three items, one per line" (format compliance)
• "Respond in exactly one sentence" (constraint following)

Pass criteria: Response must match expected format precisely

Domain Tier (Application-Specific)

Purpose: Verify domain-specific functionality

Characteristics:

• Specific to your agent's use case
• Tests actual functionality
• May require configuration

Examples (for a support bot):

• "What is your company's return policy?" (RAG verification)
• "Create a support ticket for: test" (tool use verification)
• "Summarize this conversation: [transcript]" (capability test)

Pass criteria: Application-specific correctness

Metrics That Matter

Primary Metrics

Secondary Metrics

Derived Metrics

Verdict System

Raw metrics need interpretation. Here's our framework:

UP

All systems normal.

• Reachability ≥95%
• Eval pass rate ≥80%
• TTFB SLA met
• No regional failures

DEGRADED

Something's wrong, but it's working.

• Reachability 20-95%, OR
• Eval pass rate <80%, OR
• TTFB SLA exceeded, OR
• Partial regional failures

DOWN

Not working.

• Reachability <20%
• Majority failures
• Consistent errors

Why Thresholds?

We use thresholds, not worst-case, for a reason.

A single failed test shouldn't tank your status. Networks are flaky. Devices have issues. Transient errors happen.

What matters is: "Is the agent generally working?"

Threshold-based verdicts answer that question without creating alert fatigue.

Alert Design

What to Alert On

What NOT to Alert On

• Single failed test (use thresholds)
• Transient latency spikes (use percentiles)
• Expected maintenance windows (use suppression)
• Known issues being worked (use acknowledgment)

Alert Channels

Match alert urgency to channel urgency.

Implementation Checklist

Minimum Viable Monitoring

• Basic uptime check
• One gold prompt (health tier)
• TTFB tracking
• Alerting on DOWN

Production-Ready Monitoring

• Multiple evaluation prompts (health + contract)
• Geographic distribution (3+ regions)
• Latency percentiles (P50, P95, P99)
• Regional breakdown
• Alerting on DEGRADED
• Recovery alerts

Enterprise Monitoring

• Domain-specific evaluation prompts
• Custom SLA thresholds
• Webhook integration
• Trend analysis
• SLA reporting
• Multi-agent coordination monitoring

The Monitoring Maturity Model

Level 0: None

No monitoring. Learn about issues from customers.

Level 1: Basic Uptime

Ping checks. Know when it's completely down.

Level 2: Performance Aware

Latency tracking. Know when it's slow.

Level 3: Semantically Aware

Evaluation prompts. Know when it's broken but responding.

Level 4: Geographically Aware

Multi-region tests. Know when it's broken somewhere.

Level 5: Predictive

Trend analysis, proactive alerts. Know before it breaks.

Most teams are at Level 1 or 2. The opportunity is Levels 3-5.