Agent Reliability Playbook

Operational practices for maintaining production agent reliability.

Daily Operations

Morning Check (5 minutes)

1. Check monitoring dashboard - any alerts fired overnight?
2. Review current status - all agents UP?
3. Check latency trends - any spikes?
4. Verify scheduled jobs ran successfully

Weekly Review (30 minutes)

1. Review availability metrics vs SLA
2. Check latency percentile trends
3. Review any incidents from the week
4. Identify agents trending toward DEGRADED
5. Update capacity projections

Monthly SLA Report

Availability percentage per agent, latency percentiles, eval pass rate, incident count and summary, comparison to previous months, action items.

Scenario Playbooks

Scenario: Agent Goes DOWN

Immediate actions (first 5 minutes):

1. Acknowledge alert
2. Verify not false positive (check from different network/device)
3. Check if deploy happened recently
4. Check agent provider status page

Scenario: Agent Shows DEGRADED

Investigation: Which tests are failing? What's the latency distribution? Which evaluation prompts are failing? When did degradation start?

Common causes: Partial infrastructure issue, high load, model quality degradation, partial deployment in progress.

Scenario: Eval Pass Rate Dropping

This is serious - the agent is returning wrong answers.

Common causes: Model update changed behavior, context/prompt configuration changed, upstream data source issue, rate limiting causing truncated responses.

Scenario: Latency Spike

Scenario: Regional Failure

Common causes: CDN configuration issue, regional deployment failure, ISP routing problems, geo-blocking misconfiguration.

Scenario: Alert Storm

Don't panic. Look for common cause (usually infrastructure). Identify if monitoring system itself is healthy. If monitoring healthy: major incident, escalate. If monitoring unhealthy: fix monitoring first.

Troubleshooting Guides

"Can't Reach Agent Endpoint"

1. DNS resolution - Can you resolve the hostname? Is DNS pointing to correct IP?
2. Network connectivity - Can you ping the IP? Are there firewall issues?
3. TLS/SSL - Is the certificate valid? Is it expired?
4. Service running - Is the agent process running? Correct port?
5. Load balancer - Is the LB healthy? Are backends registered?

"Agent Responds But Answers Wrong"

1. Recent changes - Any deploy in last 24 hours?
2. Model state - Is model loaded correctly? Provider issues?
3. Context/RAG - Is context being retrieved? Right context?
4. Prompt configuration - Has system prompt changed?
5. Dependencies - Are all data sources available?

"Intermittent Failures"

Common patterns:

• Regular interval failures → cron job interfering
• Specific times → traffic spike, maintenance
• Random → race condition, resource exhaustion

Communication Templates

Status Page Update (Incident Start)

Status Page Update (Resolved)

Metrics to Watch

Real-Time (Dashboard)

Escalation Paths