A structured approach to building and operating reliable AI agents.
Overview
This framework covers principles, practices, metrics, and processes for agent reliability. Designed to be adopted incrementally - start where you are, improve continuously.
Reliability Principles
Principle 1: Semantic Correctness Matters
An agent that responds but gives wrong answers is broken. Monitoring must include semantic validation, not just uptime checks. HTTP 200 doesn't mean "working."
Principle 2: Users Are Everywhere
Your monitoring should test from where your users are, not where your servers are. Single-location monitoring misses regional failures.
Principle 3: Latency Is Correctness
A correct response that takes too long is a failed response. Performance is a reliability concern - users abandon slow agents.
Principle 4: Failures Are Normal
Things will break. Invest in detection and recovery, not just prevention. Optimize for MTTD and MTTR.
Principle 5: Measure What Matters
If you're not measuring it, you're not managing it. Instrument everything. Track trends. Make decisions based on metrics.
Reliability Practices
Practice 1: Continuous Validation
Run semantic validation continuously, not just at deployment. Agents can degrade from model changes, dependency failures, and configuration drift.
- Configure evaluation prompts that verify core functionality
- Run checks every 5-15 minutes
- Track pass rates over time
- Alert on declining rates
Practice 2: Geographic Distribution
Test from multiple locations matching your user base. Use residential IPs where possible. Track per-region metrics and alert on regional degradation.
Practice 3: Threshold-Based Alerting
Use thresholds to determine status, not individual failures. A single failed test doesn't mean your agent is down. Require sustained failure for DOWN verdict.
Practice 4: Latency Budgets
Allocate specific latency budgets to components. Knowing "context retrieval took 80% of the latency budget" tells you where to optimize.
Practice 5: Incident Learning
Treat every incident as a learning opportunity. Conduct blameless post-mortems, document root causes, and track action items.
Reliability Metrics
Primary Metrics
| Metric | Definition | Target Range |
|---|---|---|
| Availability | % of checks returning valid response | 99-99.9% |
| Eval Pass Rate | % of evaluation prompts answered correctly | 95-99% |
| TTFB P50 | Median time to first byte | <1-3s |
| TTFB P95 | 95th percentile TTFB | <3-6s |
Operational Metrics
| Metric | Definition | Target |
|---|---|---|
| MTTD | Mean time to detect incidents | <15 min |
| MTTR | Mean time to resolve incidents | <2 hours |
| Incident Count | Number of incidents per period | Minimize |
| Change Failure Rate | % of deploys causing incidents | <5% |
Reliability Processes
Process 1: Change Management
- Test changes in staging environment
- Validate staging with semantic checks
- Deploy to production
- Run immediate post-deploy validation
- Monitor for 24 hours
- Roll back if quality degrades
Process 2: Incident Response
Response: Acknowledge → assemble response team → begin investigation.
Mitigation: Implement workaround or rollback → stabilize → confirm resolution.
Post-Incident: Conduct post-mortem → document findings → track action items.
Process 3: On-Call
- Rotation schedule with escalation paths
- Runbooks for common issues
- Acknowledge alerts within 15 minutes
- Document actions taken
Implementation Roadmap
Phase 1: Foundation (Weeks 1-2)
Set up agent monitoring, configure basic evaluation prompts, enable alerting for DOWN status, document baseline.
Phase 2: Expansion (Weeks 3-4)
Add geographic distribution, implement TTFB tracking, configure DEGRADED alerts, set up SLA dashboards.
Phase 3: Operations (Weeks 5-8)
Define incident response process, create on-call rotation, develop runbooks, implement post-mortem practice.
Phase 4: Optimization (Ongoing)
Review metrics weekly, tune alert thresholds, improve MTTD/MTTR, reduce incident frequency.
Reliability Culture
Blameless Culture
Focus on systems, not individuals. Ask "what failed?" not "who failed?" Encourage reporting of near-misses and share learnings widely.
Ownership
Teams own their agents' reliability. Engineering time is allocated to reliability. Reliability is a first-class requirement.
Common Anti-Patterns
"We'll add monitoring later" - Every week without monitoring is a week of invisible failures.
"Tests are enough" - Tests verify code. Monitoring verifies production.
"We can't afford downtime" - Fear of change leads to fragile systems. Use safe deployment practices.
"We need perfect reliability" - 100% is impossible. Set achievable targets, improve incrementally.
