Complete guide to agent reliability metrics: definitions, calculations, and usage.
Metric Categories
| Category | Purpose |
|---|---|
| Availability | Is the agent reachable? |
| Correctness | Is the agent giving right answers? |
| Performance | Is the agent fast enough? |
| Operational | How well are we responding to issues? |
| Business | What's the user/business impact? |
Availability Metrics
Uptime
Definition: Percentage of time the agent endpoint returns a valid response.
Uptime = (Successful Checks / Total Checks) × 100Successful check: HTTP 2xx response received within timeout.
| Use Case | Target |
|---|---|
| Internal tool | 99% |
| Production API | 99.5% |
| Critical service | 99.9% |
Availability
Definition: Percentage of time the agent is both reachable AND functional.
Availability = (Checks Passed / Total Checks) × 100
Where "passed" means:
- HTTP response received (2xx or handled 4xx)
- Response within timeout
- Evaluation prompts passed (if configured)Distinction from Uptime: Uptime asks "Does it respond?" - Availability asks "Does it work?"
Effective Uptime
Definition: Availability accounting for semantic correctness.
Effective Uptime = Uptime × (Eval Pass Rate / 100)An agent returning 200 OK with wrong answers has high uptime but low effective uptime.
Correctness Metrics
Eval Pass Rate
Definition: Percentage of evaluation checks returning correct answers.
Eval Pass Rate = (Correct Gold Responses / Total Evaluation Prompts) × 100What makes a response "correct":
- Health tier: Expected answer appears somewhere in response
- Contract tier: Response matches expected format exactly
| Tier | Target |
|---|---|
| Basic | 90% |
| Standard | 95% |
| Premium | 98% |
Health Tier Pass Rate
Pass rate for health-tier evaluation prompts only. These test basic reasoning with loose matching - the expected answer can appear anywhere in the response.
Contract Tier Pass Rate
Pass rate for contract-tier evaluation prompts only. These test format/schema validation with exact match or structured comparison.
Performance Metrics
TTFB (Time to First Byte)
Definition: Time from request initiation until first byte of response received.
For streaming agents, TTFB is when user sees "typing..." - the primary user experience metric for streaming.
| Percentile | Target |
|---|---|
| P50 | <2s |
| P95 | <5s |
| P99 | <10s |
Total Latency
Definition: Time from request initiation until complete response received.
Total Latency = DNS + TCP + TLS + Server Processing + TransferLatency Percentiles
P50 (Median): Half of requests faster, half slower. Represents "typical" experience.
P95: 95% of requests faster. Common SLA target.
P99: 99% of requests faster. Reveals tail latency.
Why percentiles over averages:
Example requests (ms): 100, 100, 100, 100, 5000
Average: 1080ms (misleading)
P50: 100ms (typical experience)
P99: 5000ms (worst case visible)Operational Metrics
MTTD (Mean Time to Detect)
MTTD = Sum(Detection Time - Start Time) / Incident Count| Level | Target |
|---|---|
| Basic | <30 min |
| Good | <15 min |
| Excellent | <5 min |
MTTR (Mean Time to Resolution)
MTTR = Sum(Resolution Time - Detection Time) / Incident Count| Severity | Target |
|---|---|
| Critical | <1 hour |
| High | <4 hours |
| Medium | <24 hours |
Change Failure Rate
CFR = (Deployments Causing Incidents / Total Deployments) × 100| Level | Target |
|---|---|
| Concerning | >10% |
| Acceptable | 5-10% |
| Good | <5% |
| Elite | <1% |
Verdicts
Verdict Definitions
UP: Availability ≥95%, Eval Pass Rate ≥80%, Latency within SLA, Majority of tests successful.
DEGRADED: Between UP and DOWN thresholds. Partial functionality, some tests failing, elevated latency.
DOWN: Availability <20%. Majority of tests failing, complete or near-complete outage.
UNKNOWN: Insufficient data, all tests failed, or initial state before first check.
Verdict Calculation
if probe_pass_rate >= 95% AND gold_pass_rate >= 80% AND latency_ok:
verdict = UP
elif probe_pass_rate < 20%:
verdict = DOWN
else:
verdict = DEGRADEDSystem Verdict
TARGET_EVALUATED: Test successfully reached agent. Result reflects actual agent state. Counts toward availability.
VALIDATOR_ERROR: Test failed due to monitoring infrastructure. Not agent's fault. Excluded from availability.
Error Budget
Error Budget = 100% - SLA Target
For 99.9% SLA:
Error Budget = 0.1%
Per month (43,200 minutes): 43.2 minutes allowedError Budget Policy
Budget healthy (>50% remaining): Normal operations, deployments allowed.
Budget low (<25% remaining): Increased caution, limit risky changes.
Budget exhausted: Stop non-essential changes, reliability focus only.
Metric Anti-Patterns
Average-Only Latency
Averages hide tail latency. Always use percentiles (P50, P95, P99).
Uptime Without Correctness
Misses "up but broken" state. Include gold prompt validation.
Single-Location Metrics
Misses regional failures. Track per-region and global separately.
Point-in-Time Only
Misses trends and intermittent issues. Track over time, analyze trends.
Dashboard Recommendations
Real-Time Dashboard
Current verdict per agent, last check time, rolling 1-hour availability, current TTFB P50/P95, active alerts. Refresh every 1-5 minutes.
SLA Dashboard
Period availability vs target, error budget remaining, eval pass rate, latency percentiles vs SLA, incident count. Month-to-date with historical comparison.
Trend Dashboard
Availability over time (30/60/90 days), latency trends, incident frequency, change failure rate, MTTR trend. Purpose: spot degradation before incidents.
