Agent SLA Handbook

A comprehensive guide to defining, measuring, and enforcing service level agreements for AI agents.

Why Agent SLAs Are Different

Traditional SLAs measure uptime: "Is the service responding?" Agent SLAs must also measure correctness: "Is the agent responding correctly?"

An agent can achieve 99.99% uptime while being semantically broken - returning HTTP 200 with useless or wrong responses. Traditional SLAs miss this entirely.

SLA Components

1. Availability

Definition: Percentage of time the agent endpoint is reachable and responding.

2. Semantic Correctness

Definition: Percentage of responses that are actually correct and useful.

An agent returning "I cannot help" to every query has 100% availability but 0% correctness.

3. Performance

4. Geographic Consistency

Defining Your SLA

Step 1: Understand Your Users

• Where are they located?
• What latency is acceptable for your use case?
• How critical is the agent to their workflow?
• What's the business impact of failures?

Step 2: Establish Baselines

Before committing to targets, measure current availability, eval pass rate, latency percentiles, and regional performance. Set targets you can actually achieve, then improve.

Step 3: Choose Components

Step 4: Define Measurement

For each component, specify the metric, method, frequency, and exclusions (maintenance windows, etc.).

Step 5: Set Consequences

SLAs without teeth are just targets. Consider service credits, termination rights, and reporting requirements.

Measurement Best Practices

• Use third-party monitoring: Provider-reported metrics have inherent conflicts of interest.
• Measure continuously: Spot checks miss issues.
• Include semantic checks: HTTP health checks are necessary but insufficient.
• Track percentiles: Averages hide tail latency.
• Distinguish error types: Separate agent failures from infrastructure failures.

Credit Calculations

Tiered Credits

Always cap maximum credits (per month: 50-100% of monthly fee). Uncapped credits create unbounded liability.

Common Pitfalls

Uptime-Only SLAs: Measuring only HTTP availability misses semantic failures. Include eval pass rate.

Average-Only Latency: Averages hide P99 latency spikes. Use percentiles.

Single-Location Measurement: Checking from one location misses regional failures. Require geographic distribution.

No Baseline: Setting targets without historical data leads to unachievable or meaningless SLAs.

Unclear Exclusions: Vague exclusion language creates disputes. Enumerate specific exclusions precisely.

Example SLA

Implementation

Setting Up Monitoring

1. Configure agent in monitoring system
2. Enable all relevant regions
3. Set check frequency matching SLA period
4. Configure gold prompt validation
5. Enable alerting for SLA violations

Handling Violations

1. Detect violation via monitoring
2. Document incident details
3. Calculate credit owed
4. Communicate with customer
5. Conduct post-incident review