Latency is the single variable that separates an AI agent users tolerate from one they abandon. The numbers below are drawn from representative outside-in measurements against the public endpoints of eight major model providers from three geographic regions. They are illustrative - provider performance changes week to week, and the only numbers you should make product decisions on are the ones from validations running against your own traffic. What holds across runs is the shape of the problem: p50 vs p99 gaps, geographic variance, and why TTFB, not total latency, is the metric to watch.
Methodology is reproducible; the sample numbers in the table below are a representative snapshot rather than a live feed. Run the same validations against your own endpoints to get numbers that reflect your traffic.
TL;DR
- Groq typically ranks best on p50 TTFB thanks to its custom LPU stack.
- Anthropic tends to rank best on p99 stability - the smallest tail across regions.
- Geographic variance is real: ~2x deltas between the fastest and slowest region for the same provider are common on Gemini, especially during capacity events.
- The gap between p50 and p99 varies 5–20x across providers. Choose on p99 for user-facing agents, p50 for batch.
Methodology
We want other teams to be able to reproduce or replace this benchmark. The setup:
Endpoints tested
- OpenAI - GPT-4o via chat completions
- Anthropic - Claude Sonnet 4.6 via messages
- Google - Gemini 2.5 Pro via generateContent
- Mistral - mistral-large via chat completions
- Groq - llama-3.3-70b via chat completions
- Together - llama-3.3-70b via chat completions
- Cohere - command-r-plus via chat
- Fireworks - qwen2.5-72b via chat completions
Prompt
Every provider received the same 120-token prompt: a deterministic question about a JSON schema with a 40-token expected response. This keeps output length comparable and avoids measuring how long each provider takes to generate a 2,000-token essay.
Metrics captured
- TTFB - time to first byte of the response, measured from the outside.
- Total latency - time from request start to last byte received.
- Tokens/second - decoded from streamed responses.
- Error rate - non-2xx responses per thousand.
Regions
- us-east-1 (Virginia)
- eu-west-1 (Ireland)
- ap-southeast-1 (Singapore)
Cadence
- Validations at the tightest cadence each plan supports. On Agent Status, paid plans run as short as 5-minute intervals; free-tier runs every 6 hours. For this methodology, the important property is consistency - same prompt, same cadence, same regions across providers.
Headline numbers (illustrative)
Representative TTFB p50 / p99 by provider, averaged across three regions (milliseconds). Use these as a shape, not a source of truth - the absolute numbers are snapshot-dependent.
| Provider | Model | TTFB p50 | TTFB p99 | Total p50 | Err rate |
|---|---|---|---|---|---|
| Groq | llama-3.3-70b | 180 | 2,200 | 780 | 0.12% |
| OpenAI | GPT-4o | 420 | 3,800 | 1,450 | 0.08% |
| Anthropic | Sonnet 4.6 | 510 | 2,900 | 1,720 | 0.04% |
| Fireworks | qwen2.5-72b | 540 | 4,200 | 1,910 | 0.21% |
| Together | llama-3.3-70b | 620 | 5,400 | 2,100 | 0.34% |
| Mistral | mistral-large | 690 | 4,700 | 2,240 | 0.18% |
| Cohere | command-r-plus | 740 | 3,900 | 2,310 | 0.11% |
| Gemini 2.5 Pro | 820 | 6,100 | 2,580 | 0.27% |
What these numbers actually mean
1. p50 is the marketing number. p99 is the user-experience number.
If a provider has a great p50 and a terrible p99, one in every hundred user interactions with your agent is going to feel broken. For a conversational agent making 20 calls per session, that's 20% of sessions with at least one bad call. Always tune on p99.
2. Geographic variance is not just a PoP problem.
Some providers have strong presence in every major cloud region; others are effectively single-region behind a CDN. If your users are global, the provider's p50 from the other side of the planet is what they experience, not the provider's marketing benchmark.
3. TTFB is a better leading indicator than total latency.
Total latency depends on output length, which varies by prompt. TTFB doesn't. If you're trying to catch provider degradation in real time, watch TTFB - it's the first thing to move when something is wrong on their side. For more on the distinction, see our TTFB vs latency explainer.
4. The error rate column hides what matters.
Most providers stay under 0.5% over a month. What you actually care about is whether those errors cluster - if you get 200 errors at the same minute, you have an incident, not an error rate. Always alert on change, not on absolute thresholds.
How we'd pick a provider
If you're starting fresh and can pick any provider, the heuristic we'd use - from this data alone - is:
- User-facing chat agent with global users: optimise for TTFB p99 and geographic consistency.
- Batch / async agent: optimise for total p50 and tokens/sec.
- Cost-sensitive at scale: start with a provider whose p50 meets your SLO and whose price is lowest, and only move up when the p99 starts hurting users.
And in all cases: monitor outside-in. These benchmark numbers are a snapshot. Provider performance changes week to week, especially when new models launch. If you don't have an outside-in validate running, you won't know until your users do.
Reproducing this benchmark yourself
Every tool we used is public. The validate scripts, assertion configs, and regional vantage points are what Agent Status runs for every customer by default. If you want to run this against your own endpoints - including ones behind auth - you can set up the same validations on your own agent in about 10 minutes. Framework-specific starting points: LangChain, Anthropic SDK, OpenAI Assistants.
What's next
We plan to publish updated numbers quarterly as we expand our public dataset. If there's a specific provider or model you want us to add, let us know. In the meantime, if you want live per-provider dashboards, see OpenAI status monitoring and Anthropic status monitoring.
