How we rank production AI agents from the outside.
Real tests from home networks, a public scoring formula, and plain-language quality checks. See the live AOI Index for current rankings.
What AOI measures
Outside-in reliability, not lab capability.
The Agent Outside-In (AOI) Index measures AI agent performance from the perspective of real users. We test production agents from actual consumer devices on residential networks, not datacenter VMs — measuring what users actually experience.
This isn't a benchmark. Benchmarks measure capability in controlled lab conditions. AOI measures reliability in production, continuously.
The headline number is public math: pass rate (up to 60 points), P95 latency (up to 30), plus a fixed verdict bonus (UP +10, DEGRADED +5, otherwise +0). API responses include a per-agent score_breakdown so you can audit the score without trusting a black box. See chat, voice, and MCP validation for the product surface.
Reach from home networks
Hit the production endpoint from residential nodes — same IP class as real users.
Run quality checks
Spot-checks, format, consistency, goal-driven conversations where supported.
Measure latency
Capture P95 response time on the same residential path as the quality probes.
Assign status
UP / DEGRADED / DOWN from pass rate, latency, and critical check outcomes.
Publish the AOI score
Composite 0–100 with a public formula and per-agent score_breakdown.
Why outside-in
What datacenter checks miss.
Traditional monitoring runs from datacenters. For public agents behind modern edges, that often measures a different access surface than residential users — cloud ASNs get blocked or throttled while consumer ISPs get through (and the reverse false alarm when a DC path is green but a region is down).
Probe blocked at the edge
Client-IP locale defaults
Rate-limited at the gate
Real path latency
We run scripted validation tests from real consumer devices on real networks — same IP type, region, path, and device conditions that actual users experience. The backend orchestrates; nodes execute the workloads.
Network
The distributed validating network.
Tests originate from AgentStatus's residential vantage network — the same fleet used for Benchmark and the Data Partner program. Runs dispatch to consumer devices on real networks; the backend orchestrates, nodes execute. Today that inventory includes 68 countries with active residential nodes (list below; it updates as the fleet grows).
Active vantage countries (68)
Argentina · Australia · Austria · Bangladesh · Belgium · Benin · Botswana · Brazil · Canada · Chile · China · Colombia · Côte d'Ivoire · Cyprus · Czechia · Ecuador · Egypt · Ethiopia · Finland · France · Georgia · Germany · Ghana · Hong Kong · India · Indonesia · Ireland · Italy · Japan · Kazakhstan · Kenya · Latvia · Lithuania · Madagascar · Malaysia · Malta · Morocco · Mozambique · Netherlands · New Zealand · Nigeria · Norway · Pakistan · Philippines · Poland · Romania · Rwanda · Senegal · Singapore · South Africa · South Korea · South Sudan · Spain · Sri Lanka · Sweden · Switzerland · Taiwan · Tanzania · Thailand · Togo · Tunisia · Turkey · Ukraine · United Arab Emirates · United Kingdom · United States · Zambia · Zimbabwe
Score
AOI score calculation.
The AOI Score is a composite metric from 0 to 100:
Pass rate percent × 0.6 — up to 60 points. Combines reachability with quality checks (spot-checks, review, format, consistency, goal outcomes where applicable).
P95 mapped to 0–100 (perfect at ≤1s, zero at ≥30s), then × 0.3 — up to 30 points.
Fixed points, not a percentage weight: UP +10, DEGRADED +5, everything else (including DOWN) +0.
Pass rate
What feeds the pass rate.
Uptime alone is not enough. Pass rate combines reachability with quality checks run from the same home-network tests. Each check answers one plain question — and has an explicit non-claim:
| Check | Plain question | What it does not claim |
|---|---|---|
| Reachability | Did the agent reply from a home network? | Does not mean the answer was good. |
| Spot-check questions | On example prompts, was the answer OK? | Does not cover every real user situation. |
| Answer review | For open-ended replies, did reviewers agree? | Reviewers may not know your domain deeply. |
| Format match | Did structured replies match the expected shape? | Does not prove the content was correct. |
| Consistency | Same situation — repeated, rephrased, higher stakes. Did the story stay steady? | Stable does not always mean right. |
| Goal-driven conversations | A simulated user pursues a real goal over several messages. Did they get what they came for? | Simulated users, not your live traffic. |
| Outcome double-check | When the agent claimed success, did the claim hold up against outside evidence? | Only where outside evidence exists. |
We also measure streaming behavior (time to first token, stream completion), tool and MCP calls where configured, and geo-specific behavior when location affects the answer. And we track run-after-run reliability — not just whether one reply passed, but whether a whole multi-step session would survive end to end. Those signals inform verdicts and enterprise dashboards; the public AOI score stays the transparent formula above.
Core metrics
What we measure on each run.
| Metric | Description |
|---|---|
| Response Time (P50/P95/P99) | Latency percentiles across all tests |
| Time to First Byte (TTFB) | How fast the agent starts responding. Critical for streaming. |
| Success Rate | Percentage of tests that return valid responses |
| Availability | Whether the agent responds at all |
| Connection Timing | DNS resolution, TCP handshake, TLS negotiation breakdown |
Status
How verdict feeds the score.
The public AOI score uses the agent's current probe verdict as a flat bonus. It does not invent pass-rate bands for UP / DEGRADED / DOWN — those labels come from the validation workload.
UP
DEGRADED
DOWN / other
Protocols
Supported agent types.
We validate agents across multiple protocols and platforms:
| Protocol | Examples | Support |
|---|---|---|
| OpenAI-compatible | GPT, Claude, Groq, Together | Full |
| Anthropic | Claude direct API | Full |
| Streaming SSE | Most modern chat APIs | Full |
| uAgent (Fetch.ai) | AgentVerse agents | Full |
| Poe | Poe bot API | Full |
| Perplexity | Sonar, Deep Research | Full |
| Custom HTTP | Any REST/JSON endpoint | Full |
| MCP | Model Context Protocol servers | Tool validating |
Geography
Geographic context injection.
For agents that serve location-specific content, we inject the test node's actual location into requests. This enables true geo-divergence testing — verify that your agent returns different results for users in Tokyo vs. São Paulo.
Perplexity
Custom agents
The split
What we are, and are not.
We are not
- • An inside-out observability tool
- • A tracing platform
- • A log aggregator
We are
- • An independent third-party validator
- • The user's perspective at scale
- • External audit for SLA claims
FAQ
Common questions.
| Question | Answer |
|---|---|
| Is this a lab benchmark? | No. We test production endpoints from real home networks, on a schedule, the same way a user would hit your agent. Benchmarks score capability in a controlled room. AOI scores reliability in the wild. |
| What goes into the headline AOI score? | Pass rate contributes up to 60 points, P95 latency up to 30, plus a fixed verdict bonus (UP +10, DEGRADED +5, otherwise +0). The API returns a score_breakdown so you can audit each piece. We do not hide the math. |
| What counts as a pass? | Reachability plus quality checks on example conversations. That includes whether the agent replied, whether answers made sense on spot-check questions, whether structured replies matched the expected shape, whether independent reviewers agreed on open-ended answers, and - for goal-driven conversations - whether the simulated user actually got what they came for. |
| Do you test multi-turn conversations? | Yes, where the agent supports it. A simulated user pursues a realistic goal - a refund, a booking, a troubleshooting fix - over several messages, clarifying and pushing back the way a real person would. An independent reviewer then reads the whole transcript and decides one thing: did they get what they came for? |
| Do you take the agent's word for it when it claims success? | No. When an agent says a goal was completed, we double-check the claim against outside evidence where it exists - a cited page, a structured field, a second reviewer - before counting it as a success. |
| How is this different from keyword matching? | Good answers do not always use the same words. We judge whether a reply is helpful and on-topic, so agents are not penalized for paraphrasing a correct answer. |
| Do you check if answers stay consistent? | Yes, in several ways. We repeat the same question, rephrase it, raise the stakes, add constraints, and chain follow-ups - the story should stay steady through all of it. We also ask the exact same question from several places at once to catch access gaps and harness locale branches keyed off client IP — not because residential probing relocates the agent's outbound tools. |
| Can agents pay for a better rank? | No. Rankings come from measured results only. The way to move up is to improve reachability, speed, and answer quality. |
| Is chat validation the same as voice? | Same philosophy, different transport. Chat and API agents and voice agents are validated from home networks with plain verdicts. See our platform pages for what each modality covers. |
Independence
Trust is measured, not purchased.
The AOI Index is operated by Carmel Labs / AgentStatus as an independent validator. We have no financial relationship with the agents we monitor. Rankings are determined purely by measured performance.
Agent owners cannot pay for better rankings. The only way to improve your AOI score is to improve your agent.