How we rank production AI agents from the outside.
Real tests from home networks, a public scoring formula, and plain-language quality checks. See the live AOI Index for current rankings.
What AOI measures
User-side reliability, not lab capability.
The Agent Outside-In (AOI) Index measures AI agent performance from the perspective of real users. We test production agents from actual consumer devices on residential networks, not datacenter VMs — measuring what users actually experience.
Two pillars. Reachability — did the user path open from real residential nodes (transport, auth, uptime), not AWS synthetic alone. Outcome verification — did it finish the job? Scenario, compositional, safety, and stability checks; job anchors and side-effects where they exist. Explicitly not whether the chatbot reused the same words.
This isn't a benchmark. Benchmarks measure capability in controlled lab conditions. AOI measures reliability in production, continuously.
The headline number is public math: pass rate (up to 60 points), P95 latency (up to 30), plus a fixed verdict bonus (UP +10, DEGRADED +5, otherwise +0). API responses include a per-agent score_breakdown so you can audit the score without trusting a black box. See chat, voice, and MCP validation for the product surface.
Reach from home networks
Hit the production endpoint from residential nodes — same IP class as real users.
Score outcome verification
Did it finish the job — scenario, compositional, safety, and stability checks; job anchors where configured.
Measure latency
Capture P95 response time on the same residential path as the quality probes.
Assign status
UP / DEGRADED / DOWN from pass rate, latency, and critical check outcomes.
Publish the AOI score
Composite 0–100 with a public formula and per-agent score_breakdown.
Why user-side
What datacenter checks miss.
Traditional monitoring runs from datacenters. For public agents behind modern edges, that often measures a different access surface than residential users — cloud ASNs get blocked or throttled while consumer ISPs get through (and the reverse false alarm when a DC path is green but a region is down).
Probe blocked at the edge
Client-IP locale defaults
Rate-limited at the gate
Real path latency
We run scripted validation tests from real consumer devices on real networks — same IP type, region, path, and device conditions that actual users experience. The backend orchestrates; nodes execute the workloads.
Network
The distributed validating network.
Tests originate from AgentStatus's residential vantage network — the same fleet used for Benchmark and the Data Partner program. Runs dispatch to consumer devices on real networks; the backend orchestrates, nodes execute. Today that inventory includes 68 countries with active residential nodes (list below; it updates as the fleet grows).
Active vantage countries (68)
Argentina · Australia · Austria · Bangladesh · Belgium · Benin · Botswana · Brazil · Canada · Chile · China · Colombia · Côte d'Ivoire · Cyprus · Czechia · Ecuador · Egypt · Ethiopia · Finland · France · Georgia · Germany · Ghana · Hong Kong · India · Indonesia · Ireland · Italy · Japan · Kazakhstan · Kenya · Latvia · Lithuania · Madagascar · Malaysia · Malta · Morocco · Mozambique · Netherlands · New Zealand · Nigeria · Norway · Pakistan · Philippines · Poland · Romania · Rwanda · Senegal · Singapore · South Africa · South Korea · South Sudan · Spain · Sri Lanka · Sweden · Switzerland · Taiwan · Tanzania · Thailand · Togo · Tunisia · Turkey · Ukraine · United Arab Emirates · United Kingdom · United States · Zambia · Zimbabwe
Score
AOI score calculation.
The AOI Score is a composite metric from 0 to 100:
Pass rate percent × 0.6 — up to 60 points. Combines reachability with outcome-verification checks (scenario, compositional, safety, stability, job anchors where applicable).
P95 mapped to 0–100 (perfect at ≤1s, zero at ≥30s), then × 0.3 — up to 30 points.
Fixed points, not a percentage weight: UP +10, DEGRADED +5, everything else (including DOWN) +0.
Pass rate
What feeds the pass rate.
Uptime alone is not enough. Pass rate combines reachability with quality checks run from the same home-network tests. Each check answers one plain question — and has an explicit non-claim:
| Check | Plain question | What it does not claim |
|---|---|---|
| Reachability | Did the agent reply from a home network? | Does not mean the answer was good. |
| Spot-check questions | On example prompts, was the answer OK? | Does not cover every real user situation. |
| Answer review | For open-ended replies, did reviewers agree? | Reviewers may not know your domain deeply. |
| Format match | Did structured replies match the expected shape? | Does not prove the content was correct. |
| Job stability | Same job under rephrase, stakes, and follow-ups — did the outcome hold? | Not a claim that the prose matched. Stable wrong is still wrong. |
| Goal-driven conversations | A simulated user pursues a real goal over several messages. Did they get what they came for? | Simulated users, not your live traffic. |
| Outcome double-check | When the agent claimed success, did the claim hold up against outside evidence? | Only where outside evidence exists. |
We also measure streaming behavior (time to first token, stream completion), tool and MCP calls where configured, and geo-specific behavior when location affects the answer. And we track run-after-run reliability — not just whether one reply passed, but whether a whole multi-step session would survive end to end. Those signals inform verdicts and enterprise dashboards; the public AOI score stays the transparent formula above.
Core metrics
What we measure on each run.
| Metric | Description |
|---|---|
| Response Time (P50/P95/P99) | Latency percentiles across all tests |
| Time to First Byte (TTFB) | How fast the agent starts responding. Critical for streaming. |
| Success Rate | Percentage of tests that return valid responses |
| Availability | Whether the agent responds at all |
| Connection Timing | DNS resolution, TCP handshake, TLS negotiation breakdown |
Status
How verdict feeds the score.
The public AOI score uses the agent's current probe verdict as a flat bonus. It does not invent pass-rate bands for UP / DEGRADED / DOWN — those labels come from the validation workload.
UP
DEGRADED
DOWN / other
Protocols
Supported agent types.
We validate agents across multiple protocols and platforms:
| Protocol | Examples | Support |
|---|---|---|
| OpenAI-compatible | GPT, Claude, Groq, Together | Full |
| Anthropic | Claude direct API | Full |
| Streaming SSE | Most modern chat APIs | Full |
| uAgent (Fetch.ai) | AgentVerse agents | Full |
| Poe | Poe bot API | Full |
| Perplexity | Sonar, Deep Research | Full |
| Custom HTTP | Any REST/JSON endpoint | Full |
| MCP | Model Context Protocol servers | Tool validating |
Geography
Geographic context injection.
For agents that serve location-specific content, we inject the test node's actual location into requests. This enables true geo-divergence testing — verify that your agent returns different results for users in Tokyo vs. São Paulo.
Perplexity
Custom agents
The split
What we are, and are not.
We are not
- • An inside-out observability tool
- • A tracing platform
- • A log aggregator
We are
- • An independent third-party validator
- • The user's perspective at scale
- • External audit for SLA claims
FAQ
Common questions.
| Question | Answer |
|---|---|
| Is this a lab benchmark? | No. We test production endpoints from real home networks, on a schedule, the same way a user would hit your agent. Benchmarks score capability in a controlled room. AOI scores reliability in the wild. |
| What goes into the headline AOI score? | Pass rate contributes up to 60 points, P95 latency up to 30, plus a fixed verdict bonus (UP +10, DEGRADED +5, otherwise +0). The API returns a score_breakdown so you can audit each piece. We do not hide the math. |
| What counts as a pass? | Reachability plus quality checks on example conversations. That includes whether the agent replied, whether answers made sense on spot-check questions, whether structured replies matched the expected shape, whether independent reviewers agreed on open-ended answers, and - for goal-driven conversations - whether the simulated user actually got what they came for. |
| Do you test multi-turn conversations? | Yes, where the agent supports it. A simulated user pursues a realistic goal - a refund, a booking, a troubleshooting fix - over several messages, clarifying and pushing back the way a real person would. An independent reviewer then reads the whole transcript and decides one thing: did they get what they came for? |
| Do you take the agent's word for it when it claims success? | No. When an agent says a goal was completed, we double-check the claim against outside evidence where it exists - a cited page, a structured field, a second reviewer - before counting it as a success. |
| How is this different from keyword matching? | We do not score whether the chatbot reused the same words. Outcome verification means did it finish the job — scenario, compositional, safety, and stability checks; job anchors and side-effects where they exist. Paraphrase of a correct job is fine; a fluent wrong job is not. |
| Do you check if answers stay consistent? | We check whether the job stays steady under rephrase, stakes, and follow-ups — not whether the prose matches. Separately, we hit the same endpoint from several residential places to catch access gaps and harness locale branches keyed off client IP — not because residential probing relocates the agent's outbound tools. |
| Can agents pay for a better rank? | No. Rankings come from measured results only. The way to move up is to improve reachability, speed, and outcome verification. |
| Is chat validation the same as voice? | Same philosophy, different transport. Chat and API agents and voice agents are validated from home networks with plain verdicts. See our platform pages for what each modality covers. |
Independence
Trust is measured, not purchased.
The AOI Index is operated by Carmel Labs / AgentStatus as an independent validator. We have no financial relationship with the agents we monitor. Rankings are determined purely by measured performance.
Agent owners cannot pay for better rankings. The only way to improve your AOI score is to improve your agent.