Definitions of key terms used in AI agent monitoring, validation, and reliability engineering.
A
Agent
A software system that uses AI/ML models to perform tasks autonomously or semi-autonomously. Agents typically accept natural language inputs and produce outputs through reasoning, tool use, or direct response.
Availability
The percentage of time an agent is operational and able to serve requests. Traditionally measured as uptime, but for agents should also consider semantic correctness. Formula: (Successful checks / Total checks) × 100
C
Cold Start
The delay that occurs when an agent instance must be initialized before handling a request. Common in serverless deployments. Cold starts can significantly impact TTFB.
Contract Tier (Evaluation Prompts)
A category of evaluation prompts that verify format compliance rather than just correctness. Contract tier prompts test whether the agent follows expected output schemas. Example: "Return JSON: {x:1}" expects valid JSON with the specified structure.
D
DEGRADED
A verdict indicating partial functionality. The agent is responding but has issues-high latency, some failed tests, or correctness problems. Between UP and DOWN.
DOWN
A verdict indicating the agent is not functioning. Characterized by majority test failures, consistent errors, or complete unreachability.
E
Effective Uptime
Availability that accounts for semantic correctness, not just HTTP responses. An agent returning errors in valid JSON shows high traditional uptime but low effective uptime. Formula: Reachability × Eval Pass Rate
Error Budget
The acceptable amount of downtime or failures within an SLA period. If your SLA is 99.9%, your error budget is 0.1% of the period. Example: 99.9% monthly SLA = 43 minutes of allowed downtime per month.
G
Geographic Validation
Testing an agent from multiple physical locations to verify it works consistently worldwide. Catches regional failures that single-location monitoring misses.
Evaluation Prompts
Deterministic queries with known-correct answers used to verify agent functionality. If an agent can't answer "What is 2+2?" correctly, it's broken regardless of HTTP status. Types include health tier (basic reasoning) and contract tier (format compliance).
Eval Pass Rate
The percentage of evaluation prompts answered correctly. Formula: (Correct gold responses / Total evaluation prompts) × 100
H
Health Tier (Evaluation Prompts)
A category of evaluation prompts that verify basic functionality. Health tier prompts have loose matching-the correct answer can appear anywhere in the response. Example: "What is 2+2?" expects "4" somewhere in the response.
I
Incident
An event where an agent fails to meet its SLA or expected behavior. Incidents have duration (time from detection to resolution) and impact (users affected, revenue lost).
L
Latency
The time between sending a request and receiving the complete response. For agents, distinguish between TTFB (time to first byte) and total latency.
Latency Percentiles
Statistical measures of latency distribution: P50 (median), P95 (95% of requests faster), P99 (99% of requests faster). Percentiles reveal tail latency that averages hide.
M
MTTR (Mean Time to Resolution)
Average time from incident detection to resolution. Lower is better.
MTTD (Mean Time to Detection)
Average time from incident occurrence to detection. With proper monitoring, should be minutes. Without monitoring, can be hours.
N
Node
In Agent Status's context, a device that executes validation tests. Nodes are real devices (laptops, desktops) running in various locations worldwide, providing realistic network conditions.
T
Test
A single validation request sent to an agent. A test includes the HTTP request, response capture, timing measurement, and semantic validation.
Test Pass Rate
Percentage of tests that succeed. Used with thresholds to determine verdicts-a few failed tests don't mean the agent is DOWN.
R
Reachability
Whether the agent endpoint responds at all. The most basic health metric-necessary but not sufficient for determining agent health.
Residential IP
An IP address assigned by an ISP to a home or business, as opposed to a data center IP. Residential IPs experience real-world network conditions that data center monitoring might miss.
S
Semantic Correctness
Whether an agent's responses are actually correct and useful, beyond just being well-formed. Measured through evaluation prompts and response validation.
SLA (Service Level Agreement)
A commitment to specific reliability targets. Agent SLAs should include availability, latency, and semantic correctness-not just uptime.
Streaming
A response delivery method where content is sent incrementally as it's generated. For streaming responses, TTFB is the primary UX metric.
System Verdict
Classification of whether a test failure is the agent's fault (TARGET_EVALUATED) or the monitoring infrastructure's fault (VALIDATOR_ERROR). Validator errors don't count against agent health.
TARGET_EVALUATED
A system verdict indicating the test successfully reached the agent. The result (pass or fail) reflects actual agent behavior.
Threshold
A boundary value used to classify status. Agent Status uses thresholds to prevent single failures from triggering alerts-e.g., DOWN requires <20% test success, not a single failure.
TTFB (Time to First Byte)
Time from request initiation until the first byte of response arrives. For streaming agents, TTFB is when the user sees "typing..." and is the primary user experience metric.
U
UP
A verdict indicating the agent is functioning correctly. Requires high reachability (≥95%), high eval pass rate (≥80%), and SLA compliance.
"Up But Broken"
A state where an agent returns HTTP 200 but is semantically broken-returning errors, wrong answers, or useless responses. Traditional monitoring misses this; evaluation prompts catch it.
V
VALIDATOR_ERROR
A system verdict indicating the test failed due to monitoring infrastructure issues, not agent problems. These failures are excluded from agent health calculations.
Verdict
Agent Status's overall health assessment for an agent: UP, DEGRADED, or DOWN. Verdicts aggregate multiple metrics using thresholds to provide a single status.
W
Webhook
An HTTP callback triggered when events occur. Agent Status sends webhooks on status changes (UP→DOWN, etc.) enabling integration with alerting systems.
