AgentStatus
We tested every public MCP server we could find. Calling is solved. Delivery is not.
Read the reportHere's a question that traditional monitoring can't answer:
"Is my agent working correctly?"
Not "is it responding?" Not "is it fast?" Is it actually, semantically, doing what it's supposed to do?
This is the problem evaluation prompts solve.
An AI agent returns HTTP 200. The response is valid JSON. Latency is 1.2 seconds.
Monitoring says: everything's fine.
But the response is: "I'm sorry, I cannot help with that request. Please try again later."
To every single query.
The agent is "up." It's also completely broken.
We call this state "up but broken," and traditional monitoring is blind to it.
Evaluation prompts are queries with known-correct answers.
The concept comes from machine learning, where "gold data" or "gold standard" refers to labeled examples used as ground truth.
Applied to agent monitoring:
| Gold Prompt | Expected Answer |
|---|---|
| "What is 2+2?" | Contains "4" |
| Return JSON: {"x": 1} | Valid JSON with x: 1 |
| "Echo exactly: hello" | Contains "hello" |
These are deliberately simple. Any functioning LLM can answer them. If your agent can't, something is very wrong.
The key property of evaluation prompts is determinism. There's exactly one correct answer category, and it's obvious.
This allows automated verification:
def verify_gold_prompt(response, expected):
return expected.lower() in response.lower()No human judgment required. No ambiguity. Pass or fail.
Evaluation prompts are trivially easy. This is intentional.
If we used complex prompts, failures could be:
With trivial prompts, failures mean one thing: something is broken.
Different evaluation prompts test different failure modes:
| Prompt Type | Tests For |
|---|---|
| Math | Basic reasoning works |
| Echo | Instruction following works |
| JSON | Structured output works |
| Knowledge | Model is loaded/responding |
A comprehensive gold prompt suite catches most failure modes.
Purpose: Verify basic functionality.
Characteristics:
Examples:
Prompt: "What is 2+2?"
Pass: "4", "The answer is 4", "2+2 equals 4, which is..."
Fail: "I cannot answer that", "Error", "5"Prompt: "What color is the sky on a clear day?"
Pass: "blue", "The sky is blue", "Blue, typically"
Fail: "I don't know", "Red", "[Error message]"What failures indicate:
Purpose: Verify format compliance.
Characteristics:
Examples:
Prompt: "Return JSON: {\"status\": \"ok\"}"
Pass: {"status": "ok"}
Fail: "Here's the JSON: {\"status\":\"ok\"}", {"status": "error"}Prompt: "Respond with exactly one word: yes or no"
Pass: "yes", "no"
Fail: "Yes, I can help", "Yes" (with punctuation issues)What failures indicate:
Purpose: Verify expected agent behavior.
Characteristics:
Examples (for a customer support agent):
Prompt: "What is your company's name?"
Pass: "[Your company name]"
Fail: Generic response, "I don't know"Prompt: "I need to return a product"
Pass: Contains return policy or starts return flow
Fail: "I cannot help with returns"What failures indicate:
The simplest gold prompt check:
import httpx
GOLD_PROMPTS = [
{"prompt": "What is 2+2?", "expected": "4"},
{"prompt": "Echo: hello", "expected": "hello"},
]
async def check_gold_prompts(endpoint_url, auth_headers):
results = []
for gold in GOLD_PROMPTS:
response = await httpx.post(
endpoint_url,
headers=auth_headers,
json={"messages": [{"role": "user", "content": gold["prompt"]}]}
)
content = response.json()["choices"][0]["message"]["content"]
passed = gold["expected"].lower() in content.lower()
results.append({
"prompt": gold["prompt"],
"passed": passed,
"response": content[:100]
})
return resultsWe run evaluation prompts on every validation check:
The user doesn't need to configure anything. We handle it automatically.
Not every gold prompt failure means DOWN.
Networks are flaky. Models have variance. Transient errors happen.
We use thresholds:
| Eval Pass Rate | Verdict Impact |
|---|---|
| ≥95% | No impact (UP eligible) |
| 80-95% | Warning (may contribute to DEGRADED) |
| <80% | DEGRADED |
| <20% | DOWN |
A single gold prompt failure doesn't trigger alerts. Consistent failures do.
What happens: API key expires, agent returns "Authentication required" to everything.
Gold prompt behavior:
Detection: Immediate on next check.
What happens: Agent hits rate limit, returns polite refusals.
Gold prompt behavior:
Detection: Immediate on next check.
What happens: Model version changes, behavior shifts.
Gold prompt behavior:
Detection: Immediate on next check.
What happens: Context window overflow, agent loses coherence.
Gold prompt behavior:
Detection: Immediate on next check.
What happens: Quality slowly degrades over time.
Gold prompt behavior:
Detection: Trend alerts catch gradual decline.
Evaluation prompts aren't magic. They have limits:
For these, you need:
Evaluation prompts are the smoke detector, not the fire investigator.
Start with 2-3 simple evaluation prompts:
Add more only if needed.
If your agent is a code-only assistant that refuses general queries, adjust:
Evaluation prompts are test cases. Version them with your code:
# gold_prompts.yaml
version: 2
prompts:
- type: health
prompt: "What is 2+2?"
expected: "4"
- type: contract
prompt: "Return JSON: {x:1}"
expected_json: true
expected_key: "x"Use pass rates and thresholds, not individual results.
A drop from 98% to 92% might not trigger alerts but is a warning sign.
Any agent that can't answer "What is 2+2?" is broken. Full stop.
Evaluation prompts are simple — that's the point. For catching "is my agent obviously broken?", nothing beats a trivial query and a sanity check. That catches failures no amount of HTTP status checking ever will.