Here's a question that traditional monitoring can't answer:
"Is my agent working correctly?"
Not "is it responding?" Not "is it fast?" Is it actually, semantically, doing what it's supposed to do?
This is the problem evaluation prompts solve.
The Problem
An AI agent returns HTTP 200. The response is valid JSON. Latency is 1.2 seconds.
Monitoring says: everything's fine.
But the response is: "I'm sorry, I cannot help with that request. Please try again later."
To every single query.
The agent is "up." It's also completely broken.
We call this state "up but broken," and traditional monitoring is blind to it.
What Are Evaluation Prompts?
Evaluation prompts are queries with known-correct answers.
The concept comes from machine learning, where "gold data" or "gold standard" refers to labeled examples used as ground truth.
Applied to agent monitoring:
| Gold Prompt | Expected Answer |
|---|---|
| "What is 2+2?" | Contains "4" |
| Return JSON: {"x": 1} | Valid JSON with x: 1 |
| "Echo exactly: hello" | Contains "hello" |
These are deliberately simple. Any functioning LLM can answer them. If your agent can't, something is very wrong.
Why This Works
Principle 1: Determinism
The key property of evaluation prompts is determinism. There's exactly one correct answer category, and it's obvious.
This allows automated verification:
def verify_gold_prompt(response, expected):
return expected.lower() in response.lower()No human judgment required. No ambiguity. Pass or fail.
Principle 2: Simplicity
Evaluation prompts are trivially easy. This is intentional.
If we used complex prompts, failures could be:
- Legitimate capability limits
- Edge case behavior
- Interpretation differences
With trivial prompts, failures mean one thing: something is broken.
Principle 3: Coverage
Different evaluation prompts test different failure modes:
| Prompt Type | Tests For |
|---|---|
| Math | Basic reasoning works |
| Echo | Instruction following works |
| JSON | Structured output works |
| Knowledge | Model is loaded/responding |
A comprehensive gold prompt suite catches most failure modes.
The Gold Prompt Taxonomy
Health Tier
Purpose: Verify basic functionality.
Characteristics:
- Any reasonable LLM can answer
- Loose matching (answer anywhere in response)
- Tests core reasoning capability
Examples:
Prompt: "What is 2+2?"
Pass: "4", "The answer is 4", "2+2 equals 4, which is..."
Fail: "I cannot answer that", "Error", "5"Prompt: "What color is the sky on a clear day?"
Pass: "blue", "The sky is blue", "Blue, typically"
Fail: "I don't know", "Red", "[Error message]"What failures indicate:
- Model not responding properly
- Authentication issues returning error messages
- Rate limiting returning polite refusals
- Model degradation
Contract Tier
Purpose: Verify format compliance.
Characteristics:
- Tests structured output capability
- Strict matching (exact format required)
- Tests what API consumers actually need
Examples:
Prompt: "Return JSON: {\"status\": \"ok\"}"
Pass: {"status": "ok"}
Fail: "Here's the JSON: {\"status\":\"ok\"}", {"status": "error"}Prompt: "Respond with exactly one word: yes or no"
Pass: "yes", "no"
Fail: "Yes, I can help", "Yes" (with punctuation issues)What failures indicate:
- Structured output broken
- Prompt template corrupted
- Model version changed
- Function calling broken
Behavioral Tier
Purpose: Verify expected agent behavior.
Characteristics:
- Tests agent-specific functionality
- May require configuration
- Tests what your users need
Examples (for a customer support agent):
Prompt: "What is your company's name?"
Pass: "[Your company name]"
Fail: Generic response, "I don't know"Prompt: "I need to return a product"
Pass: Contains return policy or starts return flow
Fail: "I cannot help with returns"What failures indicate:
- RAG system broken
- Tool integration failed
- System prompt not loading
- Configuration drift
Implementation
Basic Implementation
The simplest gold prompt check:
import httpx
GOLD_PROMPTS = [
{"prompt": "What is 2+2?", "expected": "4"},
{"prompt": "Echo: hello", "expected": "hello"},
]
async def check_gold_prompts(endpoint_url, auth_headers):
results = []
for gold in GOLD_PROMPTS:
response = await httpx.post(
endpoint_url,
headers=auth_headers,
json={"messages": [{"role": "user", "content": gold["prompt"]}]}
)
content = response.json()["choices"][0]["message"]["content"]
passed = gold["expected"].lower() in content.lower()
results.append({
"prompt": gold["prompt"],
"passed": passed,
"response": content[:100]
})
return resultsWhat Agent Status Does
We run evaluation prompts on every validation check:
- Send evaluation prompts alongside regular prompts
- Verify responses against expected patterns
- Calculate pass rates (health, contract, overall)
- Factor into verdict (UP/DEGRADED/DOWN)
- Alert on pass rate drops
The user doesn't need to configure anything. We handle it automatically.
Pass Rate Thresholds
Not every gold prompt failure means DOWN.
Networks are flaky. Models have variance. Transient errors happen.
We use thresholds:
| Eval Pass Rate | Verdict Impact |
|---|---|
| ≥95% | No impact (UP eligible) |
| 80-95% | Warning (may contribute to DEGRADED) |
| <80% | DEGRADED |
| <20% | DOWN |
A single gold prompt failure doesn't trigger alerts. Consistent failures do.
Common Failure Patterns
The Authentication Expire
Gold prompt behavior:
- "What is 2+2?" → "Authentication required. Please..."
- Health tier: FAIL
- Contract tier: FAIL
The Rate Limit Wall
Gold prompt behavior:
- "What is 2+2?" → "I'm currently unavailable. Please try..."
- Health tier: FAIL
- Contract tier: N/A
The Model Swap
Gold prompt behavior:
- Return JSON: {x:1} → "Here's the JSON you requested: {x:1}"
- Health tier: PASS (reasoning works)
- Contract tier: FAIL (format broken)
The Context Collapse
Gold prompt behavior:
- "What is 2+2?" → Incoherent response or truncation
- Health tier: FAIL (or partial)
- Contract tier: FAIL
The Silent Degradation
Gold prompt behavior:
- Pass rates decline: 98% → 95% → 90% → 85%
Limitations
Evaluation prompts aren't magic. They have limits:
What They Catch
- Binary functionality (works/doesn't work)
- Format compliance
- Obvious failures
What They Don't Catch
- Subtle quality degradation
- Hallucination on complex queries
- Domain-specific correctness
- Tone/style drift
For these, you need:
- Human evaluation
- Domain-specific tests
- User feedback analysis
- More sophisticated evaluation frameworks
Evaluation prompts are the smoke detector, not the fire investigator.
Best Practices
1. Don't Over-Engineer
Start with 2-3 simple evaluation prompts:
- One math (health)
- One echo (health)
- One JSON (contract)
Add more only if needed.
2. Match Your Agent's Capabilities
If your agent is a code-only assistant that refuses general queries, adjust:
- Don't use "What is 2+2?"
- Use "Write a hello world in Python" instead
3. Version Your Evaluation Prompts
Evaluation prompts are test cases. Version them with your code:
# gold_prompts.yaml
version: 2
prompts:
- type: health
prompt: "What is 2+2?"
expected: "4"
- type: contract
prompt: "Return JSON: {x:1}"
expected_json: true
expected_key: "x"4. Don't Alert on Single Failures
Use pass rates and thresholds, not individual results.
5. Monitor Pass Rate Trends
A drop from 98% to 92% might not trigger alerts but is a warning sign.
