AgentStatus
We tested every public MCP server we could find. Calling is solved. Delivery is not.
Read the report"We run tests before deploy. Isn't that enough?"
No. And here's why.
Tests verify that code works in a test environment. Production validation verifies that your agent works in production, right now, from real locations.
The best teams do both. Here's how.
Tests answer: "Does the code work?" Production validation answers: "Does the deployed agent work?"
Different questions. Both matter.
When: After deploying, before considering deploy "complete"
What: Trigger Agent Status validation, fail deploy if agent is unhealthy
# GitHub Actions
deploy:
runs-on: ubuntu-latest
steps:
- name: Deploy
run: ./deploy.sh
- name: Wait for deployment
run: sleep 30
- name: Validate Agent
env:
AGENTSTATUS_API_KEY: ${{ secrets.AGENTSTATUS_API_KEY }}
AGENT_ID: ${{ secrets.AGENTSTATUS_AGENT_ID }}
run: |
# Trigger validation
RESPONSE=$(curl -s -X POST \
"https://api.agentstatus.dev/agents/$AGENT_ID/run" \
-H "Authorization: Bearer $AGENTSTATUS_API_KEY")
DECISION_ID=$(echo $RESPONSE | jq -r '.decision_id')
# Poll for results
for i in {1..30}; do
sleep 5
RESULT=$(curl -s \
"https://api.fabric.carmel.so/api/agentstatus/runs/$DECISION_ID" \
-H "Authorization: Bearer $AGENTSTATUS_API_KEY")
STATUS=$(echo $RESULT | jq -r '.status // "pending"')
if [ "$STATUS" = "completed" ]; then
VERDICT=$(echo $RESULT | jq -r '.overall_verdict')
if [ "$VERDICT" = "DOWN" ]; then
echo "❌ Agent validation FAILED"
exit 1
fi
echo "✅ Agent validation PASSED: $VERDICT"
exit 0
fi
done
echo "⏱️ Validation timed out"
exit 1
- name: Rollback on failure
if: failure()
run: ./rollback.shBenefits: Catch deploy issues immediately. Auto-rollback on failure. Confidence before marking deploy complete.
When: Before promoting from staging to production
What: Validate staging agent meets quality bar
promote-to-production:
runs-on: ubuntu-latest
steps:
- name: Validate Staging
env:
AGENTSTATUS_API_KEY: ${{ secrets.AGENTSTATUS_API_KEY }}
STAGING_AGENT_ID: ${{ secrets.STAGING_AGENT_ID }}
run: |
./scripts/validate-agent.sh $STAGING_AGENT_ID
- name: Promote to Production
if: success()
run: |
./promote-to-production.sh
- name: Validate Production
env:
AGENTSTATUS_API_KEY: ${{ secrets.AGENTSTATUS_API_KEY }}
PROD_AGENT_ID: ${{ secrets.PROD_AGENT_ID }}
run: |
./scripts/validate-agent.sh $PROD_AGENT_IDBenefits: Catch issues before they reach production. Gate promotion on quality. Validate both environments.
When: During gradual rollouts
What: Validate canary before expanding
canary-deploy:
runs-on: ubuntu-latest
steps:
- name: Deploy to Canary (5%)
run: ./deploy-canary.sh
- name: Wait for traffic
run: sleep 300 # 5 minutes of canary traffic
- name: Validate Canary
run: |
RESULT=$(./scripts/validate-agent.sh $CANARY_AGENT_ID)
LATENCY=$(echo $RESULT | jq '.latency_p50_ms')
GOLD_RATE=$(echo $RESULT | jq '.gold_pass_rate')
PROD_LATENCY=$(./scripts/get-prod-baseline.sh latency)
if [ $LATENCY -gt $(($PROD_LATENCY * 150 / 100)) ]; then
echo "Canary latency too high: $LATENCY vs $PROD_LATENCY"
exit 1
fi
if [ $GOLD_RATE -lt 90 ]; then
echo "Canary gold rate too low: $GOLD_RATE"
exit 1
fi
echo "Canary healthy"
- name: Expand Rollout
if: success()
run: ./expand-rollout.sh
- name: Rollback Canary
if: failure()
run: ./rollback-canary.shBenefits: Quantified canary health. Automatic comparison to baseline. Expansion gated on quality.
When: On a schedule (nightly, weekly)
What: Comprehensive validation without blocking deploys
# .github/workflows/nightly-validation.yml
name: Nightly Agent Validation
on:
schedule:
- cron: '0 2 * * *' # 2 AM daily
jobs:
validate:
runs-on: ubuntu-latest
steps:
- name: Full Validation
run: |
for AGENT_ID in $PRODUCTION_AGENTS; do
./scripts/full-validation.sh $AGENT_ID
done
- name: Generate Report
run: ./scripts/generate-report.sh
- name: Send Report
run: ./scripts/send-report.shBenefits: Comprehensive coverage. Historical trend data. Doesn't block deploys.
A reusable script for all patterns:
#!/bin/bash
# scripts/validate-agent.sh
AGENT_ID=$1
TIMEOUT=${2:-120} # seconds
INTERVAL=${3:-5} # seconds
# Trigger validation
echo "Triggering validation for $AGENT_ID..."
RESPONSE=$(curl -s -X POST \
"https://api.fabric.carmel.so/api/agentstatus/agents/$AGENT_ID/run" \
-H "Authorization: Bearer $AGENTSTATUS_API_KEY")
DECISION_ID=$(echo $RESPONSE | jq -r '.decision_id')
if [ "$DECISION_ID" = "null" ] || [ -z "$DECISION_ID" ]; then
echo "Failed to trigger validation"
exit 1
fi
echo "Decision ID: $DECISION_ID"
# Poll for results
ELAPSED=0
while [ $ELAPSED -lt $TIMEOUT ]; do
sleep $INTERVAL
ELAPSED=$((ELAPSED + INTERVAL))
RESULT=$(curl -s \
"https://api.fabric.carmel.so/api/agentstatus/runs/$DECISION_ID" \
-H "Authorization: Bearer $AGENTSTATUS_API_KEY")
STATUS=$(echo $RESULT | jq -r '.status // "pending"')
if [ "$STATUS" = "completed" ]; then
VERDICT=$(echo $RESULT | jq -r '.overall_verdict')
LATENCY=$(echo $RESULT | jq '.summary.latency_p50_ms')
GOLD=$(echo $RESULT | jq '.summary.gold_pass_rate')
echo "Verdict: $VERDICT"
echo "Latency P50: ${LATENCY}ms"
echo "Eval Pass Rate: ${GOLD}%"
if [ "$VERDICT" = "DOWN" ]; then
echo "❌ FAILED"
exit 1
elif [ "$VERDICT" = "DEGRADED" ]; then
echo "⚠️ DEGRADED"
exit 0 # or exit 1 if you want to fail on degraded
else
echo "✅ PASSED"
exit 0
fi
fi
echo "Status: $STATUS (${ELAPSED}s elapsed)"
done
echo "⏱️ Timeout after ${TIMEOUT}s"
exit 1Define quality thresholds and enforce them:
quality-gates:
min_gold_pass_rate: 95
max_latency_p50_ms: 2000
max_latency_p95_ms: 5000
required_regions:
- us
- euCompare to historical baseline:
LATENCY_THRESHOLD = 1.5 # 50% regression
GOLD_THRESHOLD = 0.95 # 5% drop
if current['latency_p50_ms'] > baseline['latency_p50_ms'] * LATENCY_THRESHOLD:
print("Latency regression detected")
sys.exit(1)
if current['gold_pass_rate'] < baseline['gold_pass_rate'] * GOLD_THRESHOLD:
print("Quality regression detected")
sys.exit(1)Validate multiple agents together using matrix strategies:
strategy:
matrix:
agent:
- name: support-bot
id: agent_001
- name: sales-bot
id: agent_002
- name: docs-bot
id: agent_003