AgentStatus
We tested every public MCP server we could find. Calling is solved. Delivery is not.
Read the report"Our agent is fast.""How fast?""Average latency is 2 seconds.""TTFB or total?""...what's the difference?"
This conversation happens constantly. Let's clear it up.
How long until the first part of the response arrives.
Request sent → [network] → Server starts → [processing] → First byte arrives
↑
TTFBWhat it measures: When does the user see something?
For streaming responses: When does "typing..." appear?
How long until the complete response arrives.
Request sent → ... → First byte → [streaming content] → Last byte arrives
↑
Total LatencyWhat it measures: When is the interaction complete?
For user experience, TTFB is often more important.
Empty screen: User thinks it's broken. Typing indicator: User knows it's working. Content streaming: User is engaged.
The transition from "empty" to "something" is the critical moment. That's TTFB.
Research shows:
TTFB is when the user stops wondering "is this broken?"
For streaming agents, total latency is somewhat irrelevant.
Consider:
The first one, despite 15s total, provides a better experience. The user sees content appearing immediately. They read as it streams. By the time it's done, they've absorbed most of it.
The second one? Five seconds of nothing. User has already switched tabs.
Total latency matters when:
For user-facing streaming agents, TTFB is king.
Mistake 1: Only tracking total latency. If your monitoring only shows "response time: 8s", you don't know if TTFB was 800ms (fine) or 7.5s (terrible). Completely different user experiences.
Mistake 2: Using averages. Latency distributions are not normal. Averages lie.
9 requests: 500ms
1 request: 10,000ms
Average: 1,450ms
Reality: 90% are fast, 10% are terribleUse percentiles: P50, P95, P99.
Mistake 3: Ignoring variance.
Agent A: P50 = 1s, P99 = 1.5s (consistent)
Agent B: P50 = 800ms, P99 = 8s (variable)Agent B's average is better. Agent B's experience is worse. Consistency matters.
| Metric | What It Tells You |
|---|---|
| TTFB P50 | Typical user experience |
| TTFB P95 | Experience for most users |
| TTFB P99 | Worst case (tail) |
| Total P50 | Typical completion time |
| Total P95 | Most completions |
| Variance | Consistency |
Minimum: TTFB P50/P95 and Total P50/P95.
Recommended thresholds:
| Agent Type | TTFB Target |
|---|---|
| Interactive chat | <2s |
| Complex reasoning | <5s |
| Research/RAG | <3s |
| Code generation | <3s |
If TTFB > 5s for any interactive use case, there's a UX problem.
Recommended thresholds:
| Response Type | Total Target |
|---|---|
| Short answer | <5s |
| Paragraph | <15s |
| Long document | <60s |
| Complex multi-step | <120s |
These depend heavily on response length and complexity.
We track both, but weight TTFB more heavily in verdicts.
An agent with TTFB: 500ms, Total: 30s (streaming a detailed response) is healthier than one with TTFB: 8s, Total: 10s (slow start). The first provides good UX. The second doesn't.
1. Enable streaming. If you're buffering the full response, stop.
# Bad: Buffer and return
response = await model.generate(prompt)
return response
# Good: Stream immediately
async for chunk in model.stream(prompt):
yield chunk2. Reduce cold starts. Serverless cold starts destroy TTFB. Solutions:
3. Optimize initial processing. What happens before the first token? Common culprits:
Move what you can to parallel or pre-processing.
4. Edge caching for context. If RAG retrieval is slow, consider: caching frequent queries, edge-deployed vector stores, pre-computed summaries.
5. Speculative generation. Start generating before full context is ready. Begin with likely response prefix, adjust as context arrives. Works for predictable patterns.
6. Regional deployment. Physics is real. If your server is in US and user is in Asia, minimum TTFB is ~200ms (just network). Realistic TTFB: 500ms+ (plus processing). Deploy closer to users.
For multi-step agent systems, decompose latency:
Total TTFB Budget: 2000ms
- Network round trip: 200ms
- Auth/validation: 100ms
- Context retrieval: 300ms
- Model inference start: 400ms
- Buffer before stream: 200ms
- Margin: 800ms
Actual: 1200ms (under budget ✓)When you exceed budget, you know where to optimize.
Every test captures:
dns_ms: DNS resolution timetcp_ms: TCP connection timetls_ms: TLS handshake timettfb_ms: Time to first bytetotal_ms: Total response timePlus percentiles aggregated across tests:
ttfb_p50_ms, ttfb_p95_ms, ttfb_max_mslatency_p50_ms, latency_p95_ms, latency_p99_msAlert when:
Don't alert on:
Your agent's perceived speed is when the first byte arrives.
For streaming agents: optimize for TTFB, track percentiles, set SLAs on TTFB, and decompose latency. Everything after the first byte is bonus time.