"Our agent is fast."
"How fast?"
"Average latency is 2 seconds."
"TTFB or total?"
"...what's the difference?"
This conversation happens constantly. Let's clear it up.
The Two Metrics
Time to First Byte (TTFB)
How long until the first part of the response arrives.
Request sent → [network] → Server starts → [processing] → First byte arrives
↑
TTFBWhat it measures: When does the user see something?
For streaming responses: When does "typing..." appear?
Total Latency
How long until the complete response arrives.
Request sent → ... → First byte → [streaming content] → Last byte arrives
↑
Total LatencyWhat it measures: When is the interaction complete?
Why TTFB Matters More (Usually)
For user experience, TTFB is often more important.
The Psychology of Waiting
Empty screen: User thinks it's broken. Typing indicator: User knows it's working. Content streaming: User is engaged.
The transition from "empty" to "something" is the critical moment. That's TTFB.
Research shows:
- Users tolerate 5 seconds of streaming content
- Users abandon at 3 seconds of empty screen
TTFB is when the user stops wondering "is this broken?"
Streaming Changes Everything
For streaming agents, total latency is somewhat irrelevant.
Consider:
- TTFB: 800ms, Total: 15s (streaming a long response) - feels fast
- TTFB: 5s, Total: 6s (slow start, short response) - feels broken
The first one, despite 15s total, provides a better experience. The user sees content appearing immediately. They read as it streams. By the time it's done, they've absorbed most of it.
The second one? Five seconds of nothing. User has already switched tabs.
When Total Latency Matters
Total latency matters when:
- Non-streaming (rare for agents)
- Agent-to-agent calls (intermediate step)
- Actions that block on completion
- SLA guarantees that include full response
For user-facing streaming agents, TTFB is king.
Measuring Correctly
Common Mistakes
Mistake 1: Only tracking total latency. If your monitoring only shows "response time: 8s", you don't know if TTFB was 800ms (fine) or 7.5s (terrible). Completely different user experiences.
Mistake 2: Using averages. Latency distributions are not normal. Averages lie.
9 requests: 500ms
1 request: 10,000ms
Average: 1,450ms
Reality: 90% are fast, 10% are terribleUse percentiles: P50, P95, P99.
Mistake 3: Ignoring variance.
Agent A: P50 = 1s, P99 = 1.5s (consistent)
Agent B: P50 = 800ms, P99 = 8s (variable)Agent B's average is better. Agent B's experience is worse. Consistency matters.
What to Track
| Metric | What It Tells You |
|---|---|
| TTFB P50 | Typical user experience |
| TTFB P95 | Experience for most users |
| TTFB P99 | Worst case (tail) |
| Total P50 | Typical completion time |
| Total P95 | Most completions |
| Variance | Consistency |
Minimum: TTFB P50/P95 and Total P50/P95.
Setting SLAs
TTFB SLAs
Recommended thresholds:
| Agent Type | TTFB Target |
|---|---|
| Interactive chat | <2s |
| Complex reasoning | <5s |
| Research/RAG | <3s |
| Code generation | <3s |
If TTFB > 5s for any interactive use case, there's a UX problem.
Total Latency SLAs
Recommended thresholds:
| Response Type | Total Target |
|---|---|
| Short answer | <5s |
| Paragraph | <15s |
| Long document | <60s |
| Complex multi-step | <120s |
These depend heavily on response length and complexity.
Agent Status's Approach
We track both, but weight TTFB more heavily in verdicts.
An agent with TTFB: 500ms, Total: 30s (streaming a detailed response) is healthier than one with TTFB: 8s, Total: 10s (slow start). The first provides good UX. The second doesn't.
Improving TTFB
Quick Wins
1. Enable streaming. If you're buffering the full response, stop.
# Bad: Buffer and return
response = await model.generate(prompt)
return response
# Good: Stream immediately
async for chunk in model.stream(prompt):
yield chunk2. Reduce cold starts. Serverless cold starts destroy TTFB. Solutions:
- Keep warm instances
- Use provisioned capacity
- Optimize package size
- Pre-warm on schedule
3. Optimize initial processing. What happens before the first token? Common culprits:
- RAG retrieval (fetch context before generating)
- Tool selection (deciding what to call)
- Auth validation (checking credentials)
Move what you can to parallel or pre-processing.
Deeper Optimizations
4. Edge caching for context. If RAG retrieval is slow, consider: caching frequent queries, edge-deployed vector stores, pre-computed summaries.
5. Speculative generation. Start generating before full context is ready. Begin with likely response prefix, adjust as context arrives. Works for predictable patterns.
6. Regional deployment. Physics is real. If your server is in US and user is in Asia, minimum TTFB is ~200ms (just network). Realistic TTFB: 500ms+ (plus processing). Deploy closer to users.
The Latency Budget
For multi-step agent systems, decompose latency:
Total TTFB Budget: 2000ms
- Network round trip: 200ms
- Auth/validation: 100ms
- Context retrieval: 300ms
- Model inference start: 400ms
- Buffer before stream: 200ms
- Margin: 800ms
Actual: 1200ms (under budget ✓)When you exceed budget, you know where to optimize.
Monitoring Implementation
What Agent Status Tracks
Every test captures:
dns_ms: DNS resolution timetcp_ms: TCP connection timetls_ms: TLS handshake timettfb_ms: Time to first bytetotal_ms: Total response time
Plus percentiles aggregated across tests:
ttfb_p50_ms,ttfb_p95_ms,ttfb_max_mslatency_p50_ms,latency_p95_ms,latency_p99_ms
Setting Alerts
Alert when:
- TTFB P95 exceeds threshold (poor user experience)
- TTFB P50 increasing over time (degradation)
- Total P99 exceeds reasonable bound (hung requests)
Don't alert on:
- Single slow request (normal variance)
- Total latency for streaming (expected to be longer)
