LlamaIndex's retrieval layer is where most production failures live - empty retrievals on valid queries, stale embeddings after a re-index, and latency spikes on the vector DB that don't show up in your app's process metrics. LlamaIndex exposes a lot of inside-out hooks, but they can't tell you when a user in Mumbai sees a 12-second query while the query engine reports 400 ms from your server. This page covers the outside-in layer that catches those cases.
What actually fails in a LlamaIndex agent
- Vector DB latency spikes (Pinecone, Weaviate, Qdrant, or pgvector) that your app-level tracing undersamples.
- Empty retrievals on valid queries after an embedding-model swap or stale index.
- Query engine composition errors where the sub-question generator hallucinates an impossible sub-query.
- Response synthesizer truncation when source nodes exceed the context window.
- Regional latency variance when the vector DB is single-region but your users aren't.
What to monitor
There are two layers. The inside layer is what LlamaIndex itself exposes via callbacks, tracing, or logs - token counts, step timings, tool invocations. Use your existing tracing tool (LangSmith, Helicone, Langfuse, OpenLLMetry) for that. The outside layer is what your users actually experience end-to-end. That's what outside-in monitoring covers.
The five outside-in checks we run on every LlamaIndex agent:
- HTTP 200 from the agent endpoint.
- JSON parseable response.
- Required fields present (depends on your output schema).
- Content assertion - a specific substring or value the response must contain.
- Latency under your SLO (2–6 seconds end-to-end, depending on vector DB).
10-minute setup
- Add a validate endpoint to your LlamaIndex server that runs the same pipeline a user would.
- Define the 5 assertions above in Agent Status, pointed at that endpoint.
- Configure checks from 3 regions that cover your user base.
- Pick the tightest cadence your plan supports (paid plans on Agent Status run as short as 5 minutes) and route alerts to Slack or your paging channel.
- Ship a deliberate regression to test that an alert fires - then revert.
Validation endpoint example
A useful LlamaIndex validation hits the full query engine - retrieval + synthesis - not a raw vector search. The whole point is to catch cases where retrieval returns something but the synthesizer produces nonsense from it.
# validation.py - expose /validation on your LlamaIndex server
from fastapi import FastAPI
from llama_index.core.query_engine import RetrieverQueryEngine
from your_app import build_query_engine
app = FastAPI()
engine: RetrieverQueryEngine = build_query_engine()
@app.get("/validation")
def validation():
response = engine.query("What is our refund policy?")
nodes = response.source_nodes
return {
"ok": len(nodes) > 0 and "refund" in str(response).lower(),
"answer": str(response),
"retrieved": len(nodes),
"top_score": nodes[0].score if nodes else 0.0,
}How this complements inside-out tracing
Outside-in validations tell you that something broke and whether users were affected. Inside-out tracing tells you why it broke - which step in the chain, which tool call, which retrieval, which model response. Run both. When a validate pages us, the first place we look is our inside-out traces. See our write-up on outside-in monitoring for the conceptual framing.
Related
- /monitor/langchain - Same pattern for LangChain chains.
- /monitor/haystack - RAG pipelines on Haystack.
- /blog/what-is-outside-in-monitoring - Why the outside-in layer matters.
- /platform - The product overview.
Frequently asked questions
Does this replace LlamaIndex's observability integrations?
No. Those trace indexing and query internals (inside-out). Agent Status validates the answer your users get (outside-in), so the two are complementary.
What LlamaIndex failures does it catch?
Empty or stale retrieval, query-engine composition errors, sub-question generators that hallucinate impossible sub-queries, and responses that return 200 while missing required content.
Do I need to instrument my code?
No. Point Agent Status at your query endpoint, define content assertions, and you're live in about 10 minutes - no code changes.
Why does region matter for a RAG app?
Retrieval latency and availability vary by geography. Agent Status validates from 70 countries using residential exits, surfacing regional issues central monitoring misses.
Ready to try it?
Agent Status runs this setup out of the box. The free tier includes 30 tests/month on 1 agent from 3 regions with a 10-minute setup and no code changes to the agent itself. See pricing or jump straight to the platform overview.
