Haystack pipelines fail where their components fail. An empty document store after a re-index. A misconfigured retriever that silently returns zero documents. A reader model that throws on long documents but keeps the pipeline's overall result looking 'ok'. Haystack's pipeline abstraction is powerful - and opaque. Outside-in validations let you assert on the end-to-end output shape without instrumenting every component.
What actually fails in a Haystack agent
- Document store indexes that silently shrink after a partial re-index.
- BM25 or dense retrievers returning zero documents on valid queries after config drift.
- Reader models throwing on long context and the pipeline returning an empty answer.
- PromptNode output that no longer matches the downstream OutputParser's expected schema.
- Component version skew after a pip upgrade that changes the pipeline graph's wiring.
What to monitor
There are two layers. The inside layer is what Haystack itself exposes via callbacks, tracing, or logs - token counts, step timings, tool invocations. Use your existing tracing tool (LangSmith, Helicone, Langfuse, OpenLLMetry) for that. The outside layer is what your users actually experience end-to-end. That's what outside-in monitoring covers.
The five outside-in checks we run on every Haystack agent:
- HTTP 200 from the agent endpoint.
- JSON parseable response.
- Required fields present (depends on your output schema).
- Content assertion - a specific substring or value the response must contain.
- Latency under your SLO (2–6 seconds end-to-end for RAG pipelines).
10-minute setup
- Add a validate endpoint to your Haystack server that runs the same pipeline a user would.
- Define the 5 assertions above in Agent Status, pointed at that endpoint.
- Configure checks from 3 regions that cover your user base.
- Pick the tightest cadence your plan supports (paid plans on Agent Status run as short as 5 minutes) and route alerts to Slack or your paging channel.
- Ship a deliberate regression to test that an alert fires - then revert.
Validation endpoint example
Validation the Pipeline.run entry point. Haystack pipelines are DAGs; you want to assert on the final node's output, not an intermediate one. An intermediate-node validation won't catch a broken synthesizer.
# validation.py - expose /validation on your Haystack server
from fastapi import FastAPI
from haystack import Pipeline
from your_app import build_rag_pipeline
app = FastAPI()
pipe: Pipeline = build_rag_pipeline()
@app.get("/validation")
def validation():
out = pipe.run({"query": "What is our refund policy?"})
answer = out["generator"]["replies"][0] if out.get("generator", {}).get("replies") else ""
docs = out.get("retriever", {}).get("documents", [])
return {
"ok": bool(docs) and "refund" in answer.lower(),
"answer": answer,
"retrieved": len(docs),
}How this complements inside-out tracing
Outside-in validations tell you that something broke and whether users were affected. Inside-out tracing tells you why it broke - which step in the chain, which tool call, which retrieval, which model response. Run both. When a validate pages us, the first place we look is our inside-out traces. See our write-up on outside-in monitoring for the conceptual framing.
Related
- /monitor/llamaindex - Another RAG-first framework.
- /monitor/langchain - Chain-based alternative.
- /blog/ttfb-vs-latency - Why TTFB is the right SLO metric.
- /platform - The product overview.
Frequently asked questions
Does Agent Status replace Haystack's tracing?
No. Haystack tracing is inside-out (pipeline internals); Agent Status is outside-in (the answer users get). Most teams run both.
What Haystack failures does it catch?
Retrieval returning empty context, generation nodes that respond 200 with malformed output, and pipeline latency that exceeds your SLO under regional load.
Do I need to instrument the pipeline?
No. Validate the user-facing endpoint with content assertions; 10-minute setup, no code changes.
Where can it test from?
70 countries with residential exits, so regional retrieval and latency problems surface before users report them.
Ready to try it?
Agent Status runs this setup out of the box. The free tier includes 30 tests/month on 1 agent from 3 regions with a 10-minute setup and no code changes to the agent itself. See pricing or jump straight to the platform overview.
