Semantic Kernel plans can fail when a skill is unavailable, when the planner hallucinates a skill that doesn't exist in the kernel, or when a skill's input schema changes underneath a plan that was working yesterday. The planner's output looks like clean code even when it's referencing skills that no longer match. Outside-in validations catch the cases where the plan runs to completion but does the wrong thing.
What actually fails in a Semantic Kernel agent
- Planner generating function calls to skills that were renamed or removed.
- Skill input-schema drift after a semantic function is edited, silently breaking older plans.
- Kernel.InvokeAsync returning a partial result when a downstream skill errors without propagating.
- Planner choosing the wrong skill when two skills have overlapping descriptions.
- Token budget exhausted during plan assembly on large kernels.
What to monitor
There are two layers. The inside layer is what Semantic Kernel itself exposes via callbacks, tracing, or logs - token counts, step timings, tool invocations. Use your existing tracing tool (LangSmith, Helicone, Langfuse, OpenLLMetry) for that. The outside layer is what your users actually experience end-to-end. That's what outside-in monitoring covers.
The five outside-in checks we run on every Semantic Kernel agent:
- HTTP 200 from the agent endpoint.
- JSON parseable response.
- Required fields present (depends on your output schema).
- Content assertion - a specific substring or value the response must contain.
- Latency under your SLO (4–10 seconds end-to-end for planner + invoke).
10-minute setup
- Add a validate endpoint to your Semantic Kernel server that runs the same pipeline a user would.
- Define the 5 assertions above in Agent Status, pointed at that endpoint.
- Configure checks from 3 regions that cover your user base.
- Pick the tightest cadence your plan supports (paid plans on Agent Status run as short as 5 minutes) and route alerts to Slack or your paging channel.
- Ship a deliberate regression to test that an alert fires - then revert.
Validation endpoint example
Validation the planner-plus-invoke path, not a single skill. The whole point of Semantic Kernel is that the planner picks the skill for you - validation the combined output so you catch planner drift, not just skill availability.
// validation.ts - expose /validation on your Semantic Kernel server (TypeScript)
import express from "express";
import { Kernel } from "@microsoft/semantic-kernel";
import { buildKernel } from "./your-app";
const app = express();
const kernel: Kernel = buildKernel();
app.get("/validation", async (_req, res) => {
const plan = await kernel.createPlan("Find the current refund policy and summarise it");
const result = await kernel.invokeAsync(plan);
const text = String(result ?? "");
res.json({
ok: text.toLowerCase().includes("refund"),
answer: text.slice(0, 500),
});
});How this complements inside-out tracing
Outside-in validations tell you that something broke and whether users were affected. Inside-out tracing tells you why it broke - which step in the chain, which tool call, which retrieval, which model response. Run both. When a validate pages us, the first place we look is our inside-out traces. See our write-up on outside-in monitoring for the conceptual framing.
Related
- /monitor/autogen - Multi-agent conversations.
- /monitor/langchain - Python chain-based alternative.
- /blog/what-is-outside-in-monitoring - Why outside-in matters.
- /platform - The product overview.
Frequently asked questions
Does this replace Semantic Kernel's telemetry?
No. SK telemetry traces planner and skill internals (inside-out); Agent Status validates the user-facing result (outside-in). They're complementary.
What Semantic Kernel failures does it catch?
Missing or unregistered skills, planner hallucinations, input-schema drift, and plans that return 200 without producing the required output.
Do I need code changes?
No - point Agent Status at your endpoint, define assertions, and go live in about 10 minutes.
How often can it run?
The free tier is 30 tests/month from three regions; paid plans run as often as every 5 minutes from 70 countries.
Ready to try it?
Agent Status runs this setup out of the box. The free tier includes 30 tests/month on 1 agent from 3 regions with a 10-minute setup and no code changes to the agent itself. See pricing or jump straight to the platform overview.
