Catch the regression before you merge.
Every PR runs the same evaluation prompts from where your users actually live, diffs against your baseline, and posts a verdict back to GitHub before merge.
AgentDiff is free. No code changes, no extra runner, install in under two minutes.
The contrast that matters
Your CI runs in us-east-1. Your users live in São Paulo.
AgentDiff is free. No code changes, no extra runner, install in under two minutes.
How it works
Install once, then get a verdict on every pull request.
AgentDiff is a GitHub App, not yet another CI job. No new YAML, no new secrets in your runner. Just an outside-in verdict that updates the PR status.
Install
Add the GitHub App and point it at your agent's preview URL. Auth secrets stay encrypted at rest.
Open a pull request
On every PR sync we run paired validations against base and head, one residential node per region.
Get a verdict
Pass, warn, or fail flips the GitHub Check. The PR comment shows a per-region matrix and one-click rerun.
AgentDiff is free. No code changes, no extra runner, install in under two minutes.
Geographic regression testing, not generic CI evaluation.
We run the same prompt twice, against base and head, from one residential node — so any divergence shows up before you merge.
Geo-emergent regressions
Tool calling that works in us-east and breaks in São Paulo gets caught and labelled as a regression.
Auth surface drift
Bearer, custom-header, and basic-auth are validated end-to-end. The wrong shape returns AUTH_ERROR rather than passing silently.
Latency-paired diffs
p50 and p95 are reported per region. Validations are paired sequentially from the same node, so latency changes attribute to the agent — not the network.
Flake-resistant by design
A single one-retry exponential backoff (300 ms then 900 ms) handles transient 5xx, 429, and timeouts. Per-validate attempts surface in the report.
Coverage
Works with the agent shapes you already ship.
No SDK to install on your agent. AgentDiff validations whatever HTTP surface you already expose.
AgentDiff is free. No code changes, no extra runner, install in under two minutes.
Sits next to your evals in the same PR check pipeline.
Keep EvalView, Evidently, and unit tests. AgentDiff adds the one check they can't run — the same prompt, validated from real residential networks.
Your evals stay green. AgentDiff catches what they can't see.
What AgentDiff is not.
Is it a replacement for EvalView or Evidently?
No — run them together.
They check golden datasets. We check geographic behavior from real residential ISPs. Both signals matter.
Is this a generic CI evaluation tool?
No — it does one thing.
Paired validations against the same agent from the same residential node, then a verdict. Nothing else.
Will it help with a localhost-only project?
Not yet — ship a preview URL first.
AgentDiff validates deployed endpoints from real ISPs. If your agent never leaves your laptop, come back later.
Is it a consumer-facing persona experiment?
No — it is engineering infrastructure.
Built for the engineers shipping agents to production. No roleplay simulations, no synthetic personas — just real validations from the networks your users actually use.
