27–78%
of benchmark-reported agent “successes” are corrupt successes
The right end-state reached through a broken, unsafe, or cheating path. Outcome-only scoring cannot see the difference.
Agent harness · self-hosted · alpha
Belay
The agent harness: sandbox any agent, verify each step by replaying it against real state, and keep a deterministic trace.
Belay sits as a transparent proxy between an AI agent and the tools it calls — recording what crossed, running the tools inside a sandbox, and replaying every tool call against its restored pre-state to render a verdict grounded in re-execution and a state diff.
Install from PyPI
uv tool install belay-harnessOr run the whole engine as a container:
docker run --rm ghcr.io/haqaliz/belay --help01 — The problem
A run can look done and be wrong.
Two measured numbers explain why Belay exists: the work that reports success is not reliably successful, and the usual judge of it is not reliably right.
up to 35%
LLM-judge false positives
Verdicts flip 10–30% on trivial reorderings. A guess about correctness is not a verification of it.
Belay is the third thing: not a framework that builds the agent, not observability that records what it did — the harness that answers “was this step actually correct?” by replaying the tool call in a sandbox and diffing observed against claimed state.
02 — How it works
Record. Contain. Replay. Compound.
Four moves, in order. Each one exists because the previous one cannot answer the question alone.
-
Transparent proxy
Belay sits in front of the MCP server you already run. Bytes are forwarded verbatim in both directions, and every frame is recorded to an append-only trace (
.jsonl). -
Sandbox + snapshot
The server runs under the platform sandbox — macOS Seatbelt, or Linux Landlock + seccomp. A write outside the scope is refused by the kernel and recorded as a denial; the network is denied by default. Each turn's pre-state is snapshotted before the call reaches the server.
-
Per-turn replay
For each recorded
tools/call, Belay restores its pre-state, re-invokes the server, and diffs the observed effect against the recorded reply and the tool's declared contract. No model is consulted. -
Deterministic trace & compounding corpus
Every run is captured exactly and can be re-run. Every caught failure becomes a labeled, replayable case;
belay corpus runre-replays the whole corpus as a regression suite.
03 — Demo
The demo is the negative control.
A committed capture of a real agent run (claude -p, told only “make the tests pass”). A real agent fixed the bug honestly and said so; Belay agrees, and shows exactly what that agreement covers.
The corrupt-success shape is measured elsewhere and at scale — 11/60 = 18.3% in the Phase-0 mint (with its decomposition).
Replay it yourself: demo/README.md.
04 — The verdict
Four statuses. One of them is not a pass.
- PASSThe replay reproduced the recorded result and the observed effect conformed to the declared contract and the task policy — on the dimensions Belay checks.
- WARNOne of the four statuses a turn can reduce to. The reduction is worst-status-wins across the A1 and A2 axes.
- FAILRe-execution or the invariant check refuted the step — including a cheating agent whose trace is perfectly faithful.
- UNVERIFIEDBelay tried and could not verify, and says so by name — a named cause, never a guess in the passing direction.
UNVERIFIED is never rendered as PASS. A turn Belay could not verify is an abstention with a named cause; absence of evidence is never evidence of absence.
05 — Status & limits
Alpha, and honest about it.
Belay's entire value is an honest verdict, so its limits are documented as precisely as its claims. Read the coverage and limits before trusting any output.
- Status
- Alpha. The record → sandbox → replay → verdict spine plus the failure corpus (C1–C6) is built and merged.
- Platforms
- macOS and Linux, both measured. macOS Seatbelt + APFS clonefile; Linux Landlock + seccomp. Python 3.10+.
- Dependencies
- Zero runtime dependencies — stdlib only.
- Where it runs
- Self-hosted. Traces and state stay on your box; nothing is uploaded, ever. No analytics, no cookies, no third-party requests on this page either.
Known and deliberate limits: Belay sees what crosses the MCP boundary and nothing else; it has no network instrument, so a PASS excludes the network dimension; the default invariant's precision has still not been measured; and replay re-invokes the servers you name, never one inferred from the trace.