Make your AI agent prove it: a deterministic delivery gate

The agent said it was done. The gate disagreed.

A real run from this week: the builder reported done, the critic scored 55 and returned a gap list with names and weights. Two checks were failing, one of them was a secret scan that found a token in the diff. No opinion was involved — the report attached the raw output of every check. This is what a deterministic delivery gate looks like when you stop asking the model whether the work is good and start measuring.

This skill is now published as a real package. pip install agent-test-loop and npx ai-test-loop both resolve, the repository is open (MIT), and the free version is a real download — not a teaser.

Why the same context lies

The classic failure: one context writes the code, runs the tests, reads the result and decides whether it is done. The model that produced the bug is the model that judges the fix. The gate breaks that loop by moving the judgement into a separate process that executes real commands and returns numbers, not prose.

• BuilderWrites the change. No veto power over the verdict.
• CriticRuns every declared check against the real artifact. Scores it. Never asks a model if it is good.
• ReviewerIndependent context, fresh, no justification from the builder. Returns strict JSON.
• Security gateFail-closed. A secret, a high-severity finding or an injection issue blocks delivery even with a perfect score. Inconclusive counts as blocked.

Rule zero: builder ≠ critic ≠ reviewer. If one context does all three, the loop is theatre.

The gate, step by step

1. Declare the checks before writing codeFor each check: name, command, weight and what “pass” means. Weights sum to 100. {"checks":[{"name":"pytest","cmd":"set -o pipefail; pytest -q","weight":40},{"name":"no_secret_scan","cmd":"gitleaks detect --no-banner","weight":30}]}
2. Run the criticIt executes every command for real, captures the evidence and never asks a model whether the work is good — it measures. python scripts/loop_runner.py --config loop.json --round 1
3. Score, compare, decideAbove the threshold goes to the reviewer; below, take the largest gap, fix it and run again. Cap the rounds (8 is plenty): if it never converges, the check set is wrong, not the code.
4. Independent review before deliverySame artifact, fresh context, no justification from the builder. A panel of 2-3 specialists (quality and security always) returning strict JSON.
5. Security gate, fail-closedA secret in the diff or history, a high-severity finding, an auth/path-traversal/injection issue block delivery even with perfect tests. An inconclusive answer counts as blocked, never as pass.
6. Report with score, evidence and what was not verifiableThe report states what was measured, with raw output, and what could not be checked — no “probably fine”.

Four ways a green loop lies

• Piping hides the exit codepytest | tail returns the exit code of tail (0), so the critic scores 100 with a failing test hidden in the output. Use set -o pipefail inside every check.
• A timeout shorter than the check is a silent zeroThe critic is killed by its own timeout, the orchestrator reads score 0 with an empty gap list, and loops with nothing to fix. Set the timeout above the real duration and always keep the critic's raw log.
• A short-circuiting “pass”A server answering 200 may be the wrong server: on a port collision the check tests a different app and fails for the wrong reason. Assert identity (title, version, a known payload), not just reachability.
• A stale baseline regenerated to go greenRegenerating a visual baseline “to pass” erases the regression the test exists to catch. Baselines change by explicit decision, justified in the history — never to silence a check.

Install it now

The package ships loop_runner.py, the scoring rubric, the reviewer prompt and the security checks. Two commands, two ecosystems, same gate.

pip install agent-test-loop
# or
npx ai-test-loop

See the AI Test Loop skill Copy Install Command

Rather have it installed in your project? Guided setup from R$ 1.500, with the first cycle run together.

Repository (MIT): github.com/Samuelfmedeiros/ai-test-loop