The problem: the builder grades its own homework

An agent reviewing its own output reviews nothing: it confirms the intent it already had. The symptoms are familiar — review rounds that never converge, “it's done” repeated, and the defect showing up later in the hands of whoever trusted it.

The fix is not asking the model to “be rigorous”. It is moving the score out of the agent: a script that measures with real commands, and a separate context that audits the score.

Four roles, never merged into one context

• Orchestratorthe agent running the task: picks the checks, drives the rounds, fixes infrastructure and owns the report.
• Builderwhoever wrote the artifact: fixes one class of gap per pass.
• Critica script you write (loop_runner.py + your checks): executes every command, keeps the raw output and prints SCORE= and GAPS=.
• Reviewera different agent/model in a fresh context, with no justification from the builder: it can reopen the loop.

Rule zero: builder ≠ critic ≠ reviewer. If one context does all three, the loop is theatre.

The loop, step by step

1. Declare the checks before writing codeFor each check: name, command, weight and what “pass” means. Weights sum to 100. {"checks":[{"name":"pytest","cmd":"set -o pipefail; pytest -q","weight":40}, {"name":"no_secret_scan","cmd":"gitleaks detect --no-banner","weight":30}]}
2. Run the criticIt executes every command for real, captures the evidence and never asks a model whether the work is good — it measures. python scripts/loop_runner.py --config loop.json --round 1
3. Score, compare, decideAbove the threshold goes to the reviewer; below, take the largest gap, fix it and run again. Cap the rounds (8 is plenty): if it never converges, the check set is wrong, not the code.
4. Independent review before deliverySame artifact, fresh context, no justification from the builder. A panel of 2-3 specialists (quality and security always) returning strict JSON.
5. Security gate, fail-closedA secret in the diff or history, a high-severity finding, an auth/path-traversal/injection issue block delivery even with perfect tests. An inconclusive answer counts as blocked, never as pass.
6. Report with score, evidence and what was not verifiableThe report states what was measured, with raw output, and what could not be checked — no “probably fine”.

What the output actually looks like

Red is the broken case (the gate blocks); green is the one that ships. Both are real runs, not an illustration.

$ python scripts/loop_runner.py --config loop.json --round 3
SCORE=55
GAPS=[pytest::exit_1, no_secret_scan::missing]
GATE=failed
$ echo $?
1

$ python scripts/loop_runner.py --config loop.json --round 7
SCORE=100
GATE=passed
$ echo $?
0

Four ways a green loop lies

• Piping hides the exit codepytest | tail returns the exit code of tail (0), so the critic scores 100 with a failing test hidden in the output. Use set -o pipefail inside every check.
• A timeout shorter than the check is a silent zeroThe critic is killed by its own timeout, the orchestrator reads score 0 with an empty gap list, and loops with nothing to fix. Set the timeout above the real duration and always keep the critic's raw log.
• A short-circuiting “pass”A server answering 200 may be the wrong server: on a port collision the check tests a different app and fails for the wrong reason. Assert identity (title, version, a known payload), not just reachability.
• A stale baseline regenerated to go greenRegenerating a visual baseline “to pass” erases the regression the test exists to catch. Baselines change by explicit decision, justified in the history — never to silence a check.

What changes day to day

The point is not the score: it is the conversation. “Is it done?” stops being the builder's opinion and becomes SCORE=100 with the command output attached — or a gap list with names and weights. When something could not be verified, it is written down instead of turning into a false green.

The same pattern holds for any deliverable: code, a pipeline, a data migration, a document someone will use to decide.

Start now

The skill that does this ships as AI Test Loop — the package includes loop_runner.py, the scoring rubric, the reviewer prompt and the security checks. The repository is open (MIT) and the store version adds the ready-made guards.

See the AI Test Loop skill Get the free version

Rather have it installed in your project? Guided setup from R$ 1.500, with the first cycle run together.