Two reasons, opposite responses

• The page changedBlock the release. That is what the test is for.
• The page did not change, the capture is not reproducibleAnimation mid-frame, blinking caret, timestamp, ad slot, font fallback, scrollbar. Fix the capture, not the UI.

A gate that cannot separate them becomes noise and someone turns it off. No AI in the loop: same input, same verdict.

The four verdicts (and what to do with each)

• PASSWithin the pixel budget and the region limit. Ship.
• FAILBudget exceeded, with regions listed as x/y/w/h. Look at diff.png and the region crops first.
• STALEThe baseline does not belong to this build (git HEAD, mask config or the PNG itself changed). Re-baseline deliberately or unstage the change — never “just re-run”.
• ERRORUnreadable PNG or size mismatch. That is a capture bug (viewport changed?), not a regression.

STALE exists because the classic failure of visual testing is not flakiness — it is a baseline quietly re-generated until the test went green. Tying the baseline to git HEAD plus a hash of the mask config makes re-baselining a deliberate, reviewable act.

Exit codes are the contract

0 PASS   1 FAIL   2 STALE   3 ERROR

python scripts/visual_gate.py fingerprint baseline/home.png \
    --url https://app.example/home --viewport 1280x720 \
    --config baseline/home.masks.json

python scripts/visual_gate.py compare baseline/home.png actual/home.png \
    --meta baseline/home.png.meta.json --masks baseline/home.masks.json \
    --max-diff 0.001 --max-regions 1 --bundle artifacts/home

With --json you get the verdict, the diff percentage, the region count and every region's bounding box — enough for CI to publish what moved instead of “the test failed”.

Budgets that survive contact with reality

• --max-diff is a safety net, not the primary signalAntialiasing and font hinting shift a few thousand pixels with zero real change; a real component move shifts tens of thousands. Start at 0.001 (0.1%) and tighten once masks are in place.
• --max-regions is the signal that catches small, meaningful changesA wrong colour on a 60×40 button is ~0.26% of a 1280×720 page, but exactly one region. Budget 1.
• --tolerancePer-channel delta (default 0) for capture noise on gradients. If you need more than ~8, the capture is the problem.
• MasksRemove whole rectangles from the comparison and are reported (masked_pixels) — nobody hides a regression inside one without it showing in the report.

Determinism checklist (before blaming the gate)

1. Fixed viewport and DPR 1DPR scales the layout and invalidates every baseline.
2. Disable animationprefers-reduced-motion: reduce and *{animation:none!important;transition:none!important}.
3. Wait for fonts and networkdocument.fonts.ready and network idle — a fallback font diffs the whole page.
4. Hide the caret, the scrollbar and the dev overlayNone of it is page content, and all of it moves pixels.
5. Freeze time and randomnessIf the UI renders date/time or Math.random, freeze it before capturing.
6. Mask only what is genuinely dynamicAd slot, avatar of the day, live counter, A/B slot.
7. Capture in the same container as CIA different font stack means different pixels everywhere.

Evidence: the gate is tested in both directions

15 tests passing — pytest -q  (28.9s)
  identical screenshots            -> PASS, diff 0.00000%, 0 regions
  one 60x40 block changed          -> FAIL, 1 region at the exact painted coordinates
  same change inside a mask        -> PASS, masked_pixels >= the masked area
  change outside the mask          -> FAIL (a mask must not hide the rest of the page)
  two changes with --max-regions 1 -> FAIL; with --max-regions 2 -> PASS
  stale baseline (git HEAD differs)-> STALE, exit 2
  mask config changed              -> STALE
  640x480 vs 1280x720              -> ERROR, "size mismatch" (never a silent PASS)
  decoder vs PIL + ImageMagick     -> identical pixels across 6 encodings
  16-bit capture                   -> high bytes equal the 8-bit rendering
  failure bundle                   -> diff.png re-decodes, red area >= the changed region

The two-way test is the point: a gate tested only against “no change” is decoration. Both directions are asserted — including the case that bites in production: an optimised baseline (optipng, pngquant, ImageMagick) that a naive decoder refuses.

And the PNG decoder is stdlib (zlib + struct) — no PIL, OpenCV or ImageMagick on the runner: a CI gate that needs a C library on every machine breaks on the machine that matters.

The pitfalls that cost the most

• Baseline committed without its .meta.jsonWithout it the gate cannot detect staleness. Commit both.
• Re-baselining to make CI greenExactly what the STALE verdict exists to prevent: the diff is in the bundle, review it before writing a new baseline.
• Masking more than ~10% of the pageYou stopped testing that page. Prefer freezing the widget over masking it.
• Comparing different viewports or DPRsYields ERROR by design: it is a capture bug, not a regression.
• A PASS on a page you never capturedsweep reports MISSING and fails the run — an empty actual directory is not “all good”.
• Running the gate against a dirty treeFor release decisions, fingerprint the baseline at a commit and compare only when CI is on that same ref.

Start now

The Visual Regression Gates skill ships the stdlib gate, 15 tests, the Playwright capture template, the mask format and both guides (capture determinism, mask/fingerprint strategy).

See the Visual Regression Gates skill See all skills

Want it wired into your pipeline? Guided setup from R$ 1.500, with the first cycle run together.