Roger MLOps Guard

Long-training guard: GPU, checkpoint and disk with nobody watching.

How to start

1. Declare the curves onceSchedule, expected loss band, tokens/s floor, VRAM ceiling and the frozen-log limit — in the config, so the critic has something to disagree with reality about.
2. Bring the critic up as a daemonEvery verdict is filed in the ledger with the observation that produced it. Output that is only JSON is never accepted as a score.python -m roger.critic --daemon # --once for a single verdict
3. Send every checkpoint through safe_io.publish()On 9p/drvfs a cp under memory pressure truncates SILENTLY — and that is what corrupts resumes. publish writes to staging, fsyncs, checks the hash, moves and prunes the old ones.
4. Turn on the night watchMeasures liveness by sentinel (log and process), kills the stuck training and relaunches from the last GOOD checkpoint, under a relaunch budget and a crash-loop breaker.python -m roger.night_watch

When you will use it

• Training alive with the log frozen, at 3amThe sentinel measures liveness and decides wedge, grind, zombie or real work — without that, every watchdog mistakes a live process for a running training.
• A resume stuck on a truncated checkpointsafe_io.publish() writes to staging, fsyncs, checks the hash and only then moves: the half-written state that corrupts resumes on 9p/drvfs stops existing.
• A watchdog that kills nothing (or kills the wrong thing)It only kills after the verdict, with the reason logged, relaunching from the last good checkpoint, under a relaunch budget and a crash-loop breaker.
• The disk filling up overnightCheckpoint hygiene and pruning, with the disk measured before it becomes an emergency — including the host disk, which is usually the one that fills first.
• A nightly loop inheriting yesterday's good scoreA deterministic critic scores the curves you declared (LR x schedule, grad norm, loss velocity, tokens/s, VRAM, staleness) and files every verdict in an append-only ledger.

What comes in the package

• scripts/roger/critic.py — the deterministic critic and the verdict ledger
• scripts/roger/night_watch.py — sentinel liveness: wedge vs grind vs zombie vs real work
• scripts/roger/gpu_cure.py — cure and relaunch of a stuck training, with the reason logged
• scripts/roger/safe_io.py — crash-proof I/O: staging, fsync, hash, move, prune
• scripts/roger/hygiene.py — loop hygiene: ANSI, untrusted-output quarantine, regression and flap
• scripts/quarantine.py — proof by retry on the input that was quarantined
• scripts/cuda_tflops_bench.py — GPU throughput arbiter
• scripts/host_vram_probe.ps1.txt — VRAM view from the host side — rename it to .ps1

Requirements

• Python 3.10+ (pure stdlib, zero dependencies)
• nvidia-smi for the GPU side
• the training curves declared in the config

Where it does not fit: It is not a training framework and does not touch your model: it plugs into YOUR loop (critic daemon + safe_io.publish + night_watch). Thresholds are per phase, never global.

Prices in Brazilian real (BRL). Stripe charges in BRL.

← all skills