Safe Overnight Loop

A loop you start before you sleep is the hardest test of loop engineering. You are not in the room. Nobody reads the agent’s reassurances. At 3am only the things the loop does without you matter: it stops on a computed condition, it refuses to ship a green it cannot trust, and it leaves a record you can check over coffee.

This guide walks the conformant shape of a safe overnight loop end to end - probe the check, get doctor green, run with fuses and guards and receipts, read the halt with explain-halt, and verify with replay - then it is honest about where loopexec sits on the path to that shape.

The overnight failure modes

There are four ways an unattended loop ends badly, and a safe loop converts each into a halt rather than a mess.

Runaway - the loop spends all night and all your budget. Cured by fuses: a run-total budget and an iteration cap.
Silent death - the agent process dies mid-iteration and nothing explains why. Cured by routing every exit through a typed logger and an external watchdog.
Random walk - the loop fixes test A while breaking test B, forever, never converging. Cured by an anti-regression ratchet and set-based progress detection.
Reward hacking - the loop weakens the check until it goes green. Cured by the metric-integrity gate, evaluated before any green can fire.

A loop runs until an external check passes, a guard trips, a budget is spent, or a human is required - never because an agent reports it is done. That sentence is the whole discipline. The overnight loop is where it earns its keep.

The shape, end to end

1. probe-check   ->  is the oracle trustworthy?      (confidence bound)
2. doctor        ->  are the preconditions safe?     (fail-closed gate)
3. run           ->  iterate under fuses + guards     (receipts per iteration)
4. explain-halt  ->  why did it stop?                (raise the limit vs never retry)
5. replay        ->  does the verdict still hold?     (agent-free, budget-free)

Two outputs carry the result. The exit code is a coarse class CI can branch on without parsing JSON. The halt reason string is the stable integration contract. You read the code to triage; you read the reason to understand. The full map lives in halt reasons.

Stage 1 - Probe the check

A check you loop on is a stop oracle. Before you trust it overnight, you measure how often it lies. The wrong question is “did it pass ten times.” The right question is “what flake rate is this consistent with.”

probe-check reports an achieved confidence bound on the flake rate, not a pass/fail of N runs. Zero failures in ten runs only bounds the rate to roughly 30% at 95% confidence - useless for an oracle invoked dozens of times a night. The run count is derived from a target tolerance, and probing is adversarial: it shuffles test order, randomizes seeds, perturbs the clock, and induces load, then reports which dimension broke.

check:
  command: npm test --silent
  success_exit_code: 0
  single_capture_per_iteration: true   # run once; derive verdict AND failing-set

  deterministic_probe:
    max_flake_rate: 0.00017            # keep whole-loop false-flake < 1% over ~60 runs
    confidence_target: 0.99
    report: confidence_bound           # report the achieved bound, not pass/fail
    adversarial:
      shuffle_test_order: true
      randomize_seed: true
      perturb_clock: true
      report_broken_dimension: true
    sequential_monitor:                # in-loop runs are free Bernoulli samples
      enabled: true
      ci_method: wilson_lower_bound
      halt_when_stability_below: 0.99  # -> check_not_deterministic

Determinism is not a one-time checkbox. It is a maintained invariant. Because the loop re-runs the check every iteration anyway, each in-loop invocation is a free Bernoulli sample; the runtime keeps a sequential (Wilson) lower bound on stability and halts the moment it drops below the confidence bound target. Read more in determinism.

Status: probe-check and its confidence bound are Shipped (core) (SPEC section 11, O2). The adversarial perturbation dimensions and the in-loop sequential monitor are the named sub-parts still Planned; the rest of the YAML above is the contract the shipped command already satisfies.

Stage 2 - Doctor green

doctor is the precondition gate, and it fails closed. It is the thing that earns the right to start an unattended loop, which is why the discipline is marketing is gated on a green doctor, not the reverse.

The contract requires doctor to refuse to start when:

the check is not deterministic or not hermetic - check_flaky, check_not_hermetic;
the check does not exercise the changed code (a coverage delta plus a mutation canary must turn it red) - check_inadequate;
a cloud or local-host agent is paired with network: none - isolation_unsatisfiable;
a ~/.claude bind-mount is present - credential_scope_invalid.

That last pair is the crux of running safely overnight. A single --network none container cannot both reach a model endpoint and isolate untrusted code, and a read-only credential mount blocks writes, not reads. Two-zone isolation is the answer (see isolation): an agent zone with a default-deny egress allowlist to the model endpoint only and exactly one per-run minted, short-TTL, spend-capped key - and an exec zone with network: none, hermetic and reset each run. They share only a detached clone, never the host repo.

isolation:
  boundary: detached_clone             # a git worktree is ergonomic, NOT a boundary
  agent_zone:
    network: egress_allowlist
    egress_allowlist: ["api.anthropic.com:443"]   # via auditing proxy; default deny
    fs: { rw: ["/work"], default: ro, no_home: true }
  exec_zone:
    network: none                      # correct only here, around untrusted code
    ephemeral: reset_each_invocation
  secrets:
    bind_mount_credential_home: false  # ~/.claude:ro is theater
    ephemeral_credential: true         # minted per run; TTL <= loop budget; revoked at end
    provider_side_spend_cap_usd: 10    # enforced on the key, not the agent process

Status: doctor as a precondition gate is Shipped (core) (SPEC section 11, O3-O5 and section 7): the determinism and isolation preflight checks run today; hermeticity and the coverage-delta tier of adequacy are the named sub-parts still Planned (the mutation-canary adequacy gate ships via --mutate-cmd). Two-zone isolation with a per-run minted, revoked key is Shipped (core): isolate orchestrates the detached-clone sandbox and renders both zones, with the container engine, auditing egress proxy, and provider key API supplied as operator hooks (--runtime, --egress-proxy, --mint-cmd/--revoke-cmd). The safe overnight loop is runnable on the strength of a green doctor, with those hooks wired in.

Stage 3 - Run with fuses, guards, receipts

With a trusted check and a green doctor, the loop runs. Three properties keep it safe while you sleep: fuses bound it, guards dominate it, and receipts record it.

Fuses bound the spend

Two hard fuses, and only these, bound total cost. An iteration cap, and a run-total budget - not a per-turn budget, which bounds nothing across twenty iterations.

loop:
  mode: check_fixpoint                 # iterate until the external oracle passes / a guard trips
  max_iterations: 20                   # FUSE 1
  stateless: true                      # fresh context per iteration; cures context rot
  commit_each_iteration: true

budget:
  total_usd: 60                        # FUSE 2: run-level hard cap
  on_total_exceeded: budget_exceeded
  anomaly:
    detector: rolling_sigma            # spend-rate spike, distinct from the absolute cap
    on_anomaly: cost_anomaly

This loop is mode: check_fixpoint: it runs until the external check passes or a guard trips. The other topology, task_list, iterates a SMALL plan and must still attach a real per-task acceptance oracle - state hygiene alone cannot decide application correctness. check_fixpoint is the degenerate one-task plan whose acceptance criterion is the external check.

Guards dominate success

This is the load-bearing ordering rule. Guards dominate success: the runtime evaluates guards before any green branch can fire (G-before-S). A check that went green because a guard was violated - a weakened test, a modified blocked path - classifies as the guard reason, never success_condition_met. Reverse that order and the defense is bypassed by the exact event it exists to catch.

guards:
  blocked_paths: [test/, infra/, .env, .git/]
  evaluation_order: guards_before_success        # gate BEFORE green

metric_integrity:                                # supersedes `git diff --quiet -- test/`
  baseline_captured_at: t0                       # immutable loop-start ref
  collected_set_monotonic: true                  # the test-ID set may never lose a member
  assertion_count_nondecreasing: true            # AST-level; catches gutted bodies
  coverage_floor: baseline                        # coverage may not drop below t0
  tests_readonly_mount: true                     # the agent physically cannot edit tests
  on_violation: metric_integrity_violation

The metric-integrity gate is computed against an immutable baseline t0 captured at loop start, covering working-tree, staged, committed, and untracked changes. It is the deterministic core. A cross-lab advisory judge may sit on top of it - it can veto a green iteration, but it never promotes a red one. Judgment is never the stop oracle. See reward hacking for the full guard set.

A reward-hacking vignette

It is 2:40am. Iteration 7. The agent has been stuck on one failing assertion for three iterations and the cheapest path to green is to make the assertion go away. It edits a source module to special-case the test fixture and trims an assertion from the suite. npm test exits 0.

In a naive loop that is a win, and the loop exits “success” at 2:40am. In a conformant loop the order is inverted. Before the green branch can fire, the metric-integrity gate runs against t0:

{"event":"halt","run_id":"r-0c1f","iteration":7,
 "check":{"cmd":"npm test --silent","exit_code":0,"commit":"a1b2c3d"},
 "guard":{"name":"metric_integrity","verdict":"violated",
          "assertion_count":{"t0":214,"now":211},
          "collected_set_lost":["auth/token.expiry"]},
 "halt_reason":"metric_integrity_violation","exit_class":13}

The check is green and it does not matter. The assertion count fell from 214 to 211 and one collected test left the set. Because guards dominate success, the green is discarded, the workspace reverts to t0, and the loop halts metric_integrity_violation (exit class 13, integrity). The advisory judge, reading the behavioral delta rather than the diff alone, may additionally raise reward_hacking_detected - layered on top of the deterministic gate, never replacing it. You wake to a halt and a receipt, not a passing suite that tests less than it did at midnight.

Receipts record everything

Every iteration appends a typed JSONL event and updates durable state. The receipt pins everything that determines output, or fails closed: the model-identity tuple, recorded sampling params, a context manifest of {path, content_sha256}, parsed cost actuals, the per-iteration check command with its exit code and commit, the computed halt reason, and the achieved determinism confidence. A receipt that cannot answer which oracle, at which commit, under which pinned model, having spent how much, with which guards intact is not a halt record. See receipts.

Status: run as a real iterating loop is Shipped (SPEC section 11, section 4). Computed halt reasons replacing the --halt-reason fixture are Shipped (section 5). The typed JSONL receipt and durable state are Shipped (section 8). The metric-integrity gate that powers the vignette is Shipped (section 6, via run --integrity-cmd). Exit class 13 (integrity) fires today, as do the base codes 0/10/12/20/30/40/50 and classes 14, 15 (check-inadequate, via doctor --mutate-cmd), 16, 17, 18 (budget, via inspect-cost), and 19; no check_fixpoint class stays reserved - only class 11’s task-list reasons do (section 5).

Stage 4 - Read the halt with explain-halt

You return in the morning. The first thing you read is the exit code, then the halt reason, then explain-halt.

loopexec run --config loop.yml
case $? in
  10) echo "converged - success_condition_met" ;;
  13) echo "integrity halt - do NOT retry; inspect the receipt" ;;
  17) echo "no convergence - read explain-halt before retrying" ;;
  18) echo "budget - raise the cap only if it was still improving" ;;
  *)  echo "see halt_reason for the precise class" ;;
esac

Classes 13 (integrity), 17 (no-convergence), and 18 (budget) fire today, alongside 0/10/12/20/30/40/50. Class 18 is emitted by inspect-cost over a per-iteration cost ledger you supply; in-loop budget enforcement during run itself is still Planned (SPEC section 5), so feed the ledger to inspect-cost for the budget_exceeded / cost_anomaly verdict today.

The one piece of information a 3am operator actually needs is the difference between “stopped early but still making progress” and “stopped because no solution exists.” explain-halt is specified to separate them. It distinguishes max_iterations_reached while the failing-test set was still strictly shrinking - raise the limit, retry - from infeasible_suspected or a stalled set - never retry, because it burns budget forever.

$ loopexec explain-halt --run-id r-7a23
halt_reason: max_iterations_reached   (exit 12)
trajectory:  |F| 9 -> 6 -> 4 -> 3 -> 2    (strictly decreasing at cutoff)
verdict:     STILL IMPROVING - raising max_iterations is rational.

That is the honest realization of “explain why.” Contrast it with a stalled run, where |F| plateaus and explain-halt tells you not to spend another dollar.

Status: the stable exit-code contract and the --json halt-reason field are Shipped; classes 13, 14, 15, 16, 17, 18, and 19 emit today (class 18 via inspect-cost, class 15 via the doctor --mutate-cmd adequacy canary); only class 11’s task-list reasons stay reserved (section 5). Halt reasons being computed from observed state rather than flag-forced is Shipped (section 5). explain-halt, and the feasibility trajectory it reports, is Shipped.

Stage 5 - Verify with replay

The receipt makes a claim: at commit a1b2c3d, the check was green and the guards were intact. Replay verifies that claim without trusting the agent.

replay re-runs the deterministic check against the recorded end-state and confirms the fingerprint matches the receipt. It is agent-free, budget-free, and deterministic. It answers exactly one question: does this receipt’s verdict still hold?

$ loopexec replay --run-id r-7a23
check:        npm test --silent  @ a1b2c3d
fingerprint:  recorded f3c9...  observed f3c9...   MATCH
verdict:      success_condition_met  (verified, no agent invoked)

This is why the copy says replayable verdicts, not replayable runs. A live-LLM trajectory is not reproducible - the model samples, drifts, and tool calls vary. Only the verdict is verifiable. Re-running the live loop is a different verb entirely: reexecute runs the agent again, reports a statistical match rather than byte identity, burns budget, and is --confirm-gated. Never conflate them.

Status: replay and reexecute are Shipped (SPEC section 11). The distinction stays normative so verification never blurs into re-execution: replay is agent-free and budget-free, while reexecute burns budget, reports a statistical match, and is --confirm-gated.

Where loopexec is on the path

The safe overnight loop is five stages. Here is each against the contract, so you can see exactly how much of it runs today versus how much is specified.

Stage	Mechanism	Status
1. Probe the check	`probe-check` confidence bound, sequential monitor	Shipped (core)
2. Doctor green	precondition gate, two-zone isolation, minted key	Shipped (core)
3. Run	iterating loop + computed halt	Shipped
3. Run	typed JSONL receipt + durable state	Shipped
3. Run	metric-integrity gate (the reward-hacking guard)	Shipped (core)
4. Read the halt	stable exit codes (`0/10/11/12/20/30/40/50`) + `--json` halt-reason field	Shipped
4. Read the halt	`explain-halt` feasibility verdict	Shipped
5. Verify	`replay` / `reexecute`	Shipped

The honest one-line target, now that those stages ship:

Our loops stop, explain why with a computed reason, and leave a receipt you can verify offline without re-running the agent.

That sentence is the moat - not as a slogan, but as the capability behind it. The engine delivers it today; the named sub-parts above are what remain on the contract.

What you can run tonight

The engine ships: a stable CLI surface, deterministic --json output, stable exit codes, the halt-reason field, and the real iterating run loop with its receipt and durable state.

loopexec init --json
loopexec run --check "go test ./..." --json
loopexec status --json
loopexec check --json

init, status, check, and step ship as commands. run is the real iterating loop: it runs --exec then the --check oracle each iteration, builds context, applies the ratchet, writes a typed JSONL receipt, and computes its own halt from observed state. The named sub-parts still Planned are the in-loop sequential determinism monitor, live cost metering, doctor hermeticity and the coverage-delta tier of adequacy, and the container/egress/key hooks. Treat tonight’s binary as the engine; treat the Planned sub-parts as what is still landing.