Determinism & probe-check

A loop stops when an external check passes. That check is the only thing standing between “converged” and “ran out of budget.” So the check is not a detail. It is the stop oracle, and a flaky stop oracle is worse than none: it halts on noise, accepts on luck, and signs a receipt that means nothing.

The rule is short. No deterministic check, no loop. Everything on this page is the engineering behind that sentence.

The check is the stop oracle

In a check_fixpoint loop the check decides convergence directly. In a task_list loop each task carries its own acceptance oracle. Either way, the loop trusts an exit code to mean what it says.

If that exit code is non-deterministic, every guarantee downstream is built on sand. A green that flickers red breaks acceptance. A red that flickers green ends the loop on a false fixpoint. The halt reason is computed from observed state, and the state it reads is the check’s verdict. Certify the oracle first, or do not run.

”Run it ten times” is the wrong instrument

The intuitive probe is: run the check ten times; if it is green every time, call it deterministic. This is statistically underpowered, and not by a little.

By the rule of three, zero failures in n runs bounds the failure probability to roughly 3/n at 95% confidence. So:

0 failures in 10 runs  ->  flake rate p <= 0.30 (95% confidence)

A check that sails through a ten-run probe is statistically consistent with a 30% flake rate. That is not a deterministic oracle. That is a coin you have not flipped enough.

It gets worse, because the probe validates the wrong horizon. A real loop invokes the check on every iteration - call it ~60 invocations across a run. To hold the probability of even one false flake under 1% across all of them, the math is unforgiving:

1 - (1 - p)^60 < 0.01   =>   p < 0.00017   (about 0.017% per check)

And to certify a per-check rate that low by the rule of three, you would need:

n >= 3 / 0.00017  ~=  17,900 clean probe runs

So the path from a ten-run probe to a trustworthy oracle is not “run it more.” Seventeen thousand nine hundred runs is not a bigger checkbox - it is a confession that pass/fail of N runs is the wrong instrument. The fix is a change of shape, not a change of number.

Caution

A check that passes a small fixed-count probe carries an unstated confidence bound. “It was green ten times” is a claim about p <= 0.30, not a claim about determinism. The number you did not compute is the number that halts your loop at 3am.

probe-check reports a confidence bound

The shape change: stop reporting pass/fail, start reporting an achieved confidence bound.

loopexec probe-check is specified to derive its run count from a target tolerance (max_flake_rate) rather than a magic constant, then report the tightest flake bound it could establish. Ten is a floor, not a target.

check:
  deterministic_probe:
    max_flake_rate: 0.00017          # target tolerance - run count is DERIVED from this
    confidence_target: 0.99
    derived_runs: "ceil(3 / max_flake_rate)"  # rule-of-three; 10 is a floor, not a target
    report: confidence_bound         # report the achieved bound, not a pass/fail
    fingerprint: [exit_code, stdout_normalized, stderr_normalized]

The contract specifies the output as a bound, not a verdict:

{
  "tool": "loopexec",
  "status": "ok",
  "achieved_flake_bound": 0.0021,
  "confidence": 0.95,
  "runs": 1428,
  "broken_dimension": null
}

You read achieved_flake_bound and decide whether your loop can tolerate it. The instrument hands you the number; the policy is yours.

Capability honesty

probe-check and its confidence bound are Shipped (core) (SPEC section 11). The shape of the report - a bound, not a pass/fail - is what the binary emits today. Adversarial perturbation and the in-loop monitor below remain Planned.

The loop maintains a sequential bound

A pre-flight probe expires on iteration 1. Determinism is a property of (check x codebase), and the loop mutates the codebase every iteration. A clean probe at t0 says nothing about the check against the diff you have not written yet.

So the bound is not measured once - it is maintained. A loop re-runs the check every iteration, which makes every in-loop invocation a free Bernoulli sample of the check’s stability. loopexec is specified to fold those samples into a sequential lower bound (Wilson) on stability, and to halt check_not_deterministic the moment that bound drops below confidence_target.

    sequential_monitor:
      enabled: true
      ci_method: wilson_lower_bound
      reprobe_affected_subset_each_iter: true
      halt_when_stability_below: 0.99   # -> check_not_deterministic

This is the core inversion: determinism is not a checkbox you tick at startup. It is an invariant the runtime is specified to maintain - measured continuously, for free, against the actual code under test, and converted into a computed halt the instant it fails.

Capability honesty

The in-loop sequential (Wilson) monitor is Planned (SPEC section 11), though the iterating run loop that would feed it is Shipped. Read this section as the runtime contract - what loopexec is specified to maintain - not as behavior the current binary enforces.

Adversarial probing

A probe that runs the suite in one fixed order, on an idle box, at one wall-clock time, certifies the easy case. Real flakiness lives in the dimensions the lazy probe holds constant: test order, RNG seed, the clock, the timezone, concurrency, and machine load.

probe-check is specified to perturb those dimensions on purpose - and, critically, to report which dimension broke, not merely that something did.

    adversarial:
      shuffle_test_order: true
      randomize_seed: true
      induce_cpu_load: true
      max_parallelism: true
      perturb_clock: true
      report_broken_dimension: true

“The check is flaky” is not actionable. “The check fails under shuffled test order” is a bug report. The broken_dimension field is the difference between a halt you can fix and a halt you can only stare at.

Four properties, four halt reasons

“Deterministic” is routinely used to mean four different things. They are distinct properties with distinct cures, and the contract keeps them as four separate halt reasons so the fix is never ambiguous.

Property	The question it answers	Halt reason	Exit class
Determinism	Same inputs, same verdict?	check_flaky / check_not_deterministic	14 - oracle-untrusted
Hermeticity	Sealed from clock, net, ports, shared cache?	check_not_hermetic, hermeticity_violation	14 - oracle-untrusted
Adequacy	Does the check exercise the changed lines?	check_inadequate	15 - check-inadequate
Idempotency	Does it still pass on rerun in the same workspace?	check_has_side_effects	14 - oracle-untrusted

Collapsing these into one word hides the cure. Pulling them apart names it.

Determinism != hermeticity. A check that touches a real database, binds a fixed port, or reads a shared cache is non-deterministic by construction - it will either block the loop forever or silently poison the stop condition. Determinism is the symptom you measure; hermeticity is how you engineer it. The contract requires a deterministic check to be hermetic: frozen clock, seed, timezone, and locale; OS-assigned (:0) ports; ephemeral fixtures; pinned tool versions. A non-hermetic check is rejected, not looped on.

Determinism != adequacy. A check that returns 0 regardless of the diff is perfectly deterministic and perfectly useless. This is the false fixpoint: green on code the check never ran. Determinism asks “is the verdict stable?”; adequacy asks “is the verdict about my change?” The contract requires a coverage delta (every changed or added line is exercised) plus a mutation canary (a mutant injected into changed lines must turn the check red). Stable but blind => check_inadequate.

The false fixpoint

Determinism and adequacy fail in opposite directions and a green light looks identical from both. A flaky check halts you on noise. An inadequate check halts you on nothing - it converges instantly on a verdict that never touched your diff. Determinism alone cannot catch it; that is why adequacy is its own gate.

Determinism != idempotency. A check that passes once and fails on rerun in the same workspace (it wrote coverage.db, left a fixture behind, mutated shared state) is not flaky - it is side-effecting. The cure is a reset, not an edit to the check. Reporting this as flakiness would send you chasing a nondeterminism that is not there. The contract probes twice in one workspace to catch it, and reports check_has_side_effects => cured by reset.

Note

Idempotency failures are the most misdiagnosed, because the second run is deterministic - it deterministically fails. The signal is not variance across runs; it is a clean pass followed by a clean fail in the same workspace. Same inputs, different verdict, zero randomness.

Where doctor fits

Determinism and idempotency are measured by re-running the check, so they are also tracked continuously inside the loop. Hermeticity and adequacy are preconditions - properties of the check’s construction, checked once before the loop earns the right to start. Both kinds of gate converge on doctor, the fail-closed gate that refuses to run a loop on an untrustworthy oracle. Per the spec, doctor runs the pre-flight determinism probe and rejects a non-hermetic check (check_not_hermetic) or an inadequate one (check_inadequate) before the first agent call, alongside the two-zone isolation preconditions.

Status

This page describes the contract. Track what the binary does against SPEC section 11.

Capability	Status
probe-check confidence bound (O2)	Shipped (core)
In-loop sequential (Wilson) monitor (O2)	Planned
doctor adequacy gate (mutation canary, O4)	Shipped (core)
doctor adequacy coverage-delta tier + hermeticity (O3)	Planned

The doctor adequacy gate now ships its mutation canary: a mutation planted by --mutate-cmd in an isolated copy of the workdir that leaves the --check green halts check_inadequate (15). Hermeticity (O3) and the coverage-delta tier of adequacy remain contract requirements, not enforced gates - read those as “loopexec is specified to reject this,” not “loopexec rejects this today.” The honesty matters more than the optimism: a determinism story is only as credible as its weakest unverified claim.

---

Determinism is the precondition. Once the oracle is trustworthy, guards dominate success decides what a green verdict is allowed to mean, and the halt reasons turn every stop into a computed, replayable verdict.