Experiment 2026-07-05 — write guard + min-p 0.0: combined smoke measurement

Main goal reminder: measure harness/server changes against terminal-bench 2.0 score on the local qwen models — rigorously, not by vibes. The loop is change → benchmark → reward delta → keep/revert.

Why this note

Since the last benchmark (overnight preamble batch, 2026-07-05) several changes landed but none has a benchmark number yet — we drifted into build/config mode. This experiment closes the loop with a fast smoke run, accepting a deliberate rigor tradeoff (see Decision rule).

What changed vs. the last benchmarked state (the confounded variables)

Mechanical write guard (WRITE_GUARD=on, default) — blocks truncated / oversized write calls; the model chunks. e2e-validated (mechanism fires), benchmark-UNMEASURED.
--min-p 0.0 on the qwen bases (config.yaml, server-side) — aligns sampling with Qwen's recommendation (llama.cpp default was 0.05). This shifts the baseline for every run, so it is a second, tangled variable.
ornith flags (--min-p 0.0 + --reasoning-budget 8000) — irrelevant to a qwen3.6-35b-a3b run; only matters when benchmarking ornith later.

Not a factor: the ~/.pi interactive port + llama-local-hooks do not affect benchmark runs — the harness renders its own in-container extension.

Decision rule (agreed 2026-07-05)

Run smoke on the combined current state (both #1 and #2 active).
Visible improvement vs. baseline → accept, attribute loosely ("one of these helped", not which).
Ambiguous / no clear signal → go back and isolate: WRITE_GUARD on/off A/B, and/or min-p 0.0 vs 0.05 (config.yaml + restart) A/B.
Caveat: two variables changed at once, so a positive result is a "does the current bundle help at all" gate, not a clean attribution.

Task set + baseline

Model: qwen3.6-35b-a3b (the primary MoE).
K = 1 (fast first look; K=1 is noisy — a single flip could be variance; bump to K=3 for a firm number at ~3× the time).
Tasks — chosen because they have a KNOWN fail baseline the write guard targets, so a flip is interpretable:
circuit-fibsqrt, dna-assembly, regex-chess, write-compressor — RULE 1 write-guard targets. Baseline (preamble batch): 0/6 flipped, 15 oversized-write length-stops; circuit-fibsqrt made 9/9 writes at 2–8× the limit. A flip to PASS here = the write guard working.
regex-log — canary, must stay PASS.
The default smoke set (regex-log / openssl-selfsigned-cert / polyglot-rust-c / path-tracing) does not exercise the write guard, so it is the wrong set for THIS question.

Command

make smoke MODELS=qwen3.6-35b-a3b \
  SMOKE_TASKS="circuit-fibsqrt dna-assembly regex-chess write-compressor regex-log"
# watch live in a 2nd terminal:  make watch

Results land in

runs/smoke__qwen3.6-35b-a3b__<stamp>/ — read with make results and make failures (wg:N = write-guard blocks, length-stop counts). Compare pass/fail + length-stops against the preamble-batch baseline above.

Outcome (run `smokeqwen3.6-35b-a3b20260705-175320`, 3h30m wall)

Verdict: no visible improvement on the decisive axis → ISOLATE (per the rule). mean reward 0.20, 1/5 pass.

task	result	agent	out-tok	notes
regex-log	PASS	2m05s	20k	canary ✓
regex-chess	FAIL	15m39s	117k
dna-assembly	FAIL	37m40s	250k
write-compressor	ERR	30m00s	216k	timeout; 1 assertion away
circuit-fibsqrt	ERR	2h00m	804k	full agent timeout; runaway; 1 assertion away

Key findings (from transcripts, not just the table): - The write guard IS active and DID fire — 15 "write guard active" startup markers, and block markers: 10 oversized + 10 truncated. Mechanism now validated in-benchmark, not just e2e. But firing did not flip any target task — all 4 still FAIL/ERR. - The targets fail for reasons the guard doesn't touch: circuit-fibsqrt is a runaway (804k out, length:9, pr:13, full 2h timeout) — blocking its writes just fed a retry/churn loop, didn't stop it. Two tasks (circuit-fibsqrt, write-compressor) are "one assertion away" near-misses. - So the write-guard hypothesis ("mechanical RULE 1 flips the target tasks") is not supported here. The guard fixes a real but non-decisive failure mode for these tasks.

Confounds / follow-ups the result demands: 1. Guard net effect unknown — does block→retry ADD churn (esp. on the runaway)? → WRITE_GUARD on/off A/B, same tasks/min-p. 2. min-p 0.0 looser sampling may worsen runaways → min-p 0.05 vs 0.0 A/B. 3. Cheap first: read circuit-fibsqrt's transcript to see WHY it ran 2h (guard-retry vs min-p looping vs task difficulty) before spending 7h on a blind A/B. 4. Tooling gap: failure_modes.py wg:N=0 despite ~10–20 real blocks — it reads message events but the block markers are console.error in trial.log. Fix so guard activity shows in make failures.

circuit-fibsqrt transcript diagnosis (the 2h ERR)

Task: build a gates.txt logic-gate netlist (<32000 lines) computing fib(isqrt(N))%2^32. Correct approach = a generator script (ref solution is a 444-line solve.sh). 129 turns, 123 bash, 4 write, 804k out-tok. - Model chose right (a gen_gates.py generator) but wrote it at ~75k chars → truncated (length) 4×; the write guard correctly blocked all 4. - It then built gen_gates.py via 119 bash heredocs — but 118 use cat > (OVERWRITE), only 1 >>, and there are only 9 unique bodies among 119 writes; the last 8+ are byte-identical (same 1886-char file). → a pure repeated-action loop: rewrite the same broken generator until the 2h timeout. (Generator was 75k chars vs the 444-line reference → the model's design is ~5× bloated / wrong; task is genuinely very hard, near-miss at best.) - Verdict on our variables: the write guard is not the culprit — it blocked the truncated writes correctly, the model routed to bash (which the guard doesn't govern) and looped there. min-p 0.0 is probably not the driver either: this is a high-level action loop, not token-sampling diversity (moderate confidence, unproven). So the expensive WRITE_GUARD / min-p A/B is low priority — the transcript already shows neither is decisive for this task. - The real gap (new lever): NO harness mechanism catches a repeated- identical-action loop. Runaway recovery only fires on no-toolCall text runaways; every loop turn here has a bash toolCall, so it never triggered (rec:0). The write guard only governs write. A loop detector on repeated near-identical tool calls (esp. same-file cat > overwrites) → nudge/abort would fail-fast and reclaim ~1.5h/occurrence. This is CLAUDE.md score-lever #2 ("loop fixes"), now with a concrete signature to key on.

Follow-up 2026-07-06 — root cause was the pi VERSION, not the write mechanics

Tracing circuit-fibsqrt's truncations (all length-stops at output=32000 despite declared maxTokens=32768) led to the real bug: the benchmark installed the FROZEN @mariozechner/pi-coding-agent@0.73.1, whose buildBaseOptions clamped output to Math.min(maxTokens, 32000) (gh earendil-works/pi#4539). The active @earendil-works fork (what native runs) fixes it. Changes made: - MinimalPi.install() now installs @earendil-works/pi-coding-agent@0.80.2 (== native); e2e_replay.py mirrors it. e2e on 0.80.2: 4/6 green (drop-in). - DEFAULT_MAX_TOKENS 32768→65536 (now honored), CONTEXT_WINDOW→196608, COMPACT_AT_TOKENS→180000 (coupled to -c 262144). NEXT: re-run the write-heavy failures (circuit-fibsqrt, write-compressor) + regex-log canary on 0.80.2 + 64k to see if honoring the budget flips them. (dna-assembly/regex-chess made ZERO writes → maxTokens is irrelevant to them.)

Follow-up 2026-07-06 (later) — the 0.80.2 reruns all OOMed; harness artifact, now fixed

The "NEXT" reruns (smoke 20260706-003503/-060745/-065840/-071847) are INVALID for the maxTokens question: every circuit-fibsqrt trial died NonZeroAgentExitCodeError exit 134 — V8 heap OOM inside the container, ~7 min into streaming the first giant write. Root cause: pi print mode's per-tick message_update events each carry the full partial message → O(max_tokens²) bytes (~6 GB at 64k) queued on an unbounded stdout buffer (core/output-guard.js); a 1–2 CPU container never drains it. Affects 0.73.1 too (mock replay OOMs even earlier) — the clamp removal just quadrupled the volume. Container-RAM (8G) and NODE_OPTIONS heap (6G) workarounds both died, as the math predicts; both knobs reverted to off. Fixed at the source 2026-07-06: MinimalPi.install() + e2e_replay.ensure_pi() patch print-mode.js to drop message_update/tool_execution_update events (nothing consumed them — the pipeline grep'd them out). Mock-validated (48k-delta write: OOM@512MB unpatched → 316MB peak patched). The write-heavy rerun (circuit-fibsqrt, write-compressor + regex-log canary) still needs to happen, now on the patched harness.