Full suite — four log-driven harness fixes (MinimalPi, K=1)
Purpose: re-run the full 89-task suite after fixing four problems found in the
20260718 log analysis. Same config as the 20260718 checkpoint for a clean
before/after: MinimalPi, model qwen3.6-35b-a3b, K=1, AGENT_TIMEOUT_MULT=2.0,
all config/harness.env defaults, no fast-fail caps. Baseline to beat:
35/89 = 39.3% (suite__20260718-104156).
RESULT: 40/89 = 44.9% (+5 net vs 35/89). Elapsed 32h09m.
40 pass · 34 fail · 15 errored (13 AgentTimeoutError + 2 killed-by-signal).
Churn under K=1 is large again: 14 gained, 9 lost — most individual flips
are variance (the hidden-criterion near-miss finding), so read the totals, not
the per-task diff. All-time 35b union grows (pytorch-model-recovery is a
first-ever pass — it had never run in any prior suite).
Fix attribution (the point of the run)
Fix #1 — CLEAN WIN, not variance. pytorch-model-recovery FAIL/ERR(never
ran) → PASS reward 1.0 (439 KB pi.txt, real attempt). Was
NonZeroAgentExitCodeError "Unknown option: -" in both prior suites. The
"Task:\n" prefix un-bricked it and the model solved it.
Fix #4 — target flipped. torch-tensor-parallelism (the 89,121-char
visible-text task that motivated the trim) FAIL → PASS. The
runaway-text trim hook was active in all 89 trials. Can't fully separate from
K=1 variance, but the mechanism fired on exactly the task it was built for.
(Observability gap: no per-trim stderr marker, so I can't count how often the
trim actually engaged — added as a follow-up.)
Fix #3 — works.rec:N now renders (feal-differential rec:2, gcode rec:1,
password-recovery rec:1, etc.); 20260718 reported all of these as rec:0.
Fix #2 — backstop, did not need to fire. No WRITE GUARD hard-stop in any
trial: the parser-explosion storm (torch-pipeline-parallelism wg:2547 last
run) did not recur this K=1 run (it FAILed cleanly at 23m instead of ERR@30m).
The escalation stays a validated backstop for when the storm returns.
No regression is attributable to my changes
The 9 losses (fix-git, kv-store-grpc, query-optimize, break-filter-js-from-
html, code-from-image, financial-document-processor, llm-inference-batching-
scheduler, mcmc-sampling-stan, overfull-hbox) are clean-fail K=1 variance.
The only always-on behavior change is the fix-#4 trim, and it NEVER touched
the suspicious quick tasks: fix-git/kv-store-grpc/query-optimize had max
visible text 550–734 chars, far below the 8000-char trim threshold.
The 2 killed-by-signal errors are qemu-VM tasks starved in the 1–2 core /
2 GB containers: install-windows-3.11 exit 137 (SIGKILL / kernel OOM),
qemu-alpine-ssh exit 143 (SIGTERM / teardown). Not the pi V8-heap OOM (134),
not my exit-0 hard-stop (0 markers). Environment resource limits, orthogonal
to the fixes.
Bottom line: fix #1 is a guaranteed +1 forever; fixes #2/#3/#4 are correct and
active; the net +5 also carries K=1 noise, so a K≥3 replication is the real
signal for the borderline flips (unchanged from the 20260718 caveat).
Fixes shipped in this run (vs 20260718-104156)
Dash-leading instruction prefix (MinimalPi.run()). pi's CLI rejects a
positional starting with - and has no -- terminator, so
pytorch-model-recovery (the only task whose instruction.md starts with -)
died instantly with "Unknown option" in BOTH prior suites — it had never
actually run. run() now prepends "Task:\n" when the instruction starts
with a dash. Watch pytorch-model-recovery: it should reach turn 1 this time
(may still fail the medium task, but it will genuinely attempt it).
Write-guard storm hard-stop (WRITE_GUARD_STOP_AFTER=8). A single
garbled length-stopped generation whose leaked <tool_call> markup the
parser exploded into ~2547 near-identical write_file calls burned
torch-pipeline-parallelism's whole 30-min cap (wg:2547). The write guard now
counts blocked truncated writes per assistant message and ends the session
(exit 0 → verifier scores on-disk state) after 8 in one length-stopped turn.
Marker WRITE GUARD hard-stop.
Revived rec: counter (observability). failure_modes.py + the
transcript viewer matched the dead pre-2026-07-09 marker trimming runaway;
the recovery hook prints nudging (recovery N/2). 13 recovery nudges in the
last suite were reported as rec:0. Now counted + flagged; also added a
write_escalation flag for fix #2.
Runaway-text trim (TRIM_RUNAWAY_TEXT=on). The recovery hook only nudges
no-toolCall turns, so torch-tensor-parallelism's 89,121-char visible-text
block (on a toolUse turn) replayed into every later request. A new context
hook trims assistant visible text >8000 chars to head+tail (idempotent,
cache-safe), leaving thinking/tool calls/results/user messages intact.
Validation before launch: all deterministic gates green (check_write_guard,
check_context_strip incl. new storm + trim probes, check_transcript, +9 others),
extension renders/loads with no leftover sentinels, preflight OK, and live
e2e ALL GREEN (write-guard-truncated 3/3, write-guard-complete-pass 3/3 after a
relative-path prompt fix, context-strip-survives 3/3, preamble-visible 3/3).
What to check when this finishes
pytorch-model-recovery: PASS/FAIL/ERR instead of the instant CLI death.
torch-pipeline-parallelism: should end via WRITE GUARD hard-stop far under
the 30-min cap instead of churning a 2547-write queue.
torch-tensor-parallelism: shorter input growth (the 89k text no longer
replays); check whether the trim changed its verdict.
rec:N now populated in make failures / transcript flags — sanity-check it
matches the recovery nudges.
Overall pass count vs 35/89; the churn caveat (K=1) still applies — a K≥3
replication is the real signal for the individual flips.
loop-guard hard-stopped the sessiona bash command timed outempty final message (no text, no tool call) ×2long reasoning (21,708 chars) ×13loop-guard blocked a repeated call ×10
loop-guard hard-stopped the sessionempty final message (no text, no tool call)long reasoning (16,266 chars) ×6loop-guard blocked a repeated call ×15loop-guard escalation nudge
loop-guard hard-stopped the sessionempty final message (no text, no tool call)long reasoning (25,415 chars) ×19loop-guard blocked a repeated call ×10loop-guard escalation nudge