35/89 = 39.3% (baseline suite 20260703: 33/89 = 37.1%). Elapsed 26h45m.
Purpose: user-requested checkpoint of the whole fix stack (pi 0.80.2 +
print-mode/exit patches, maxTokens 65536, qwen-tools surface, preamble,
write/loop guards, bash timeout 90s, reasoning-budget 8000 server-side)
before deciding on an official K=5 leaderboard run. Config: all
config/harness.env defaults, AGENT_TIMEOUT_MULT=2.0, no fast-fail caps.
Headline
Net +2 tasks vs the pre-fix baseline, but the churn under K=1 is large:
11 gained, 9 lost. Confirms the "hidden-criterion near-miss / run K≥3"
variance finding — the two suite runs agree on far fewer tasks than the
totals suggest.
5 first-ever passes (not in the prior 44-task all-time union):
break-filter-js-from-html, build-pov-ray, large-scale-text-editing,
llm-inference-batching-scheduler, overfull-hbox. All-time 35b union is now
49/89 = 55.1%.
Gained vs 20260703: the 5 above + financial-document-processor,
fix-ocaml-gc, headless-terminal, openssl-selfsigned-cert, query-optimize,
reshard-c4-data. Lost vs 20260703: build-pmars, count-dataset-tokens,
feal-linear-cryptanalysis, largest-eigenval, merge-diff-arc-agi-task,
qemu-startup, rstan-to-pystan, sqlite-db-truncate, tune-mjcf.
12 errored trials (13.5%): 11 AgentTimeoutError (caffe-cifar-10,
circuit-fibsqrt, crack-7z-hash, feal-linear-cryptanalysis,
make-mips-interpreter, qemu-alpine-ssh, qemu-startup, rstan-to-pystan,
torch-pipeline-parallelism, train-fasttext, write-compressor) + 1
NonZeroAgentExitCodeError (pytorch-model-recovery). Post-session-exit hang
is fixed, so these are real cap exhaustions, not the teardown bug.
NB feal-linear-cryptanalysis and qemu-startup were baseline PASSES that
became timeouts — worth transcripts before the K=5.
Tokens: in=533.7M (cache 517.3M = 96.9%), out=7.2M. Roughly 2.25× the
20260703 input total — the fixed harness runs longer sessions (loop guard
keeps trials alive to the cap instead of dying early).
Leaderboard context
Comfortably above little-coder (24.6 ± 3.2 on the official board). Official
submission would target terminal-bench 2.1 (2.0 board closed), require
AGENT_TIMEOUT_MULT=1.0 (this run used 2.0 — expect some of the long passes
to become fails there) and K=5.
New fast-false-done shape for this task: wrote run.py, ran a 2-command
self-test, declared "Done." at 19s agent time. Its own check passed; the
hidden criteria did not. Different from the earlier deadlocked-smoke-test
shape (fast__…113104) — this is the hidden-criterion near-miss class, K≥3
territory.
loop-guard hard-stopped the sessionempty final message (no text, no tool call)long reasoning (20,667 chars) ×7loop-guard blocked a repeated call ×13loop-guard escalation nudge
Baseline-PASS → ERR regression checked: no guard interference (zero
loop-guard blocks; one legitimate empty-final recovery nudge late in the
session). The model spent the 1h cap rewriting a multithreaded brute-force
attack in C — hard-task variance, not harness.
#
result
total
agent
in/out tok
flags
1
ERR
1h04m
1h00m
4406467/292016
trial errored: AgentTimeoutErrora bash command timed out ×2empty final message (no text, no tool call)long reasoning (22,118 chars) ×17runaway / empty-final recovery fired
loop-guard hard-stopped the sessionempty final message (no text, no tool call)long reasoning (21,435 chars) ×4loop-guard blocked a repeated call ×10loop-guard escalation nudge
Instant death, 0 tokens: pi rejected the task prompt itself —
Error: Unknown option: - You are given a PyTorch state dictionary….
instruction.md is the only one of the 89 that starts with -; pi's CLI
(cli/args.js) errors on any positional starting with a single dash and has
no -- terminator. The 20260703 baseline trial died identically, so this
task has NEVER run in any suite. Harness fix: prefix dash-leading
instructions in MinimalPi.run().
Baseline-PASS → ERR regression checked: no guard interference. 100 shell
commands / 39 write_file thrashing QEMU serial-over-telnet bridging
approaches (-serial pty + socat at the end) until the 30-min cap. NB the
20260703 "pass" was itself a pass-despite-AgentTimeout, so this task was
always marginal at this budget.
765 assistant turns / 105.8M input tokens in 66 min (~20% of the entire
run's input) grinding ~5s turns at ~170k ctx, then self-declared done and
failed. Nothing bounds turn count or per-trial input tokens — recorded as a
wall-clock-lever observation in the ledger.
#
result
total
agent
in/out tok
flags
1
FAIL
1h07m
1h06m
105783730/365299
long reasoning (30,860 chars) ×14loop-guard blocked a repeated call ×5
wg:2547 is not 2547 generations — it is ONE garbled length-stopped
generation. The model looped emitting Qwen <tool_call> XML as text; the
parser split it into ~2547 near-identical write_file calls with wire
markup leaked inside the content argument. The write guard blocked every
one (correct per-call), but pi processed the whole queue until the 30-min
AgentTimeout killed it mid-write of the giant message_end line (truncated
at exactly the 64KiB pipe buffer). No escalation path exists for this
shape — the loop guard never counts write-guard blocks. Proposed
per-message blocked-call cap in the ledger.
New recovery blind spot: an 89,121-char visible-TEXT reasoning block in a
turn that still ended in a toolCall (stop=toolUse). The runaway recovery
only fires on no-toolCall turns, so this replayed ~22k tokens into every
later request. Trial also went to lg:11 and was ended by the loop-guard
hard-stop.
#
result
total
agent
in/out tok
flags
1
FAIL
34m33s
28m25s
7519104/182399
loop-guard hard-stopped the sessionempty final message (no text, no tool call)long reasoning (24,062 chars) ×7loop-guard blocked a repeated call ×11loop-guard escalation nudge
Loop-guard escalation worked end-to-end here: 38 blocks, 2 nudges, then
hard-stop at 45m — one of 7 trials where the hard-stop fired (first
suite-scale validation of the mechanism). Still one assertion away on the
verifier.
#
result
total
agent
in/out tok
flags
1
FAIL
46m06s
45m30s
37788187/324880
loop-guard hard-stopped the sessionempty final message (no text, no tool call) ×2long reasoning (28,676 chars) ×13loop-guard blocked a repeated call ×38loop-guard escalation nudge ×2