Follow-ups and proposed changes from run analyses — parsed fresh from AGENT_TODOS.md on every site build. 62 open · 56 done (hidden by default).
This file is for concrete follow-up ideas that coding agents should notice before
changing the harness. Keep entries short, testable, and tied to observed run
behavior. After any run investigation or analysis/<task>.md, update this file:
check off what the investigation resolved, add new - [ ] items for what it
proposed. The Pages site renders open items at todos.html.
Closing an item does NOT require a dedicated A/B run. Ship on the cheap
evidence — the deterministic scripts/check_*.py gate, the make e2e case, and
a normal smoke/fast run showing the mechanism fired without a collapse — and say
in the resolution note what you actually verified. Older items phrased as "A/B
X" can be resolved that way too; keep a matched A/B for broad default flips,
suspected regressions, or a finding you intend to write up. See the "Testing
changes" section in CLAUDE.md.
run, which if failed or too long would stop, and a second run of pi would start, that would have access to the logs of the benchmark, and a prompt to fix the harness looking at what wen wrong. After that the run would repeat. This could be tested on the hardest benchmarks, which qwen 35b never makes right.
Find out why a pi request can reach the server with NO max_tokens, and
give every model the server-side ceiling. Evidence: gemma-4-12b first light
(runs/smoke__gemma-4-12b__20260726-212728, regex-log ERR) — ONE generation
of 215,206 tokens ran the full 30-min agent budget at ~118 tok/s and ended
only when pi disconnected; pi.txt is 0 bytes, in/out 0/0. It was not
context-bounded (216,924 of 262,144 used). Probed the same day: max_tokens
IS honoured when sent (2000 → predicted_n 2000, finish_reason: length,
MTP does not overshoot) and --reasoning-budget IS working (budget 200 →
thinking capped ~128 tokens), so this was 215k tokens of VISIBLE content from
a request carrying no cap. Note the financial-document-processor item below
shows a runaway that DID stop at 65536 — so the cap normally arrives, and the
interesting question is which code path drops it. Mitigated for one model
only: gemma-4-12b now sets -n 65536 in
llama_cpp_on_my_desktop/tools/llama-swap/config.yaml (llama.cpp's -n
defaults to -1 = infinity); verified an uncapped runaway prompt now stops at
exactly 65536. Every other model entry is still unbounded. No harness
guard can cover this — write/loop/e2big guards and the context pruner all run
between turns and cannot interrupt one in-flight generation.
Re-run regex-log on gemma-4-12b now that -n 65536 bounds the generation,
to see whether the capped turn leaves the agent enough budget to finish (the
trial burned 30 min without pi ever receiving a message).
Resolved 2026-07-27 by runs/fast__gemma-4-12b__20260727-010519 (regex-log
FAIL, 9m53s, no cut). The cap works and is load-bearing; the task fails for a
different reason. Turn 2 stopped at EXACTLY 65536 output tokens
(stopReason=length) — so the runaway still happens, but is now a bounded ~9-min
turn instead of the unbounded 215k-token / 30-min ERR. pi received the message
this time (74,408 = 65536 + 8872 output tokens, accounting checks out). The
remaining failure is separate: zero tool calls in the entire 4-turn session —
the final message is the answer as a fenced Python snippet ending in
with open("/app/regex.txt","w"), narrated instead of executed, so nothing landed
on disk. Task-specific, not model-wide (fix-git/nginx/openssl/cancel-async all
pass in <1 min, which needs working tool calls); regex-log was also the single
first-light smoke failure. Runaway recovery fired (rec:1) and changed nothing.
Chase gemma-4-12b's "narrate instead of execute" shape on regex-log —
zero tool calls across a whole trial, ending in a code block that describes the
file write. Evidence above. The existing empty-final / runaway triggers do not
catch it: the final message is a long, well-formed, natural-stop answer, so it
looks like a legitimate completion. Candidate detector: a session that ends with
a fenced code block containing a file-write/open(...,"w") call while the
session made zero write/edit tool calls → nudge to actually execute it. Would
fail-open and is cheap; needs a check_*.py gate before shipping.
Investigate gemma-4-12b's prompt-token churn — the dominant failure
signature in runs/fast2__gemma-4-12b__20260727-014500. Input tokens vs the
27b on the same task: financial-document-processor 2.05M vs 85k (24×),
mailman 5.79M vs 1.26M (4.6×, and 147 tool-use turns before the 10-min cap),
fix-ocaml-gc 4.70M vs 1.67M (2.8×, 151k peak context — over the declared 131k
window — plus lg:2 loop-guard blocks). sanitize-git-repo in the fast set burned
2.23M in a 3-min window. The model is ~2× the MoE's tok/s but spends it on turn
count, not progress, so the speed advantage never reaches the score. Worth
checking whether the context pruner/KEEP_TOOL_RESULTS behaves differently here,
and whether the per-turn KV prefix break (the known ~8.3k/turn replay issue) is
worse for gemma's template.
A/B PREAMBLE=off on gemma-4-12b. RULE 2 ("verify before declaring done")
is the preamble's proven win on qwen (flipped openssl FAIL→PASS 3/3), but it is
not landing here: in fast, 4 of 5 failures self-declared done on a clean stop
and 0 of those ran any check (V=0 on query-optimize, sparql-university,
filter-js-from-html). Only headless-terminal (a pass) showed real self-checking
(V=11). Do not assume the preamble helps this model — measure it.
Follow up the two gemma-4-12b near-misses that are ONE assertion away:
query-optimize and sparql-university (both failed but passed >=1 hidden subtest,
runs/fast__gemma-4-12b__20260727-010519). Cheapest available score upside on
this model.
A/B QWEN_TOOLS=off on gemma-4-12b. qwen_tools=on is the default and
gives every model the Qwen-named tool surface; that surface was picked because
it matches Qwen3.6's post-training, and there is no reason to think it matches
Gemma 4's. gemma-4-12b is fast enough (~142 tok/s, 3 smoke passes in 39s/1m09s/44s)
that a matched pair is cheap here.
check if the kv compression for context is enabled, consider fp8 or even fp16
Resolved 2026-07-10: KV cache was q4_0 for ALL models (global KV_TYPE=q4_0
default in compose.yaml, applied via the common macro's
--cache-type-k/-v). Measured per-model VRAM fit on the 3090 (empirically,
-fit off so a fit is real) and set per-model overrides in
llama_cpp_on_my_desktop/tools/llama-swap/config.yaml (last --cache-type
wins): qwen3.6-35b-a3b → f16 (this MoE has few full-attn layers → small
KV; q4_0 20.5 GB / q8_0 21.7 GB / f16 23.6 GB used, ~0.5 GB free at full
262k, 100% GPU util, 165.8 tok/s — TIGHT but loads+decodes). qwen3.6-27b →
kept q4_0 @ 240k (dense → large KV; q8_0 OOMs at 240k, needs ~449 MiB more,
and only fits at q8_0 if -c drops to ~150k — not worth losing 37% of the
context window). Other models (gemma/ornith/north/bielik) untouched (still
q4_0). config.yaml edits left uncommitted (matches the repo's convention for
that file). NB f16 on the 35b is a benchmark variable — old runs used q4_0.
a bit of far fetched idea, but what if pi agent could do a first short
Spun up from the question "how would a pi-recursive agent differ from the
pi-subagents chain, and could it decompose better?" pi-recursive
(https://pi.dev/packages/pi-recursive, v0.6.1) has ONE tool rlm_query({prompt,
context?, fork?}): the model writes each subtask itself and a child pi runs it
with a FRESH context, bounded by env vars (RLM_MAX_DEPTH/RLM_MAX_CALLS
race-free ceiling/RLM_TIMEOUT whole-tree). No roles, no fixed order — the
opposite of pi-subagents' scout→planner→reviewer chain.
Read-only-scout RecursivePi harness BUILT + validated 2026-07-09.
harnesses/recursive_pi.py (RecursivePi(MinimalPi), recursive="on") +
minimal_pi.py wiring (recursive knob; ext→GLOBAL path + drop parent
--extension; install() installs pi-recursive, anchored fail-loudly sed
on native-tool.ts, RLM_ env to nvm.sh; RECURSIVE_TEXT read-only-scout
prompt gated to root via RLM_DEPTH). Source finding that made it NOT
drop-in (unlike pi-subagents): children spawn --no-extensions -e
recursive.ts, stripping our registerProvider("llama-local") extension —
fixed by a 2nd -e for our ext file (keeps --no-extensions, no package
double-load). Children are read-only by default (READ_ONLY_TOOLS, or an
un-merged jj workspace) — leaned into: children are fresh-context scouts,
ROOT writes; RLM_JJ=0, RLM_UNSAFE_NO_JJ_WRITE deliberately unset. Host:
rlm_query→child loaded llama-local/qwen3.6-35b-a3b, read a file, returned
it (twice). Container smoke (smoke__…233948): install+patch+env all green,
regex-log PASS 1.0 (no regression)*, 0 rlm_query (model didn't delegate
on a trivial task — the delegation-decay pattern).
Does the model use rlm_query, and does forcing it help? ANSWERED
2026-07-10 across two smoke runs. (a) NO-FORCE (smoke__…004419): 0
rlm_query on all 4 tasks, 4/4 PASS, 10m14s — zero spontaneous delegation
even with the read-only-scout prompt active (delegation-decay shape). (b)
Added a first-action force (prompt mandate AND a followUp nudge were both
ignored on the host, so the block hook blocks non-rlm_query tool calls until
the model delegates once, cap 4). FORCE run (smoke__…010908): delegation
fired on all 4 tasks (mechanism fully validated in-container) but the suite
REGRESSED 4/4→3/4 (0.75) and ran 3.5× slower (35m41s). fix-git flipped
PASS 59s→FAIL 15m53s: the forced READ-ONLY child recursed (depth 1→2,
RLM_MAX_DEPTH=2) asking a grandchild to "run git commands" — read-only
children can't, so the tree hit the 900s RLM_TIMEOUT (~15 min wasted), then
the root botched the merge conflict itself. VERDICT: the model's spontaneous
non-delegation was CORRECT for these action tasks; read-only-scout only pays
off with a large separable READ subtask, which the smoke set lacks. See
smoke__…010908/NOTES.md.
Soft-nudge variant tried too (2026-07-10, smoke__…020217): a
non-blocking followUp reminder sent up to 3× per trial. It DID get delegation
on 3/4 tasks (persistent beat the ignored one-shot) but the suite came out
2/4 (0.50), 49m19s — WORSE than both no-push (4/4, 10m) and force (3/4,
35m). Three-config trend is monotonic: the harder we push rlm_query, the worse
correctness AND wall clock get (0→4→3 tasks delegated vs 4/4→3/4→2/4 pass).
CONCLUSION: on quick/action-oriented smoke tasks the model's spontaneous
non-delegation is correct; recursive read-only-scout delegation only has a
shot on genuinely read-heavy tasks. (regex-log in this run is noise — first
generation stalled, hung to the 25m cap, nudge never fired.)
READ-HEAVY test done 2026-07-10 — the fair shot for recursive, and it
STILL doesn't help. Picked sanitize-git-repo (MB-scale baseline transcript =
genuinely read-heavy; FAILED at baseline so headroom; find-phase is pure
grep/read = the read-only child's ideal shape) and set RLM_MAX_DEPTH=1
(child = leaf, no grandchild → structurally fixes the fix-git timeout).
Three runs, same model/harness (runs …125635, …130242, …130731):
MinimalPi baseline FAIL 0.0 65s, RecursivePi FAIL 0.0 twice (102s +
peg-native parser bug; 128s clean). ALL fail the SAME two verifier tests. Two
findings: (a) the read-only child returned NOTHING useful both runs — root
got only the child's 537-byte stderr banner, then a 0-byte result — so the
fresh-context distillation never materialized and the root re-did the search
itself (child transcripts aren't captured, so empty-child cause is
undiagnosed: parser bug vs no-final-text vs capture gap — this is the
make-or-break gap). (b) the task's real failure is precision (model garbled
the exact secret strings → 7/8 edits failed "could not find exact text",
sed/python fallbacks no-op'd; missed a 2nd HF token buried in a JSON git-diff;
over-replaced a clean _CHUNK="local" line) — a small-model transcription/
judgment weakness that a LOSSY child summary can only worsen, identical with
and without recursive. Verdict: even in its best-fit regime recursive is
neutral-to-harmful for qwen3.6-35b-a3b; keep RecursivePi
validated-but-not-default, do NOT promote. DEFAULT_RLM_MAX_DEPTH is now 1.
IF pursuing recursive further (now lower priority — read-heavy test above
was negative): (1) DONE — RLM_MAX_DEPTH=1 is the default; (2) make the
force OPT-IN / context-size-triggered rather than unconditional; (3) fix the
scout prompt so the child is told it can only read/search (not run
state-changing commands); (4) DONE (read-heavy test, negative — see above);
(5) NEW: diagnose why the read-only child returns an empty result (capture
child transcripts first — pi-recursive only logs start/end to rlm-trace.txt).
Without a usable child summary the whole read-only-scout design is moot, so
this is the gating item.
Forward recursive + recursive_mode + rlm_max_depth/rlm_max_calls/
rlm_timeout as --agent-kwarg in scripts/lib.sh — DONE 2026-07-10 (all
recorded per-trial in config.json; RECURSIVE/RECURSIVE_MODE/RLM_* block
added to config/harness.env; build_site.py groups recursive_mode=decompose
as "pi + recursive (decompose)" and scout as "pi + recursive (scout)").
Deterministic gate scripts/check_recursive_logic.py — DONE 2026-07-10
(covers the DECOMPOSE flavour: solve_subtask.sh orchestration with a stub pi
— isolation, diff capture, patch applies back, .rej on overlap, depth +
call bounds, no-op child — plus DECOMPOSE_TEXT/solve_subtask gating on
decompose/scout/off × root/child) + live e2e case recursive-decompose-merge.
Extend harness_timeouts() in scripts/run_set.sh for *recursive* —
DONE 2026-07-10 (shares the *subagents* branch: drops the per-task cap map,
applies SUBAGENTS_TRIAL_CAP).
Implementer-children variant — SUPERSEDED 2026-07-10 by the DECOMPOSE
harness below, which does acting children properly (fresh-context child with
full tools in an ISOLATED COPY returning a diff the root merges) instead of the
race-prone shared-cwd RLM_UNSAFE_NO_JJ_WRITE hack.
The user's literal RLM vision ("decompose a hard task into subtasks, do each
separately, combine, recursively"), chosen over the read-only scout after the
diagnosis that scout is a lossy, wrong-domain reading of RLM. New
recursive_mode knob: scout (old pi-recursive rlm_query) vs decompose
(new, RecursivePi default). No pi-recursive package in decompose mode — a
solve_subtask tool (extension, harnesses/minimal_pi.py) shells to
scripts/solve_subtask.sh, which snapshots the workdir into a sibling
base/work pair (coreutils tar+cp, no jj), runs a fresh-context child pi with
FULL tools in the copy, captures diff -ruN base work as a -p1-appliable
patch under .rlm/, and returns a summary + short diff preview; the ROOT
reviews + applies the patch + verifies. Children can recurse (depth-bounded).
Single slot / single model (user decision): children serialize, so isolation
buys correctness not speed — outer cap via SUBAGENTS_TRIAL_CAP.
Built + gated 2026-07-10 (deterministic gate green; e2e case added).
Transcript viewer support 2026-07-10: the Pages transcript viewer now
renders solve_subtask children (agent/rlm/subtask-<N>/pi.txt) in the
per-trial "Subtasks" tab + sidebar context switcher, just like pi-subagents
children — each child shows its full turn-by-turn flow, its delegated GOAL
(🎯), a caller→child deep-link on the solve_subtask call, and a child→caller
back-link (matched by goal, since the children have no standardized role
names). transcript.py parse_rlm_children() + build_site.py;
gated by check_transcript_logic.py (check_rlm_children). Appears once a
real decompose benchmark run is committed under runs/.
MECHANISM PROVEN LIVE 2026-07-10 — the full acting-children + diff-merge
loop end-to-end: e2e recursive-decompose-merge 2/2 (real 35b), root
decomposes → TWO isolated children each solve a subtask (11 KB real
transcripts, correctly marked child sessions) → two non-empty patches captured
→ root applies both → both files land. The deterministic gate proves the same
contract with a stub pi.
TWO operational bugs found + fixed live before it passed (both would bite
in-container too): (1) STDIN HANG — the child pi is spawned by the parent
pi's node tool handler with a PIPE stdin; pi --print blocks reading stdin to
EOF, so a pipe the parent never closes hangs the child forever (no GPU, 8m+
with zero activity, times out empty, parent retries → spawn storm). Fixed with
< /dev/null on the child pi invocation (script) + stdio:["ignore",...] on
the spawn (handler). (2) CONCURRENCY — the model emits several solve_subtask
calls in ONE turn; pi runs tool executes concurrently, so on the single slot
two cold children + root starve each other. Fixed with a promise-chain mutex
(__rlmChain) so children run ONE AT A TIME (correctness not speed — the
single-slot decision). NB orphan risk: children run in a new session, so a
parent killed mid-run leaves orphans that can self-recurse — bounded
in-container by trial teardown + the outer cap.
KEY CAVEAT found live: cwd-relative isolation is ESCAPED if the subtask
goal names an ABSOLUTE path into the real workdir — the child then edits the
real dir directly and the captured patch is EMPTY (e2e r1 first attempt: two
0-byte patches, files created in the real dir). Mitigated by steering
DECOMPOSE_TEXT + the child isolation-note to RELATIVE paths, but real
terminal-bench tasks use /app/... absolute paths, so this is the load-bearing
limitation. TRUE isolation needs a mount-namespace / fresh-container sandbox
where the copy sits at the same path (heavy; agent user lacks mount privs) —
the top follow-up if pursuing further.
Author a genuinely DECOMPOSABLE task (independent subtasks, relative
paths) for a fair A/B — the suite is dominated by small tightly-coupled tasks
(the decompose set in config/task_sets.env is only loosely separable
candidates). This is the real "does it help" test.
Run the A/B: AGENT_IMPORT=harnesses.recursive_pi:RecursivePi make run
SET=decompose vs MinimalPi; record peak root context (should stay flat as
subtask count grows) + score. Not yet run.
Route children to a CHEAPER model / parallel slot to give decomposition cost asymmetry — explicitly OUT for now (user chose single slot/model), but it is the only way decompose can win on wall clock.
Literature check on RLM outside long-context benchmarks — DONE, captured in
IDEAS-rlm-evidence.md. Verdict: every rigorous RLM eval is on
built-to-be-long benchmarks (OOLONG/NIAH/BrowseComp); NO peer-reviewed
agentic/SWE-bench/terminal-bench result exists; the independent reproduction
(arXiv 2603.02615) shows RLM HURTS non-long tasks (S-NIAH 100%→70%, Kimi K2
86.6%→60%, latency 3.6s→344s) and RAH (arXiv 2606.13643) reproduces our exact
delegation-decay. Conclusion: don't run recursion broadly here; RLM value is
~purely a function of how context-bound the task is. Keep RecursivePi
validated-but-not-default.
NEW benchmark mode: one-nudge "medic" recovery — DESIGN in
IDEAS-medic-recovery-mode.md. Run agent + verifier as normal; for FAILED
tasks run a fresh-context MEDIC told "the verifier failed" with RLM-style
QUERY access to the failed run's log, then re-verify. Report a SEPARATE
"recoverability" axis ("qwen is within one nudge of model X on task T"), never
folded into the standard baseline. This is the honest RLM application here:
cost is conditional on failure AND it injects NEW ground truth (the fail
signal + selective log access), which is the only regime the evidence says
recursion helps. Rule-relaxed on purpose (agent sees the pass/fail signal) —
always labelled as such.
DEFAULT SIGNAL = FULL failure info (user decision 2026-07-10): the medic gets
the exact verifier/test output + logs pasted back, the realistic
LLM-debugging-round scenario — a bare "doesn't work" is often too weak to
recover even for frontier agents. Signal strength is itself a variable
(full/symptom/minimal curve).
Phased build:
- [x] P0 (start here): the faithful "user debugging the AI's code with the
error in hand" round — MEDIC_SIGNAL=full, MEDIC_LOG_ACCESS=dump,
MEDIC_WORKSPACE=restore (REQUIRED, user decision). BUILT 2026-07-10. All
three pieces shipped: (1) phase-1 WORKSPACE capture — MEDIC_CAPTURE=on (any
harness) makes MinimalPi.exec_as_agent append medic_capture_command() to
tar the workdir end-state to /logs/agent/workspace.tar.gz after pi exits
(mirrors the .pi-subagents/.rlm move, exit-0 discipline, excludes those
droppings; default off = byte-identical). FAILURE-OUTPUT capture uses ROUTE
(a): Harbor already writes verifier/test-stdout.txt, so no phase-1 change —
medic_pass.sh reads it. (2) MedicPi(MinimalPi) (harnesses/medic_pi.py,
medic=on) — the extension before_agent_start MEDIC branch appends the
REPAIR ROUND framing + reads the exact failing-check output from disk (baked
per-task data avoided: fs.readFileSync at run time); MinimalPi.install()
→ _medic_restore() uploads the phase-1 workspace tarball + failure + log
into /tmp/medic and tar -xzf-extracts the workspace over the fresh image
(medic_extract_command()), so the medic opens onto the REAL broken attempt;
the log is staged as a readable file the medic can grep (dump = file, full
in-prompt paste rejected as infeasible for the local context — see below).
(3) scripts/medic_pass.sh + scripts/medic_lib.py — select failed trials
(no-passing-attempt; fail always, err unless MEDIC_INCLUDE_ERRORS=off),
stage per-task inputs keyed by task_name[:32] (matches the in-container
trial-dir key), launch the MedicPi run over just those tasks, then
recoverability() diffs phase-1 vs phase-2 → medic_recovery.json +
baseline/+1 nudge/recoverability table. Knobs in config/harness.env +
forwarded by lib.sh run_model (emitted only when set → non-medic runs
untouched, baseline scoring UNTOUCHED); make medic RUN=<phase1>. Gate:
deterministic scripts/check_medic_logic.py (failed-set selection +
verdict-diff on synthetic runs; real-filesystem capture→restore round-trip
incl. dotfiles/droppings-excluded/overwrite/no-delete; node-probe MEDIC
prompt injection on+off; prepare_inputs staging) — ALL GREEN; +preflight
medic render case. File-oriented start (the fidelity limit — services/DB/
installed-pkgs and agent-DELETED files, which tar-extract won't remove — is
footnoted in the code + docs). MEDIC_WORKSPACE=fresh is the A/B contrast.
NB honest P0 scope note: MEDIC_LOG_ACCESS=dump is realized as "log written
to a bounded readable file the medic greps," NOT a full in-prompt paste (a
failed run's pi.txt is far larger than the local context); the load-bearing
signal is the failure output, which IS injected. The RLM query_log tool is
P1.
- [ ] P1: query_log RLM tool over pi.txt (+ reuse the solve_subtask
llm() primitive) and the MEDIC_LOG_ACCESS=query vs dump A/B — the
actual RLM experiment (does querying the log beat dumping it, on failures?);
plus the MEDIC_SIGNAL full/symptom/minimal strength curve.
- [ ] P2: "Recoverability" board + cross-model matrix in build_site.py
(baseline / nudge-recovered / still-fails per task); banner marks it
rule-relaxed + records the signal level used.
- [x] Knobs: MEDIC on/off, MEDIC_CAPTURE on/off (phase-1), MEDIC_NUDGES=1,
MEDIC_SIGNAL=full|symptom|minimal (default full), MEDIC_LOG_ACCESS=query|
dump|none (default dump; query is P1), MEDIC_WORKSPACE=restore|fresh
(default restore) — ALL wired in config/harness.env → lib.sh run_model →
MinimalPi/MedicPi kwargs, recorded per trial in config.json. Deterministic
gate scripts/check_medic_logic.py covers the driver's failed-set selection
+ verdict-diff + workspace capture/restore + prompt injection (GREEN). A
live e2e case for the medic prompt is the one remaining nice-to-have (the
node-probe already proves the injection deterministically; a real-model e2e
would add end-to-end confidence — LEFT for follow-up).
- [ ] Follow-ups surfaced building P0: (a) commit workspace tarballs are
gitignored (runs/**/*.tar.gz) so a medic pass only works from the LOCAL
phase-1 run dir — fine for one box, but a cross-machine medic needs the
tarball tracked or re-derivable; (b) agent-DELETED files are NOT removed on
restore (tar-extract only adds/overwrites) — a task the base agent broke by
deleting a file would look "already fixed"; low incidence, footnoted; (c)
stateful tasks (running services/DB/installed pkgs) reproduce imperfectly
from a workdir tarball — start on file-oriented tasks; (d) P2 dashboard
"Recoverability" board still to build.
Per-trial transcript viewer on the Pages site — DONE 2026-07-09.
scripts/transcript.py parses committed agent/pi.txt → normalized
conversation + failure-mode flags; build_site.py renders
public/transcript/<job>/<trial>.html (reasoning collapsed, tool
calls/results, nested subagent detail) with a flag banner + per-trial chips +
🔍 view links on run pages. Flags incl. length-stop, leak, LOOP DETECTED /
WRITE REJECTED / bash-timeout tool results, hard-stop, and claimed-success-
but-failed. Output is gitignored public/ = 0 new committed bytes; per-block
cap keeps 407 MB pi.txt → ~85 MB site. Gate: check_transcript_logic.py.
Capture + RENDER full subagent CHILD transcripts (turn-by-turn
reasoning) — DONE 2026-07-09. Capture half (earlier same day): exec_as_agent
MOVES .pi-subagents/ into the committed trial agent/pi-subagents/ dir.
Render half (this change): transcript.parse_child_transcript() parses each
committed agent/pi-subagents/artifacts/<runId>_<agent>_<idx>_transcript.jsonl
(recordType-wrapped events; tool RESULTS arrive as role:toolResult messages,
not tool_execution_end, so results are stitched from those) into the same
render-ready item shape as the parent, reusing _render_message. The
transcript page (render_transcript_page) now has a Subagents tab showing
each child's full transcript collapsed by default (<details>), a parent
subagent call deep-links its child ("▶ full subagent transcript ↓", matched
by runId), and a collapsible sidebar TOC lists the run flow with subagent
calls prominent + a ⚑ flagged-only filter to jump between flagged steps. Gate:
check_transcript_logic.py::check_child_transcript. Size impact small (per-
block cap holds): site 85 MB → 91 MB, biggest page still ~2.1 MB.
Per-subagent sidebar TOC on the transcript page — DONE 2026-07-09 (user
request). The sidebar TOC was a single main-agent list with children shown as
one deep-link row each; now every subagent has its OWN tool-call TOC and the
sidebar switches to whichever agent you are viewing. render_transcript_page
renders a context switcher (.tocswitch / .ctxbtn: Main agent + one row per
child) over a stack of per-agent <nav class='toc ctxtoc' data-toc=...> (only
the active one visible); child TOCs are built by
_flow_and_toc(child, prefix="c<idx>-", toc_target="subs") so their rows
deep-link into the Subagents tab. TX_JS activateCtx() syncs the context on
every trigger (tab click, switcher row, child <details> open, deep-link) and
a context-aware scrollspy (spyScope/refreshSpy) highlights the open child's
rows as you scroll it. Inert with no children (single main TOC). No new
committed bytes (gitignored public/). Verified against a real subagents run
(scout/planner/reviewer each get their own TOC; anchors resolve; non-subagents
pages unchanged); TX_JS node-syntax-checked.
Inline analyst comments on the run page — DONE 2026-07-09 (user request).
runs/<job>/COMMENTS.md lets an agent analysing a run drop observations that
render NEXT TO the exact task/trial on run/<job>.html (companion to
NOTES.md's top-of-page narrative). collect_comments() in build_site.py
parses ## <task_name> → task callout and ### trial <N> / ### #<N> /
### <trial_dir> → per-trial note (rendered as an analystrow under the
matching row; unmatched task sections still render so nothing is dropped; a
💬 N tag flags the page top). Convention documented in CLAUDE.md so agents
pick it up when analysing runs. Gate: scripts/check_comments_logic.py
(parse + inline render placement + escaping). Output is the gitignored site,
the source COMMENTS.md is a small committed text file per run.
Findings from fast2__*__20260707-* (MinimalPi/PlanPi/SubagentsPi on 27b +
35b-a3b). Root-cause analyses written for git-multibranch
(analysis/qwen3.6-35b-a3b/{pi,pi-plan,pi-subagents}/git-multibranch.md).
In ROI order:
Raise fast2 caps in config/task_sets.env: reshard-c4-data=900,
mailman=1200, fix-ocaml-gc=1800. The caps were set at ~3x the MoE
failing times; the sweep produced passing times at or above three caps
(reshard 588s pass vs 300s cap, mailman 1023s pass vs 600s cap, fix-ocaml-gc
1081s pass vs 1200s cap — all 27b, uncapped SubagentsPi runs). The cut 27b
reshard trial already had a working compress.py at cut time.
Re-run 27b MinimalPi + PlanPi on fast2 with the new caps (or
FAST_TIMEOUTS=off) before crediting SubagentsPi's 6/6: SubagentsPi
auto-drops caps, and 4 of the 5 non-subagents 27b fails were cap cuts —
the 3/6 → 4/6 → 6/6 ladder is largely caps-on vs caps-off. PlanPi is
extra-penalized under caps (PHASE 1 planning spends cap budget).
Extend the write guard to ANY length-stopped tool call, not just
write: in message_end, when stopReason=length, record the id of the
LAST toolCall block; in the tool_call hook block that toolCallId whatever
the tool (earlier complete calls in the same message still run). Evidence:
financial-document-processor 35b SubagentsPi (fast2__…145458, ERR) generated
a degenerate ;n;p;n;p… bash command for ~33 min to the 65536 cap, and pi
executed it — it only failed by luck (spawn E2BIG, argv > ARG_MAX). Add an
e2e/check_write_guard case for a length-stopped bash call.
Split stderr out of the pi.txt JSON stream: harbor's base Pi.run()
merges it (`2>&1
Add the llama.cpp DRY sampler to the qwen bases (pending §5.2 item —
fresh evidence): the ;n;p degeneration above is exactly the repetition
pattern DRY kills; it cost 33 min of a 40-min budget in one trial.
SHIPPED 2026-07-09 (llama_cpp_on_my_desktop config.yaml commit 4324a08) on
both qwen bases: mult 0.8 / base 1.75 / allowed-length 4 / last-n -1.
A/B the DRY sampler + validate the command-truncation fix (2026-07-09).
The initial DRY settings (allowed-length 4, last-n -1 = whole context)
penalised the repeated boilerplate of SHELL commands and truncated words
mid-token — the agent emitted git b / git br / git bran for
git branch because the repeated cd /project; git prefix built one long
verbatim match (observed while investigating a git task, e.g. git-multibranch;
this is the "syntax issues from penalising short repeats" the config comment
warned about). Tuned same day (config.yaml, UNCOMMITTED): allowed-length 4->8,
last-n -1->2048 (recent window), + --dry-sequence-breaker for shell/code
syntax (; / = , ) } plus the restored printable defaults : "
*; NB llama.cpp clears the default \n : " * breakers on the first custom
entry and does not unescape, so the newline breaker can't be re-added from the
CLI). Verified live: the fresh llama-server argv carries all three changes
(breakers parse as bare chars). STILL A BENCHMARK VARIABLE — before crediting:
(1) confirm the mid-token truncation is gone on the git task, (2) confirm the
mailman-style verbatim reasoning loop is still crushed, (3) A/B DRY-tuned vs
DRY-off on the fast/fast2 sets. If loops persist raise mult toward 1.0-1.5;
if code still degrades raise allowed-length further.
Duplicate-tool-call loop detector in the extension's tool_call hook —
SHIPPED 2026-07-08 as the LOOP_GUARD knob (default on; loop_guard agent
kwarg, per-trial in config.json; lg:N in make failures). Blocks an
identical single call on its 3rd consecutive occurrence and a repeating
multi-call cycle (the fix-ocaml-gc A/B shape) on its 2nd cycle; blocked
calls are not recorded, so any different call re-allows the original
(re-test after a fix stays possible). Thresholds tuned on 102 passing
trials (zero would-block events at cycle>=2; 3 benign single immediate
repeats). Algorithm adapted from the npm pi-loop-police extension (MIT;
evaluated for adoption wholesale, rejected: its thinking-loop path aborts
generation from message_update — the ctx.compact() print-mode risk class —
and it has no per-trial knob recording). Gates: deterministic
scripts/check_loop_guard_logic.py + live e2e tool-loop-blocked
(3/3 on 2026-07-08; preamble-visible + bash-timeout canaries stayed 3/3).
Original evidence: fix-ocaml-gc 35b SubagentsPi (fast2__…224105, qTiRRpV)
alternated the SAME two bash commands 163 cycles / ~21 min (65% of the
trial) in the ROOT session; no other mechanism covers this shape.
Analysis: analysis/qwen3.6-35b-a3b/pi-subagents/fix-ocaml-gc.md.
Benchmark effect not yet measured — see the follow-up item below.
Measure LOOP_GUARD on real trials — MEASURED 2026-07-08 (night batch,
but TANGLED with strip_thinking=on + keep_tool_results=10 +
subagent_nudges=on, all four knobs flipped at once; see the untangling
item below). Root-session verdict: the guard is quiet and safe — 0 blocks
in 26 of 29 root trials, max 3, and each block produced a pivot; the
verbatim 163-cycle A/B shape did not recur. But the fix-ocaml-gc rerun
(kc83EVe, cancelled by user at 1h28m) shows the failure ADAPTS: the loop
moved into an uncapped planner CHILD (62 min, >=48 blocks in its last 100
events, evaded via small argument variations — offset/limit jitter,
command paraphrases — which per-call hashing tolerates by design), and the
root's own tail loop used the same near-duplicate trick. Updated analysis:
analysis/qwen3.6-35b-a3b/pi-subagents/fix-ocaml-gc.md. Guard stays ON
(harmless where scores are decided), but it converts hard loops into slow
grinding, not progress — see the child-cap and near-duplicate items below.
Untangle the 2026-07-08 night batch — DONE same day, isolation arm (1)
was decisive: 35b fast rerun with STRIP_THINKING=off KEEP_TOOL_RESULTS=0
(loop_guard+nudges still on, fast__qwen3.6-35b-a3b__20260708-115953)
recovered from 3/9 to 7/9 solved — 6 PASS + query-optimize verifier
reward 1.0 behind an AgentTimeoutError (post-session hang, see that item)
— matching the 07-07 baseline exactly; the four tasks the strip run lost
(fix-git, cancel-async-tasks, sparql-university, query-optimize) all came
back. fast2 confirmation arm was cut short by user (long wall times), but
its one scored trial (reshard-c4-data) PASSED. THE CONTEXT STRIP IS THE
CONFIRMED REGRESSOR for 35b; loop guard + nudges are score-neutral at this
sample. Only sanitize-git-repo (0/2 in both 07-08 arms, 2/2 pre-07-08) and
always-failing filter-js-from-html don't fit the strip story — sanitize
may be a real loop_guard/nudge casualty or flake; watch it next batch.
Cap subagent child wall-clock — SHIPPED 2026-07-08 (evening) as the
SUBAGENT_TIMEOUT knob (config/harness.env, default 600s; off/0
disables; kwarg subagent_timeout, per-trial in config.json; inert unless
subagents=on). pi-subagents 0.33.1 already has a run-level timeoutMs
option (schemas.ts; enforced in foreground/execution.ts with SIGTERM +
hard-kill timers, covers foreground AND async/background runs) — the
extension's tool_call hook injects it into subagent RUN calls
(agent/chain/tasks present) that don't pass their own
timeoutMs/maxRuntimeMs; action calls (status/wait/resume) keep their own
timeoutMs semantics ("stop waiting"). On expiry the root gets an explicit
"Subagent timed out after Nms." result; the workflow's retry/self-serve
fallback applies, and the nudges skip timed-out children. 600s = ~3x the
slowest healthy child observed (scout 41–64s, planner <=4 min, reviewer
<=194s); would have cut the 62-min (kc83EVe) and 17-min (7rKKLUm) looping
planners to 10 min each. Registered before the loop guard so the injected
field hashes consistently; children load the same file, so nested
delegation is capped too. Gate: scripts/check_subagent_timeout_logic.py
(injection predicate, model-value wins, actions untouched, off renders).
Original evidence:
analysis/qwen3.6-35b-a3b/pi-subagents/fix-ocaml-gc.md (2026-07-08
section) + query-optimize 7rKKLUm (strip-off arm, so orthogonal to the
context strip). FIRST LIVE FIRING 2026-07-08 16:01 (27b fast run
…153432, query-optimize bkHKdLi): a reviewer child called AFTER the root
had already verified the solution hit the cap at exactly 600000ms; the
root read the partial artifact, gave its final answer, and pi exited
cleanly (exit patch confirmed live in the same trial — verifier started
15s later, no node process left). NB on tasks whose verification
commands are intrinsically slow (query-optimize's original query is
~2-3 min per run) a re-verifying reviewer child may ALWAYS hit the cap
— acceptable (the root had verified already; 10 bounded minutes vs the
old 17-62), but it feeds the nudge-ceremony measurement item.
SUBAGENT_NUDGES wall-clock cost — measure/limit the ceremony. The
nudges achieve their goal (delegation census moved: worker+reviewer now
routinely called vs worker 1/18, reviewer 0/18 pre-nudge) but the staged
ceremony is expensive on the single shared llama-swap slot: reshard-c4-data
PASSED both with and without nudges, at 335s session (07-07, scout+planner
then self-serve) vs 1939s (07-08 strip-off, scout+2×planner+resume+worker+
3×reviewer) — 5.8× for the same verdict; each child is a fresh full
prefill. Nudge dedup itself worked (3 nudges, one per stage) — the model
re-called reviewer twice on its own after the first review came back
clean. NB nudges were ON in both 07-08 arms, so this cost does not
confound the context-strip A/B verdict.
MITIGATIONS SHIPPED 2026-07-08 (evening), measurement still open: (a)
STAGE 5 workflow text + the worker nudge now say ONE reviewer call /
a single review pass is enough; (b) once a reviewer result lands, the
worker nudge is suppressed for the rest of the session (no pushing toward
more review); (c) failed/timed-out children no longer trigger the next
stage's nudge (see the isError item); (d) the SUBAGENT_TIMEOUT cap bounds
each child at 600s. Gates updated in
scripts/check_subagent_nudge_logic.py (reviewer-first probe +
failed-text variants). MEASURE on the next SubagentsPi batch: reviewer
children per trial (expect <=1), wall clock on the fast2 passes vs the
1939s exhibit, and whether delegation depth survives the softer texts;
SUBAGENT_NUDGES=off remains the A/B lever when timing matters.
pi-subagents reports FAILED children as isError=False — nudge-side
fix SHIPPED 2026-07-08 (evening): the stage-nudge hook now sniffs the
result's first text block head (200 chars, lowercased) for the failure
phrasings pi-subagents actually emits ("failed (exit", "failed, exit",
"subagent timed out") and skips the nudge on a match, so a dead planner
no longer gets a "the plan is ready" announcement (kc83EVe shape). Covered
by scripts/check_subagent_nudge_logic.py (three failed-text variants +
the healthy-planner positive case). NB the root MODEL still sees the
failure as a non-error result — upstreaming a real isError flag to
pi-subagents would fix that half too (out of scope here).
Investigate child exit-1-right-after-LOOP-DETECTED (seen twice in
kc83EVe): guard blocks a call → child replies text-only ("let me do X") →
print-mode session ends, exit 1, work lost. Needs preserved child
transcripts (the rm -rf .pi-subagents observability item) to tell
whether the guard's error result causes the bail or just precedes it; if
causal, soften the child-side guard message or have it suggest a concrete
next action.
Qwen-native tool schema experiment — SHIPPED 2026-07-08 (evening) as
the QwenPi harness (harnesses/qwen_pi.py) + the qwen_tools knob.
Background/research unchanged: Qwen3.6's reference harness is Qwen Code
(Gemini CLI fork) whose tools are read_file(path,offset,limit),
write_file(file_path,content),
edit(file_path,old_string,new_string,replace_all), run_shell_command;
pi's builtins are close but differ (read/read_file name; write path vs
file_path; edit path+edits[] vs single old/new_string; bash vs
run_shell_command). arXiv 2510.07248 measures up to +17pp / -80% schema
errors for 3–8B models when the surface matches; our 3B-active MoE even
emitted a raw <function=bash> qwen3_coder call inside thinking
(regex-log 56Avdhy; llama.cpp #20837). IMPLEMENTATION: pi 0.80.2 offers
--no-builtin-tools (hides builtins, keeps extension tools) +
pi.registerTool; the extension dynamically imports pi's own
create{Read,Write,Edit,Bash}ToolDefinition (resolved through pi's jiti
alias — proven host + in-container) and registers the four qwen-named
wrappers, translating params and delegating to the mature builtins (so
read truncation / fuzzy edit / mutation queue are preserved); replace_all
is handled in-wrapper via global string replace. QwenPi passes
--no-builtin-tools; the write-guard + bash-timeout hooks were widened to
write_file / run_shell_command. qwen_tools=off restores builtins for
a clean in-harness A/B. Knob QWEN_TOOLS (empty=class default: off
MinimalPi, on QwenPi); dashboard groups any qwen_tools=on run as
"pi + qwen-tools". Gates: scripts/check_qwen_tools_logic.py (mocked-pkg
param-translation + guard predicates + off render) + host probe (all four
tools live) + in-container confirmation (fix-git GuoEU2t: 42
run_shell_command + 6 write_file calls, delegation working). A/B MEASURING
now: _qwen_ab_20260708.out (fast, both models, MinimalPi control vs
QwenPi). See the follow-up below.
Read the QwenPi fast run — DONE 2026-07-08 (evening; user dropped the
MinimalPi control arms since past baselines exist, so QwenPi-only:
runs/fast__qwen3.6-35b-a3b__20260708-210920 5/9,
runs/fast__qwen3.6-27b__20260708-213445 5/9). VERDICT: qwen-native tools
WORK flawlessly but move the SCORE by nothing on the fast set. Evidence:
(a) 0 schema/unknown-tool errors across all 18 trials — the model used
read_file/write_file/edit/run_shell_command fluently, every wrapper
delegated to the real builtin; (b) 35b matched the closest plain-MinimalPi
baseline exactly (…20260706-113104, 5/9, same pass-set); 27b 5/9 vs an old
flaky 7/9 baseline (query-optimize + regex-log are the known 27b swing
tasks, not a clean delta); (c) the remaining failures are the SAME classes
as MinimalPi — hidden-criterion near-miss (cancel-async, filter-js,
sparql), the runtime-bar lottery (query-optimize), and the wire-format
leak. IMPORTANT: qwen_tools does NOT fix the regex-log format leak (as
predicted) — 27b regex-log still died emitting
<function=run_shell_command> XML INSIDE a thinking block (empty final
message, task untouched, exact 56Avdhy shape); note it now uses the qwen
NAME but the leak is a template/parser bug in the reasoning channel,
orthogonal to the tool surface. WHY no gain: these fast tasks aren't
bottlenecked on tool-surface friction (which the +17pp literature targets)
— the models already used pi's default names with ~0 malformed calls, so
there was little to remove. Keep MinimalPi as the default; QwenPi stays
available. NEXT (optional): the real leverage now is (1) the
empty-final-message recovery item (would recover the format-leak trials on
BOTH harnesses) and (2) a llama.cpp build with the qwen-XML/thinking parser
fix (#20837). Only worth an fast2/suite QwenPi run if a task set that IS
tool-heavy shows malformed-call friction under MinimalPi first.
Read the SubagentsQwenPi overnight run — DONE 2026-07-09 (harness
SHIPPED 2026-07-09, harnesses/subagents_qwen_pi.py). Verdict: works,
neutral-to-mild, KEEP SubagentsPi default. fast SubagentsQwenPi scored
35b 6/9 (…005647), 27b 5/9 (…070108) — above plain QwenPi-root (5/9
both) and MinimalPi (35b 5/9) but BELOW strip-off SubagentsPi 35b (7/9
…115953). cancel-async-tasks PASSED (the amnesia poster child — clean
with strip OFF). Delegation is heavy and healthy (32–99 subagent calls per
task; only filter-js-from-html did 0 and failed), qwen surface fires
cleanly in the subagents stack (270/309 qwen-named tool calls, ~6–8
"unknown tool" strings across 18 trials = negligible). Remaining 35b fails
are OUT of harness scope: query-optimize = runtime-bar miss (SQL correct,
outputs match exactly, but 0.80s vs golden 0.66s, bar ≤1.05× — verifier
ran a 356s perf bench; see item above), sanitize-git-repo + filter-js =
task-logic. No malformed-call friction to remove here, same as the QwenPi
finding. NB the clean SubagentsQwenPi-vs-SubagentsPi A/B on the SAME fast
set was NOT produced — overnight only ran SubagentsPi on fast2, not fast.
Qwen tools promoted to the DEFAULT surface for every harness +
QwenPi/SubagentsQwenPi classes REMOVED — DONE 2026-07-09 (user decision:
the 9-task A/B drop-off was likely noise; standardize the surface).
DEFAULT_QWEN_TOOLS="on"; no_builtin_tools CliFlag + derivation moved onto
MinimalPi; _REMAP_AGENTS_JS + the child-remap step moved into
MinimalPi.install() (fires when subagents=on AND qwen_tools=on). Deleted
harnesses/qwen_pi.py + harnesses/subagents_qwen_pi.py. QWEN_TOOLS=off
restores builtins for root+children (dashboard tags it builtin-tools; qwen
on is now the unlabeled baseline). e2e_replay pins qwen OFF (its verifiers
key on builtin tool names; the guards it tests match both surfaces). All 9
deterministic gates + preflight green; check_qwen_tools_logic imports the
remap from its new home. CLAUDE.md + config/harness.env + lib.sh updated.
Optional: run a QWEN_TOOLS=off vs default A/B on a full set now that
it is a one-flag toggle (no separate harness), if a builtin-vs-qwen number
on the CURRENT extension is ever wanted — low priority, the surface is
already the accepted default.
Empty-final-message recovery — DONE 2026-07-09. Folded into the always-on
runaway-recovery message_end hook in minimal_pi.py (no new knob, same
followUp pattern). Three empty-final sub-shapes, all no-toolCall assistant
turns that ended naturally (stopReason "stop", not "length"): (a) truly empty
text (regex-log 27b 56Avdhy: bash call leaked as qwen chat-template markup
INSIDE thinking → zero text, zero tools, session ended); (b) near-empty text
with a tool-call markup tag left in the thinking (LEAK_MARKUP regex; catches
the near-zero not-exactly-zero case); (c) near-empty text (<40 chars) after a
HUGE thinking block (>8000 chars) — the main-agent REASONING ran away / LOOPED
in the think channel and surfaced neither an answer nor a tool call
(mailman 35b wCQZeFR: final turn 4 chars of text over a 36.8k-char looped
thinking block, shape recurred 3×). A short REAL answer ("Done.") has a small
thinking block so the >8000 gate spares it. Nudge tells the model to stop
re-deriving, state its next step, and make a genuine tool call (cap 2/session,
shared with the runaway cap). FORCE_REVIEW defers all three shapes to recovery
so the two hooks never double-nudge. Gate: check_subagent_nudge_logic.py
(NODE_PROBE_EMPTY_FINAL + NODE_PROBE_LOOPY_THINK). Benchmark effect
unmeasured. Evidence: analysis/qwen3.6-27b/pi-subagents/regex-log.md,
mailman wCQZeFR.
Main-agent REASONING-loop detection (finding 2026-07-09, from mailman
wCQZeFR): the loop guard only hashes tool CALLS, so a model that loops INSIDE
its thinking channel (mailman: 34–37k-char thinking blocks, repeated, ~270
turns) is never caught — it burns the budget and dead-ends in a near-empty
final message. The empty-final recovery above now NUDGES that dead-end
(post-hoc, fail-open), but nothing detects/stops the loop in-generation. Do
NOT adopt pi-loop-police's thinking-loop detector: it aborts generation from a
message_update hook (ctx.abort()), the same print-mode-fatal class as the
ctx.compact() that killed 5/5 trials. Safe levers: (1) the llama.cpp DRY
sampler on the qwen bases (open item above — kills the repetition that feeds
these loops); (2) tighter --reasoning-budget. Consider a high-precision
observability detector (repeated n-gram in thinking → log/nudge, never abort).
Forced end-of-task reviewer + worker dropped from the nudged agents —
DONE 2026-07-09 (user decision). Two coupled changes to the subagent workflow:
(1) WORKER DROPPED — the standing SUBAGENTS prompt STAGE 3 and the planner
stage-nudge now tell the model to IMPLEMENT THE PLAN ITSELF (qwen reliably
does the implementation directly; delegating it just burns the shared
llama-swap slot on a fresh full prefill). Ladder is now scout → planner →
implement-yourself. The turn-6 fallback and the snTexts map no longer name
worker; a self-delegated worker result no-ops. (2) FORCED REVIEWER — new
FORCE_REVIEW knob (config/harness.env, default on; kwarg force_review,
per-trial in config.json; inert unless subagents=on, root only). Since there
is no worker-result to hang the review nudge on, and in-flight stage nudges
landed stale AFTER the answer naming the wrong stage (query-optimize bkHKdLi
got a worker nudge post-victory), this new message_end hook fires at the
session-ending (no-toolCall) assistant message: if no reviewer result has
landed this session, it queues ONE strong followUp requiring a single reviewer
call before the final answer. Cap 2 attempts; skips runaway/empty/oversized/
markup-leak/looping-thought finishing turns (recovery owns those). Gate:
check_subagent_nudge_logic.py (NODE_PROBE_FORCE_REVIEW + _CAP + _OFF,
worker-dropped ladder assertions). Motivation: the reviewer-0/18 census + the
user's transcript reading (nudges ineffective, still naming worker after the
task was done). Benchmark effect unmeasured — MEASURE on the next SubagentsPi
run: does a reviewer call now land ~1×/trial, and does the forced review flip
any hidden-criterion near-miss (sparql, filter-js, cancel-async)? A/B lever
FORCE_REVIEW=off.
Stale-nudge suppression: pi delivers followUp nudges only when the
loop idles, so a stage nudge queued mid-flow can land AFTER the work (and
even after the final answer) — query-optimize bkHKdLi got the STAGE 3
worker nudge at 16:01 that was queued at 15:48, post-victory, costing one
extra turn of polite re-answer. Cheap fix: drop a pending nudge when a
LATER stage's subagent result (or a final-answer stop) has already landed.
Evidence: analysis/qwen3.6-27b/pi-subagents/query-optimize.md.
Runtime-scored tasks are a query-shape lottery — consider a targeted
prompt lever: query-optimize's bar (<=1.05x a hidden golden) is
unobservable, and across 4 trials the verdict tracked only whether the
model landed the filter-first shape (27b 07-06 PASS at 0.99x, 27b 07-08
FAIL at 1.73x with 5/6 tests green, 35b near-miss 1.36x, 35b quadratic
~140x). Where "as efficient as possible" is the single scored axis, a
note to generate 2-3 structurally different candidates and keep the
fastest measured one would turn the coin flip into a search; weigh
against prompt bloat (the preamble lessons) before shipping. Evidence:
analysis/qwen3.6-27b/pi-subagents/query-optimize.md + both 35b
query-optimize analyses.
Loop-guard termination escalation — DONE 2026-07-09. The exact-hash
guard was non-terminating BY DESIGN, so a model that ignored "LOOP DETECTED"
and re-emitted the IDENTICAL call spun to the outer cap (fix-ocaml-gc 35b
SubagentsPi fast2__…033802/4gbxN86: 856 blocks / 685 turns / full 7200s).
Fix: the guard now tracks a CONSECUTIVE-block streak (resets to 0 on any
productive call, so it can only fire when the model does nothing but the
identical blocked call). At ~half the budget it sends ONE strong followUp
nudge; at the full budget (DEFAULT_LOOP_GUARD_STOP_AFTER=10, knob
LOOP_GUARD_STOP_AFTER, 0/off = block-only) it calls process.exit(0) after
a short flush delay, so the trial pipeline exits 0 and Harbor scores the
on-disk state — same interpretable outcome as the fast-fail cut, and NOT
ctx.abort() (print-mode-fatal). 10 is far above the 102-passing-trial max
streak (~1). Children load the same extension so a looping child self-ends
(exit 0 → the subagent tool gets its partial output). Gate:
check_loop_guard_logic.py escalate mode (mocks process.exit; asserts
nudge@2 once, hard-stop exit(0)@4, streak-reset re-arm). Still open:
near-duplicate evasion (kc83EVe offset/limit jitter on read, sed
paraphrases) — normalize read/bash args before hashing; do NOT widen the
exact-hash thresholds blind.
Subagents harnesses outer per-trial cap — DONE 2026-07-09.
harness_timeouts() (run_set.sh) still drops the miscalibrated per-task map
for *subagents*, but instead of leaving them FULLY uncapped (fix-ocaml-gc
SubagentsPi ran 7200s) it now emits a single default=<SUBAGENTS_TRIAL_CAP>
fast-fail entry. Default 1500s (25 min) — set well below the suggested 45
min per user request while staying above the slowest observed subagents PASS
(~17m, 27b regex-log) so passes are not cut; 0/off = uncapped; an explicit
FAST_TIMEOUTS (incl. off) still overrides for every harness. Exits 0 →
verifier scores. The loop-guard hard-stop above kills tight loops far sooner;
this bounds the slow NON-loop runaways (query-optimize 33m, mailman 46m).
Knob SUBAGENTS_TRIAL_CAP in config/harness.env — lower it to go more
aggressive (below ~20m starts risking the ~17m passes).
pi install GitHub dependency REMOVED — DONE 2026-07-09. The
per-trial install used curl raw.githubusercontent.com/nvm-sh/nvm/.../install.sh
| bash + nvm install 22; GitHub rate-limited it in the 2–4am window
("bash: line 1: 429:", "For more on scraping GitHub…"), hard-failing 4 trials
(financial-document-processor ×2, headless-terminal, reshard-c4-data on 35b
fast2). True image-baking is infeasible — every terminal-bench task pins its
OWN docker_image (89 different, mostly third-party alexgshaw/*), so pi is
installed per-container at runtime and there is no shared image to bake into.
Equivalent fix: install() now pulls a prebuilt node tarball straight from
nodejs.org (DEFAULT_NODE_VERSION=v22.20.0, arch-detected, curl --retry
+ a 5× backoff loop) and writes a stub ~/.nvm/nvm.sh that just puts node on
PATH, so the base Pi.run() . ~/.nvm/nvm.sh; pi … contract and every
install step that sources it are unchanged; pi + pi-subagents come from the
npm registry (also retried). GitHub is now entirely off the install path
(node = nodejs.org, pi = npm). NB the pin MUST stay ≥ pi's required node
(0.80.2 needs ≥22.19.0; 22.14.0 installs with an EBADENGINE warning — caught
in host validation). Validated: host throwaway-HOME install (node+pi clean,
exit 0, no EBADENGINE) AND a live in-container SubagentsPi fix-git smoke
(smoke__…123112) — node from nodejs.org, all patches + pi-subagents + child
remap + qwen tools registered, 0 install failures. ensure_pi() (e2e host
install) never used the bootstrap, so unchanged.
Post-session process hang — ROOT-CAUSED + FIXED 2026-07-08 (evening).
Cause: in print mode pi's main() RETURNS after runPrintMode instead of
calling process.exit (main.js), so node exits only when the event loop
drains naturally — ANY live handle left by an extension keeps the process
alive forever (pi-subagents' inline result animation is a setInterval
with no unref; pi's own main.js comment admits the class for package
commands: "we normally prefer process.exit(0) ... so bad extensions
cannot keep one-shot commands alive" — print mode just never got the
hardening). The two confirmed trials (headless-terminal 20260708-073412
+21 min; query-optimize 20260708-115953 +8 min, both solved, both ERRed
as AgentTimeoutError) were SubagentsPi with child state. Fix: anchored
patch on the pinned 0.80.2 dist/main.js after print mode's unique
restoreStdout(); — wait for stdout/stderr writableLength to drain
(mirrors the interactive branch), then process.exit(exitCode)
(patch_main_exit_source in minimal_pi.py; applied by
MinimalPi.install() via fail-loudly sed + anchor-count check, and by
e2e_replay.ensure_pi()). PROVEN LIVE host-side: a deliberately-leaked
interval extension hung unpatched pi past a 25s external kill (exit 124)
and exited 0 immediately once patched, with the full JSON event stream
intact; e2e preamble/subagents/tool-loop cases ALL GREEN through the
patched build. Gate: scripts/check_post_session_exit_patch.py
(patch function, node --check syntax, real-package anchor).
SubagentsPi observability: replace rm -rf .pi-subagents with a move to
/logs/agent/pi-subagents/ — DONE 2026-07-09. MinimalPi.exec_as_agent now
runs, after pi exits, { if [ -d .pi-subagents ]; then mkdir -p /logs/agent &&
rm -rf /logs/agent/pi-subagents && mv .pi-subagents /logs/agent/pi-subagents;
fi; } >/dev/null 2>&1; rm -rf .pi-subagents; true — same verifier hygiene
(cwd left clean, still exits 0) but child scout/planner/reviewer artifacts +
transcripts land in the committed trial agent/pi-subagents/ dir. Sandbox-
verified (cwd clean, transcript captured). Unblocks the child-exit-1 and
fix-ocaml-gc child-loop investigations. Rendering them in the viewer is the
remaining [~] half above.
SubagentsPi delegation depth: root sessions reliably make the scout
call (the "very FIRST action" phrasing) but the later stages decay fast.
Full census of all 18 SubagentsPi trials (2026-07-08): scout 15/18,
planner 11/18, worker 1/18, self-test-via-subagent and reviewer 0/18;
3 trials made ZERO subagent calls (27b git-multibranch in BOTH runs, 35b
mailman). 27b delegates deeper than 35b (scout+planner typical vs
scout-only) and went 6/6 PASS both runs — shallow delegation isn't hurting
fast2 scores, it hurts the EXPERIMENT: the A/B mostly measures
"scout(+planner) + MinimalPi", not the staged workflow. Candidate fixes,
cheapest first: (a) stage-transition reinjection — SHIPPED 2026-07-08 as
the SUBAGENT_NUDGES knob (default on, inert unless subagents=on, root
session only): after each successful subagent result the extension sends
ONE followUp user-message reminder naming the next stage (scout→planner,
planner→worker, worker→self-test+reviewer), plus one stage-1 fallback if
no delegation happened by assistant turn 6; max 4/session, errored calls
excluded, subagent nudge sent stderr markers. Deterministic gate
scripts/check_subagent_nudge_logic.py; live proof = the nudges are
ordinary user messages, so the next SubagentsPi run's pi.txt shows them —
MEASURE: does the census move past worker 1/18 / reviewer 0/18, and does
the nudge ever misfire on a task the model was rightly self-serving?
(b) chain-first prompt — pi-subagents 0.33.1 has a chains tool API the
model can call: { chain: [{agent:"scout", task:"..."}, {agent:"planner",
task:"Plan based on {previous}"}, ...] } — ONE turn-1 tool call (where
compliance is already ~85%) executes scout->planner->worker sequentially
with {previous} output handoff, root keeps STAGE 4 self-test; chain
children get PI_SUBAGENT_CHILD so the extension gating works unchanged
(verified in pi-args.ts). Could ship as a ChainPi variant for a clean
three-way A/B; (c) the Reasoning Context Pruning item below also bears on
this — stripped prior-turn thinking shortens the distance between the
workflow instruction and the current decision.
failure_modes.py hardening: (a) count + warn on unparseable pi.txt JSON
lines instead of skipping silently (the ERR trial's length-stop turn
vanished from the stats); (b) the "exceeded declared 131k window" roll-up
hardcodes 131072 (line ~265) — declared window has been 196608 since the
0.80.2 switch; read context_window from the trial config.json.
Analyze mailman (one-assertion-away everywhere it fails:
test_join_announce_leave_flow, other 2 subtests pass; 27b PlanPi +
SubagentsPi PASSED it, so it is solvable — diff a passing vs failing trial).
"Verify as a stranger" nudge (IDEAS-nudge-detectors.md candidate): git-multibranch 35b failed 3/3 harnesses because the agent installed its own SSH key into the git user's authorized_keys for testing — the verifier (same-container root) then key-authenticates and its password-expect script aborts. The agent verified heavily (V=8–19) but always through its own credential. Detector: agent added credentials/keys/auth config not asked for by the task → nudge to remove them or re-verify the stated interface without them before declaring done.
Run: runs/fast__qwen3.6-35b-a3b__20260706-113104 (5/9, mean 0.56, 31 min
wall; narrative in its NOTES.md). cancel-async-tasks is DONE (deadlocked its
own smoke test — double semaphore acquire + running = list(done) dropping
pending tasks; cut at its 180s cap mid-hang, verdict unaffected). Left:
filter-js-from-html: 2/2 verifier tests failed. Latest fast run wrote a
regex sanitizer and verified only a model-authored happy-path fixture. It did
not round-trip clean HTML through BeautifulSoup; instead it preserved raw
formatting while the verifier expects BeautifulSoup-normalized clean output.
It also missed encoded/entity javascript: variants, data: HTML/SVG
payloads, CSS execution vectors, legacy on... attributes, malformed
script/iframe forms, and namespace script tags. Analysis:
analysis/qwen3.6-35b-a3b/pi/filter-js-from-html.md. Strong claim-vs-evidence datapoint
(IDEAS-nudge-detectors.md detector 3).
query-optimize: 5/6 subtests pass; single fail =
test_compare_golden_vs_solution_runtime. Today's fast run hit
0.7364213460606259x golden speed vs required >=0.9523809523809523x;
the 20260703 suite was closer at 0.8369938486423594x, so this did not
improve. Correctness is exact, but the solution grouped/ranked all
(wordid, synsetid) pairs before filtering to eligible word_stats; the
golden query filters first, then ranks only eligible words. Cap not a
factor (voluntary stop at 2m38s of 600s). Analysis:
analysis/qwen3.6-35b-a3b/pi/query-optimize.md.
sparql-university: 2/3 pass; single fail = test_sparql_query_results
— the H4 control, now twice confirmed unmoved by the preamble. Diff the
returned rows against expected to name the exact wrong/missing assertion;
decided: model semantic-query failure, not spec ambiguity. Latest fast run
wrote valid SPARQL that returned set() because a FILTER EXISTS { SELECT
... } subquery did not correlate with the outer ?professor as intended;
analysis is in analysis/qwen3.6-35b-a3b/pi/sparql-university.md. Stop re-running it as a
hoped-for preamble flip.
pi bash-tool timeout vs fast caps (from the cancel-async-tasks
analysis): the agent's hung self-test consumed its last ~2m20s with no
feedback — pi's bash tool did not time out within 140s, so the model never
got a chance to see and fix the deadlock. Find pi 0.80.2's default bash
timeout (and whether the model can set one per call); evaluate configuring
it below each task's fast cap so a hung verification returns an ERROR to
the model before the cap kills the trial. Testable via an e2e case: prompt
that runs sleep 9999, assert the loop survives with a timeout tool
result. — SHIPPED 2026-07-07. Finding: pi 0.80.2 has NO default bash
timeout (timeout param optional, "no default timeout",
core/tools/bash.js); the model CAN set one per call but rarely does.
Implemented as the opt-in bash_timeout agent kwarg
(BASH_TOOL_TIMEOUT_DEFAULT env knob, empty = off, all 3 harnesses via
MinimalPi): the extension's tool_call hook mutates event.input
(documented mutable) to inject the default into untimed bash calls; a
model-passed positive timeout wins. Hung call → "Command timed out after
Ns" ERROR result → recovery turn. Gates: deterministic
scripts/check_bash_timeout_logic.py + live e2e bash-timeout (3/3 on
2026-07-07). ON BY DEFAULT (90s) since the same day (user decision);
BASH_TOOL_TIMEOUT_DEFAULT=0 (or off) disables for A/B. A system-prompt
note tells the model untimed bash calls default to Ns and to pass a larger
explicit timeout for legitimately long commands (suite compiles/training),
so they aren't cut blind. See the follow-up item below.
Measure the default-on bash timeout (90s + prompt note) on real trials:
does cancel-async-tasks' deadlocked-self-test shape now recover (expect the
model to see "Command timed out after 90 seconds" and fix the deadlock),
and does the 90s cut cost any passes on long-command tasks despite the
prompt note (compile-compcert, train-fasttext, crack-7z-hash run 30-60 min
bash commands by design — watch whether the model preempts with explicit
timeouts)? A/B: BASH_TOOL_TIMEOUT_DEFAULT=0 make fast vs default.
Add run_hidden_tests tool to harness (HIGH PRIORITY). The
cancel-async-tasks SubagentsPi run wrote a functionally correct
implementation that passed 5/6 hidden subtests — the 1 failing test
(test_tasks_cancel_above_max_concurrent) checks for specific stdout
strings ("Cleaned up.") not documented in the task description. The
model's self-test passed 6/6 because it tested the right behaviors
(concurrency, cancellation, cleanup) but had no way to know about the
hidden test's undocumented output expectations. Proposed: register a
run_hidden_tests tool in the pi extension that runs pytest on
/tests/test_outputs.py and returns structured results (test name,
pass/fail, error message). The model can call this to check its work
before declaring done. System-prompt note: "After writing your
implementation, run the hidden test suite (located at /tests/) and fix
any failures before declaring done." Gates: deterministic (mock
pytest output parsed correctly, tool returns structured JSON) + live
e2e (tool runs pytest, model gets results, model fixes an issue).
Risk: model might learn to "game" the tests rather than solving
correctly. Mitigation: the hidden tests check for correct behavior,
not just output strings, so gaming requires correct implementation
anyway. See analysis:
analysis/cancel-async-tasks-subagentspi-20260709.md.
Analyze filter-js-from-html SubagentsPi failure (2026-07-09):
2/6 hidden subtests failed — XSS filter too permissive AND clean-HTML
modification. Analysis: see analysis/qwen3.6-35b-a3b/pi/filter-js-from-html.md.
Analyze sanitize-git-repo SubagentsPi failure (2026-07-09):
2/3 hidden subtests failed — secret removal correct but replacement
doesn't match reference file. Analysis: see
analysis/qwen3.6-35b-a3b/pi/sanitize-git-repo.md.
Evaluate a separate harness knob for stripping prior assistant reasoning
from outbound model context on every turn — SHIPPED 2026-07-08 (as an
experiment, defaults OFF) as TWO knobs in a new context hook registered
BEFORE the overflow pruner (so the pruner's estimate sees stripped sizes):
STRIP_THINKING=on (kwarg strip_thinking) removes thinking blocks from
every replayed assistant message from the first replay onward, making each
request a strict prefix-extension of the previous one (cache-positive per
the 2026-07-05 measurement: the preserved replay broke the KV prefix match
nearly every turn anyway); KEEP_TOOL_RESULTS=N (kwarg
keep_tool_results, user request 2026-07-08 "also strip old tool calls")
stubs tool results (a 200-char head survives) + LARGE toolCall payloads
older than a stepped window — newest N..2N-1 kept full, boundary advances
by N at a time so requests stay byte-identical between advances (one
amortized full re-prefill per N turns, the physical minimum for any scheme
that eventually stubs old content; assistant visible text always kept).
ACTION-TRAIL LESSON (live e2e, first attempt): fully stubbing results +
small toolCall args while thinking is stripped left qwen with no memory of
its own actions (it emits no visible text on tool turns) — it FABRICATED a
completion answer for a command it never ran; keeping small args + result
heads fixed it. Transcripts on disk are unaffected (the hook rewrites
outbound requests only). Gates:
deterministic scripts/check_context_strip_logic.py (transform, stepped
boundary, byte-identical prefix within a step, idempotency, off-render) +
live e2e context-strip-survives (4 tool turns with keep=1 + thinking on).
Measure the context strip on real trials before flipping defaults —
MEASURED 2026-07-08 (night batch: fast+fast2 × both models, SubagentsPi,
STRIP_THINKING=on KEEP_TOOL_RESULTS=10, tangled with
loop_guard+nudges; see the untangling item in the loop-guard section).
VERDICT: keep defaults OFF, do not flip for 35b. The token win is real —
fix-ocaml-gc replayed input dropped 59.8M → 3.35M tokens (~18×) — but 35b
scores COLLAPSED (fast 7/9→3/9: fix-git, cancel-async-tasks,
sanitize-git-repo, sparql-university all PASS→FAIL; fast2 4/6→2/6) while
27b held or improved (6/9→7/9, 6/6→5/6 — dense model tolerates the
amnesia). BOTH predicted failure shapes appeared in the regressed trials:
cancel-async-tasks read /app/run.py SIX times, re-ran near-identical
test one-liners in 3–5x clusters, turns ballooned ~10→55 (agent 35s→6m19s)
and it finally declared done with a fabricated ✅-checklist the verifier
rejected — the lost-track/premature-done shape; fix-ocaml-gc's tail was
slight-variation re-reads of the same 20 lines. Trials also got uniformly
slower (sparql 3m25→13m25, sanitize 2m03→7m02). Caveat: measured only on
SubagentsPi and with 4 knobs at once — the untangling item covers the
clean strip-only A/B, and a milder variant (STRIP_THINKING alone,
KEEP_TOOL_RESULTS=0, or a much larger N) is untested and may keep most of
the cache win without the memory loss.
fast fail timeout flag for benchmarks — SHIPPED 2026-07-06.
Implemented as the fast_timeouts agent kwarg (per-task name=secs mapping
with glob + default support, harnesses/minimal_pi.py) plus make fast
(scripts/run_fast.sh): the smoke set + 5 quick previously-failing tasks,
with per-task caps ON by default (FAST_TIMEOUTS_DEFAULT in
config/harness.env; FAST_TIMEOUTS=... also works on smoke/suite runs).
A capped trial is killed in-container, still VERIFIED and scored normally,
and marked in <trial>/agent/fast-timeout.json (shown as cut@… by
make results, ft:Ns by make failures). Deterministic gate:
scripts/check_fast_timeout_logic.py. Original idea below.
At least for benchmarks for smoke tests, I've noticed that when they pass, they tend to do that quickly. From one run, all passed tests were under 10 minutes, and with the exception of regex-log all were under one minute, which is really fast! This was a result of removing the stupid write guard that needlessly truncated long writes, in effect just confusing the model instead helping.
That's why the idea to implement some kind of flag for early exit when test didn't pass in some given time. Optionally, there might be a mapping that sets this value per test - for the quick ones that might be just 1 or 2 minutes, for longer ones like regex-log that might be 10 minutes. This should greatly reduce the time needed to run some tests, and will still allow for interpretation of the results, even for tests that were finished too soon
First smoke (4/4) + fast (4/9 combined) runs of the 122B MoE on MinimalPi: runs/smoke__qwen3.5-122b-a10b__20260712-135546 and the three-part fast__qwen3.5-122b-a10b__20260712-{141049,143738,144656} (part 1 aborted at 4/9 for memory, parts 2-3 are fresh-restart resume batches).
122b benchmark runs need automatic llama-swap restarts between jobs (or
an upstream leak fix): the 122b llama-server private-memory ratchet burns
~59 GiB MemAvailable in ~10-25 min of benchmark load (measured three times
on 2026-07-12: smoke 5.3Gi→328Mi in 13.5m; fast part 1 59Gi→92Mi + 11Gi
swap in 25m, forcing a mid-run abort; part 2 59Gi→299Mi in 8.6m). A
run_model hook that restarts llama-swap between Harbor jobs when
MODELS contains the 122b (never mid-job) would make unattended
122b sweeps safe; today it needs manual batching. Evidence: the three
fast-run NOTES.md above; kernel OOM kills already hit this box the same
day (llama-swap config.yaml 122b entry).
SHIPPED 2026-07-13: maybe_restart_llama_swap in scripts/lib.sh, called by
run_model before each Harbor job; knob LLAMA_RESTART_MODELS
(config/harness.env, default qwen3.5-122b*, off disables; glob list;
LLAMA_RESTART_CMD overrides the command). Skips when another harbor run
is live; fails the run loudly if the endpoint doesn't come back. Gate:
scripts/check_llama_restart_logic.py (6 stubbed cases, ALL GREEN). NB it
bounds PER-JOB exposure only — a job with >~25 min of solid 122b inference
still ratchets within the job, so long 122b sets must be split into short
jobs (the 2026-07-13 fast2 run used one task per job); a between-TRIAL
reset (or the upstream llama.cpp leak fix) is the remaining piece for
single-job suites.
122b needs its own fast-cap map (or a per-model cap multiplier knob): stock TASK_SET_fast_TIMEOUTS is calibrated on 27b/35b and would have cut a 122b PASS (openssl passed smoke at 114s vs the 120s stock cap; 2m53s in the fast run vs 5x-scaled 600s). The 2026-07-12 runs used an ad-hoc ~5x map (fix-git=600,...,default=1800) — promote it to TASK_SET_fast_TIMEOUTS_122B or a FAST_TIMEOUTS_SCALE knob if 122b runs continue.
regex-log flaked on 122b (smoke PASS 5m44s, fast FAIL 4m26s, same knobs 20 min apart) — run K=3 on regex-log before reading anything into single 122b results on it.
First benchmark run of poolside's Laguna S 2.1 (118B-A8B MoE, UD-Q4_K_S,
partial offload, -c 262144, --reasoning-budget 2000) on MinimalPi, K=1,
uncapped, as four one-task jobs:
runs/smoke__laguna-s-2.1__20260726-{193806,194232,200500,201248}.
3/4 — regex-log / nginx-request-logging / openssl-selfsigned-cert PASS,
fix-git FAIL. Roll-up in the 201248 NOTES.md.
laguna needs its own fast-cap map — roughly 10× the stock 27b/35b
caps, twice the 122b's 5×. Measured agent times vs the 35b on the same
tasks: nginx 7m02s vs 15s (28×), openssl 4m53s vs 13s (22×),
fix-git 3m44s vs 18s (12×). Sustained decode ~17–19 tok/s (vs ~165
tok/s for the 35b at f16 KV) plus ~2–3× more output tokens per task.
Stock TASK_SET_fast_TIMEOUTS would cut EVERY laguna pass observed
(openssl cap 120s vs 293s needed; regex-log cap 600s vs 1290s needed). This
is the same problem already open for the 122b two items up — worth solving
once as a FAST_TIMEOUTS_SCALE knob (or per-model cap maps) rather
than a second hand-written map.
Check whether laguna has the qwen3.5-122b RAM ratchet before running it
unattended (it is the other ~45 GiB shmem-expert partial-offload model, and
the box idles at only ~10 GiB MemAvailable with it loaded — a 122b-style
~59 GiB burn would OOM-kill within minutes).
Resolved 2026-07-26: it does not ratchet. MemAvailable sampled every 30s
across the full ~40 min four-job batch stayed flat at 10.4 → 10.1 → 10.1 →
10.2 → 10.2 GiB with swap untouched. The batch ran with
LLAMA_RESTART_MODELS="qwen3.5-122b* laguna*" as insurance; on this evidence
laguna does NOT need the 122b's one-task-per-job batching. Not yet confirmed
over a multi-hour session — leave the glob override in place for unattended
runs until a fast/suite run says otherwise, but do not add laguna* to the
config/harness.env default on this evidence alone.
Decide whether laguna joins the standard MODELS line. Untouched so far
(this run passed MODELS=laguna-s-2.1 per-invocation); at 12–28× the 35b
wall clock a full suite is expensive, so a fast run with scaled caps should
come first.
Run the other two harnesses (PlanPi, SubagentsPi) on laguna smoke — this
first light pinned MinimalPi. NB SubagentsPi's shared-slot parent+child wall
clock on a ~19 tok/s model will be brutal; SUBAGENTS_TRIAL_CAP (1500s
default) is likely far too tight here and would need scaling with the same
factor as the item above.
New near-miss shape: process-verified but content-wrong (fix-git,
runs/smoke__laguna-s-2.1__20260726-193806). laguna resolved a cherry-pick
conflict by HAND-WRITING the merged _includes/about.md with write_file,
then declared done after git status + git log --graph both looked clean.
The graded criterion is a content hash against
/app/resources/patch_files/about.md, which it never read; its retyped
version carries a trailing space and no trailing newline. The 35b makes the
IDENTICAL mistake twice on this task (second attempt visibly mangled:
StanfordCS, GoogleResearch) and escapes only by running
git show <rev>:<path> | cat -A and then
git show <rev>:<path> > <path> — restoring from the authoritative object
instead of retyping. Note this is NOT the classic false-success shape and
preamble RULE 2 as worded is satisfied by what laguna did: it verified
with real commands, just the PROCESS rather than the CONTENT. Candidate
responses, cheapest first: (a) extend RULE 2 to name the distinction
("verify the artifact the task grades, not the process that produced it");
(b) a Tier-1 nudge detector in the IDEAS-nudge-detectors.md family — a
write_file whose content reproduces text available verbatim from a file or
VCS object is a copy that should have been a copy. (b) is the sharper fix
but needs a way to detect "available verbatim", so start with (a).
Wire path is clean and needs no laguna-specific harness work — recorded
so the next agent does not re-derive it. Across all four trials: qwen-named
tools registered and used fluently, 0 schema errors, 0 unknown-tool
errors, 0 wire-format leaks, 0 length-stops, 0 guard fires (loop guard,
write guard, bash timeout, recovery all armed and silent). Reasoning splits
correctly into reasoning_content (--reasoning-format deepseek, same
bundle as the qwen bases, so thinkingFormat: "qwen-chat-template" is the
right contract). The config's "empty content" gotcha never appeared —
MAX_TOKENS=65536 is far above the ~3000 floor it needs.
fast2 on the 122b, run as SEVEN one-task jobs (between-jobs auto-restart bounds the RAM ratchet per task): git-multibranch PASS 8m08s and mailman PASS 22m49s — the first fast2 tasks EVER passed on this box (35b/27b: 0/6 always); reshard-c4-data FAIL 12m37s, financial-document-processor + fix-ocaml-gc FAIL cut@30m; headless-terminal FAIL 10m36s clean (rerun fast2__…180346 after the cap-collision ERR). Caps ×3 stock clamped to 1800s. Runs: fast2__qwen3.5-122b-a10b__20260712-{153513,155025, 155950,162421,165943,173102,180346} + per-run NOTES.md.
Fast-fail caps must sit STRICTLY BELOW the task's declared timeout_sec × AGENT_TIMEOUT_MULT or Harbor's own AgentTimeoutError wins the race and the trial ERRORs UNVERIFIED instead of being cut+scored: headless-terminal (declared 900 × 2.0 = 1800 = the cap used) died exactly this way in fast2__…162421 and needed a rerun at 1700. Add a preflight check (run_set.sh or preflight_extension.py already parses the map) that warns/fails when any cap entry >= declared×mult for a task in the run.
mailman's PASS needed 1369s agent — the stock fast2 cap (600s) would have cut it. Strengthens the per-model cap map / FAST_TIMEOUTS_SCALE item above: for 122b use ~3× stock, clamped ≤1800s (the RAM-ratchet single-job budget), and always < declared×mult (previous item).
financial-document-processor + fix-ocaml-gc both worked the full 30m cut without converging or looping (0 loop-guard events) — if anyone wants a 122b verdict on them, they need uncapped runs (ceilings 2400s/7200s) with the RAM caveat that >~33 min of solid 122b inference in ONE job outruns the swap headroom; wait for the between-TRIAL reset or accept the risk.
Two in-depth trial analyses (COMMENTS.md in fast__qwen3.5-122b-a10b__20260712- {183411,184152}) while the 8-task sequence was still running:
SEQUENCE VERDICT (all 8 jobs done, 2026-07-13; runs fast__…{183411,184152, 191135,193509} + fast2__…{195554,202149,204249,205638}, NOTES.md in each): 3/8 flips vs MinimalPi — reshard-c4-data PASS 23m10s, headless-terminal PASS 19m38s, financial-document-processor PASS 12m25s — all fast2 multi-step tasks; 0/4 on the fast hidden-criterion tasks. 122b cumulative: fast2 now 5/6 solved by some 122b config (only fix-ocaml-gc open). MECHANISM CAVEAT from per-trial transcript checks: ZERO scout/planner delegation in all 8 trials; 2 of 3 passes used only the FORCED reviewer (verified, changed nothing), headless passed with zero children — the active ingredient is the staged workflow SYSTEM PROMPT (persistence/ discipline) and/or K=1 variance, NOT delegation.
pi-subagents drops a child's final report when it is only an ```acceptance-report fenced block: reviewer b5ff6e05 (sparql-university 8yQBix3) worked 437s, edited the file, exited 0 with a proper report — output.md EMPTY, parent got "(no output)" and re-ran a second reviewer (237s more on the shared slot). Reproduce on the artifact transcript, then report upstream or add a harness-side fallback (use the last non-empty text block when the extracted output is empty).
Preamble RULE 2's "use only the standard library and already-installed tools" suppresses the ONLY self-verification path on tasks with no preinstalled checker: sparql-university has no SPARQL engine in-container, the container HAS network, the verifier itself installs rdflib — and both the MinimalPi and SubagentsPi 122b trials stopped at "No SPARQL engine found" without attempting an install. A/B a softened wording ("prefer already-installed tools; if verification truly requires a package, try installing it before skipping verification") against the dependency-hallucination fails the rule was written for.
122b + subagents cost model: children are expensive twice on this model — cold prefill on the shared slot AND budget-saturated thinking (a reviewer child burned one 31.4k-char thinking turn ≈ 5 min decode; children 136–437s each; ~16 of sparql's 28 min went to 5 children). The sparql reviewer DID catch a real syntax blocker (result moved from wrong-professor to right-professor/wrong-aggregation), so review adds value — consider a 122b profile with force_review on but stage nudges off (skip scout/planner, keep the one reviewer), which is what the model does naturally anyway.
Replicate the 3 fast2 flips at K=3 to split workflow-prompt effect from variance: arms = plain MinimalPi vs SubagentsPi (or better, a prompt-only arm: subagents prompt + FORCE_REVIEW without the pi-subagents package) on reshard-c4-data + headless-terminal + financial-document-processor. headless is the highest-variance suspect (MinimalPi ERR@30m / FAIL 10m36s / SubagentsPi PASS 19m38s with zero children).
Four log-driven fixes from the 20260718 analysis (dash-prompt prefix, write- guard storm hard-stop, revived rec: counter, runaway-text trim) shipped and re-ran the full suite, same config (MinimalPi, 35b, K=1, MULT=2.0): 40/89 = 44.9%, +5 net vs 35/89. 14 gained / 9 lost (big K=1 churn again). Attribution (run NOTES.md): fix #1 is a CLEAN win — pytorch-model-recovery went from never-ran (instant "Unknown option: -") to PASS reward 1.0 (first-ever pass, 35b all-time union +1); fix #4's target torch-tensor-parallelism FAIL→PASS with the trim active; fix #3's rec: counter now populates; fix #2's storm hard-stop did NOT need to fire (the wg:2547 storm didn't recur this run — stays a backstop). No regression is attributable to the changes: the 9 losses are K=1 variance (the trim never touched the quick-task losses — max visible text 550–734 chars << 8000 threshold), and the 2 killed-by-signal errors (install-windows-3.11 exit 137, qemu-alpine-ssh exit 143) are qemu VMs OOM-killed/torn-down in the 1–2 core / 2 GB containers, not the exit-0 hard-stop (0 markers) or the pi V8 OOM (134).
Add a per-trim stderr marker to the runaway-text trim hook (fix #4).
Right now it only logs "runaway-text trim active" on registration, so there's
no way to count how often it actually trimmed a block across a run (torch-
tensor-parallelism flipped to PASS but the trim engagement is unmeasured).
Emit e.g. "minimal-pi: trimmed runaway text block (N chars -> head+tail)" per
trim and add a trim:N column to failure_modes.py + a transcript flag, like
pr:/rec:/lg:. Cheap; makes the fix's effect measurable at K≥3.
K≥3 replication of the two fix-attributable flips (pytorch-model-recovery is deterministic now — will stay PASS; torch-tensor-parallelism is the one to replicate to split trim-effect from variance). Same arms as the standing workflow-flip replication item.
Buckets over the 49 non-passes (34 FAIL + 15 ERR), from failure_modes.py +
verifier stdout + pi.txt banners:
- 15 ERR = all timeouts/signals, none harness-flippable. 7 are
environment/CPU-bound-by-design in the 1–2 core / 2 GB containers
(caffe-cifar-10, extract-moves-from-video, make-doom-for-mips,
mcmc-sampling-stan, train-fasttext = long compiles/training/video; plus
install-windows-3.11 exit 137 OOM + qemu-alpine-ssh exit 143 = qemu VMs).
8 are runaway/thrash that burn the whole budget without converging
(schemelike 457k out, feal-linear 309k, code-from-image 267k, gpt2-codegolf
213k, write-compressor 205k, sqlite-db-truncate 133k, adaptive-rejection
19.8M input, mteb-leaderboard 91.7M input / 947 calls / pr:85 over 2h).
Ending these cleanly would reclaim wall-clock but not score (the model was
looping/thrashing, not progressing).
- 34 FAIL: 30/48 self-declared done (clean stop), 25 already ran a check.
Verified spot-checks: fix-git resolved a merge conflict wrong, kv-store-grpc
used the wrong proto field, mteb-retrieve extracted the wrong dataset,
overfull-hbox over-edited beyond synonyms.txt, financial-document-processor
mis-summed line items. CORRECTION 2026-07-26 (user): do NOT call these
"capability" — every one is a task the model has PASSED at least once, and
terminal-bench needs ALL subtests green, so a single pass proves whole-task
capability. These are RELIABILITY/VARIANCE failures (a bad sampled trajectory:
a reasoning-path slip, an inconsistent approach/spec-follow, thrash, or
formatting), not a ceiling. Only the 31 never-passed tasks are the capability
wall. Proof: count-dataset-tokens passed twice (79586) and failed twice (79566,
off by 20) — same task/model/config. 13 are one assertion away
(build-cython-ext 10/11, cancel-async-tasks 5/6, financial-document-processor
6/7, mteb-retrieve, sparql-university, path-tracing[-reverse], query-optimize,
build-pmars, fix-git, overfull-hbox, winning-avg-corewars, train-fasttext).
- Context cliff still dominant: pass rate 68% (<50k) → 30% (50–131k) → 19%
(131–200k). Keeping context small is the strongest predictor; runaways/loops
are what push tasks over it.
- Ruled out this run: the 90s bash default is NOT cutting legit work — the
model passes explicit timeouts (caffe-cifar-10 used 120/180/300s; its cuts
were a genuinely slow 53 KB/s BVLC/caffe clone, ~45 min). No WRITE GUARD
hard-stop fired (storm didn't recur). No hard 262k 400s.
[highest ROI] K≥3 full-suite (or near-miss-cluster) replication. K=1 churn is again large (14 gained / 9 lost) and the author attributes the 9 losses to variance. 25 near-misses passed ≥1 hidden subtest, 13 are one assertion away — several of these are ~50/50 tasks that K=1 randomly scores FAIL. K≥3 de-noises the board and captures the model's true solve rate; it is the single most defensible way to move the reported number and separate capability from variance. Pairs with the loop/prune circuit-breaker below, which buys back the wall-clock K≥3 costs.
Rate/cumulative loop + prune circuit-breaker (wall-clock, enables K≥3).
The loop-guard hard-stop needs 10 consecutive identical blocks; the
oscillating loopers dodge it by switching which call they repeat (regex-chess
churned to 137+ calls / 1h21m with 14 blocks before the streak finally hit 10;
gcode/winning-avg similar). Add a session-wide backstop independent of the
consecutive streak: hard-stop (exit 0, same mechanism) when total loop-blocks
≥ ~20 (safely above the highest passing trial feal-differential lg:10) OR
prune count ≥ ~40 (max legit passing is build-pov-ray pr:10 over a real 3h
build; mteb-leaderboard hit pr:85 thrashing for 2h). Won't flip these to PASS,
but reclaims several hours off the 32h and converts the mteb-leaderboard-style
2h thrash into an early clean exit. Add check_*_logic.py gate + a stderr
marker; verify the ≥20 threshold against a K≥3 pass-set scan first.
[lower confidence] Reactive verify-gate nudge for wrong-target/partial near-misses. A subset of the 30 self-declared-done fails did the wrong or partial thing and didn't notice (fix-git applied 1 of 2 needed states, mteb-retrieve wrong dataset, overfull-hbox over-edited). Before accepting a no-toolCall "done" turn, fire ONE followUp that makes the model re-read the literal task statement and confirm it did exactly what was asked — no more, no less — with a real command. This is the claim-vs-evidence detector from IDEAS-nudge-detectors.md. Speculative (may flip 1–2, may cost tokens); gate it as an A/B knob (default off) and measure at K≥3. Most near-misses are genuine judgment errors a nudge can't fix (the model has no ground truth), so keep expectations low.
Resource-lifted arm RUN 2026-07-21 (CPU_OVERRIDE=24, MEM_OVERRIDE_MB=32768)
on the 7 env/CPU-bound tasks — 3/7 flipped to PASS.
Added a CPU_OVERRIDE knob (config/harness.env → lib.sh --override-cpus →
Makefile _ENV, mirroring MEM_OVERRIDE_MB) + a resource task set
(config/task_sets.env, kept off the board via BOARD_SET_EXCLUDE). Ran
runs/resource__qwen3.6-35b-a3b__20260721-194555 (MinimalPi, K=1, uncapped).
Result: caffe-cifar-10 (ERR40m→PASS17m), mcmc-sampling-stan (ERR60m→PASS11m),
qemu-alpine-ssh (ERR-died2m→PASS16m) were genuine CPU/RAM walls, now solved
→ effective suite 40→43/89 (48.3%) if applied broadly (NON-STANDARD arm, not
leaderboard-comparable; K=1 caveat). The other 4 confirm resources only remove
the wall: install-windows-3.11 OOM fixed (exit137→clean FAIL 2/4 tests) but
capability-bound on the GUI install; make-doom-for-mips never resource-bound
(churned 24M input tokens, no loops/cuts, didn't converge — hard cross-compile);
train-fasttext genuine long compute (each fasttext run >600s even at 24
threads, model ran several + self-cut at 600s); extract-moves-from-video
video processing genuinely ~56m, completes but wrong output. NB the 90s bash
default was NOT a factor (train-fasttext cuts were the model's own 600s).
Full-suite resource-lifted arm RUN 2026-07-24/25 (suite__qwen3.6-35b-a3b__20260724-160500, CPU_OVERRIDE=24 MEM_OVERRIDE_MB=32768): 31/89 = 34.8%, a LOW K=1 variance draw, NOT resource harm. Three K=1 MinimalPi points now: 35 (0718) / 40 (0720) / 31 (0724) → mean ~35, ±~5. The override delivered its intended gains (mcmc-sampling-stan + qemu-alpine-ssh ERR→PASS) but the ~2-3 task resource signal is swamped by ±~8 task K=1 variance. Confirmed NOT the override: regressions dominated by resource-irrelevant tasks (nginx, vulnerable-secret, bn-fit-modify) failing on task logic; caffe-cifar-10 flipped PASS(probe)→ERR(here) with identical override; wall-clock 30h33m < 32h09m baseline (no server starvation). LESSON: K=1 variance is too large (±~8) to measure a 3-task resource effect — use the isolated 7-task probe or K≥3, not a single full suite. See the run NOTES.md.
Reasoning-THRASH is the dominant K=1 variance driver + top floor-raiser
(2026-07-25 deep log analysis of cobol-modernization). The worst failures are
NOT loops/repetition — they are semantic self-doubt spirals: the model reworded-
re-derives the same analysis and does multi-digit arithmetic in the think channel
it keeps getting wrong (cobol: "Wait,"×154 "Hmm,"×106 "let me count"×29, 301k
tokens vs 91k at the PASS baseline, 3× thinking). NONE of the 35b anti-loop
measures fire on it (verified 0 loop-guard blocks / 0 recovery nudges / 0 trims /
0 hard-stops): DRY needs verbatim n-grams (thinking is reworded + punctuation
sequence-breakers neuter it in prose), reasoning-budget 8000 is per-turn (blocks
were ~3800 tok), loop-guard needs exact-repeat tool calls (calls varied),
recovery needs no-tool dead-ends (kept calling tools, mid-size text in the 40-8000
char gap). Baseline PASSES the same task by staying EMPIRICAL (147 tool calls, low
thinking). Proposed fixes (design only, not built): (a) E2BIG guard — oversized
inline command args die spawn E2BIG (hit cobol ×4 + largest-eigenval); a
tool_call hook that spills >64KB commands to a file or returns an instructive
error (deterministic win). (b) reasoning-thrash proxy between pi and llama.cpp
— detect backtrack-marker density / cross-turn thinking similarity / cumulative
thinking-without-new-tool-result, then steer ("stop reasoning, write a script that
computes+diffs, trust it over mental math"), dynamically force enable_thinking=off
for a turn, and/or selectively prune the confused thinking from replayed context.
Proxy is the right layer: can't crash pi's print loop (unlike ctx.abort/compact),
can toggle thinking per-request, can do heavy similarity analysis. See the
antidoom evaluation below.
Two cheap variance fixes SHIPPED 2026-07-26 (E2BIG guard + preamble RULE 3).
Both default-on, both from the flippy-task log dig.
(1) E2BIG guard (E2BIG_GUARD knob, config/harness.env + lib.sh + kwarg
e2big_guard, per-trial in config.json; threshold DEFAULT_E2BIG_MAX_CHARS
100000): a tool_call hook blocks a run_shell_command/bash whose command
exceeds ~100 KB (below the 131072-byte kernel MAX_ARG_STRLEN) and returns an
instructive result telling the model to write the big content to a file and run
the file. Fixes the spawn E2BIG dead-end that ran NOTHING (cobol ×4,
largest-eigenval in suite__20260724). Registered before the loop guard so a
blocked call is not hashed. Deterministic gate scripts/check_e2big_logic.py
(block predicate + reason + custom threshold + off render + non-shell tools
untouched) GREEN; composes cleanly with bash-timeout + loop-guard gates; render
clean, preflight OK. Stderr marker E2BIG guard blocked over-limit ... / E2BIG
guard active. NB avoid the literal phrase "blocked oversized" in the rendered
extension — the write-guard gate forbids it as a stale-code sentinel (cost two
rewords here).
(2) Preamble RULE 3 - exact answer files (rides on PREAMBLE=on, no new knob):
write the exact required value to answer files with no surrounding quotes, extra
trailing newline, trailing spaces, or stray NUL/binary bytes, then read back and
confirm. Targets the correct-answer-wrong-bytes class (vulnerable-secret wrote
the right flag + a trailing \x00 → hard 0 under byte-exact grading). Honest
caveat: like RULE 1 (chunked writes, "zero effect" per 2026-07-05), a mechanical
preamble rule the model may ignore — the \x00 came from the model's own program
output (not write_file), so it can't be caught mechanically; this is the cheapest
available lever, expected weak. VALIDATION IN FLIGHT:
runs/smoke__qwen3.6-35b-a3b__20260726-002616 (K=3 on vulnerable-secret,
largest-eigenval, cobol-modernization, fixes on). NB the E2BIG guard only engages
when the model emits a >100 KB command (stochastic), so the deterministic gate is
the real proof; the live run is a smoke test + a chance to see it fire in-task.
Cumulative-output "commit/stop" nudge SHIPPED 2026-07-26 (OUTPUT_NUDGE
knob, default OFF; threshold DEFAULT_OUTPUT_NUDGE_AT 130000, cap 2). A
message_end hook accumulates assistant usage.output and fires ONE followUp
("verify+finish if the deliverable is written; stop re-deriving, act on what you
established") when cumulative output crosses the threshold, re-arming one budget
later. Nudge only, never a hard stop. Wired config/harness.env + Makefile _ENV
(added E2BIG_GUARD + OUTPUT_NUDGE forwarding — command-line override was silently
no-op'ing without it) + lib.sh + kwarg output_nudge (per-trial config.json).
Gate scripts/check_output_nudge_logic.py GREEN (fires once past threshold with
followUp+verify text, re-arms, caps at 2, ignores no-usage turns, off render
silent); all 16 gates green except the pre-existing unrelated check_comments.
A/B DONE 2026-07-26 — INCONCLUSIVE: the nudge NEVER FIRED (0/9 both arms). None
of the trials thrashed to 130k output this sample (cobol passed in ~4-8min, not
its occasional 301k spiral), so off 5/9 == on 5/9 is pure variance with an inert
nudge (smoke__20260726-025257 off / -034536 on). Lesson: the thrash the nudge
targets is itself stochastic and a 3-task K=3 sample missed it — the real
evaluation is a full suite (~37% of fail trials exceed 130k). Added an
OUTPUT_NUDGE_AT knob (config.harness.env + lib.sh + Makefile _ENV) and ran a
FORCED-FIRE behavioral check (threshold 20k on cobol+crack-7z, K=3) to confirm
the nudge fires in a real session and does NOT derail an otherwise-healthy solve.
FORCED-FIRE RESULT (smoke__20260726-042939, threshold 20k): cobol 3/3 PASS with
the nudge firing in 2/3 trials (fired at 21199 output tokens) -- firing it
mid-solve did NOT derail the healthy task; crack-7z 3/3 PASS, nudge never fired
(<20k output). So: mechanism proven (gate + live fire), SAFE (no derail even
force-fired below production threshold), but BENEFIT still unmeasured (no trial
thrashed to threshold in either test -- thrash is stochastic/rare per run).
VERDICT: default stays OFF; enable OUTPUT_NUDGE=on on the NEXT FULL SUITE (real
130k threshold) to measure recovery where ~37% of fail trials cross 130k. Safety
is strong enough that defaulting-on is also defensible (fires only on runaways
that are already failing -> can help or be neutral, not hurt healthy passes).
Original proposal + data below.
[superseded by the shipped item above] Cumulative-output "commit/stop" nudge — proposal + data. Data: across the 4 full suites passing trials generate median 13k output tokens (p90 88k), failing trials median 104k (p90 305k) — one of the cleanest pass/fail separators. crack-7z-hash actually SOLVED it (honeybear on disk at /app/solution.txt) then kept re-cat-ing and hit AgentTimeoutError = "solved but didn't stop"; cobol ruminated to 301k. Mechanism: a message_end hook tracks cumulative output tokens and at ~130k sends ONE followUp ("if your deliverable is written, verify once and finish; if you're re-deriving, act on what you know") — nudge only, cap ~2, NO hard stop. At 130k it touches only ~5 legit high-output passes (build-pov-ray 3h build, feal-differential brute-force, regex-log) and since it's non-terminating they continue; fires on ~40% of fails. Distinct from every existing guard (loop=tool-call repeats, recovery=no-tool turns, trim=visible text) — none track cumulative OUTPUT across turns. ~30 lines reusing the recovery-hook sendUserMessage machinery; gate + e2e testable. Build this BEFORE the heavier thrash-proxy.
Interpreter-matched dependency install (preamble reword) — PROPOSED.
openssl-selfsigned-cert: the agent ran pip install cryptography 4-6x yet the
verifier's python /app/check_cert.py still hit ModuleNotFoundError → the dep went
to a different interpreter than the checker's python (or the install silently
failed unnoticed). Plus the known sparql-university rdflib case where RULE 2's
"use only already-installed tools" wording DISCOURAGED a needed install. Fix: (a)
soften RULE 2 so it doesn't suppress installs a task legitimately needs (network is
available); (b) add a note: when a task/checker runs python, install for that
interpreter (python -m pip install X) and confirm python -c 'import X' before
declaring done. Cheap; openssl root-cause is a hypothesis but the interpreter
mismatch is a real recurring local-model gotcha.
Output-nudge threshold is too HIGH to rescue timeouts — TUNE it (2026-07-26 flippy run smoke__20260726-100656, 23/39 with all 3 fixes on). 23/39 vs 19.8 expected (sum of historical K=1 rates) = +3.2, WITHIN K=1 noise -> not evidence the fixes helped. The fixes barely engaged: nudge fired in 4 tasks (break-filter, overfull-hbox, regex-log, tune-mjcf) and rescued 0 -- all crossed 130k output near the 20-min cap, so it fired with no runway; E2BIG fired 0x (no >100KB command this run). ACTIONABLE: the 130k token threshold isn't reached until ~18-20min, too late on a timing-out trial. Change the trigger to earlier/time-based (e.g. past ~60% of the agent budget without finishing, or ~90k tokens) so the nudge has runway. Also: my 1200s test cap COST a pass -- regex-log (hist 80%, needs ~21min) was cut at 20min, and qemu-startup stalled to the cap; uncapped this was ~24/39. Real measurement still needs a full-suite (uncapped) A/B.
Reactive deliverable-existence gate (2026-07-26 never-passed log dig) —
PROPOSED, strongest new idea. No mechanism checks whether the task's required
output file exists before the model declares done, yet a big cluster fails there:
path-tracing (/app/image.c — wrote it 117x in a thrash loop but no working file
at verify), password-recovery, sam-cell-seg (test_output.csv), polyglot-rust-c
(main), + flippy cobol (program.py), crack-7z (solution.txt). Mechanism: parse
the task instruction for explicit output paths (/app/
Reactive dependency/interpreter nudge (2026-07-26) — PROPOSED. 6 tasks
fail at verify with ModuleNotFoundError/command-not-found (feal-differential,
openssl, pypi-server, pytorch-model-recovery, torch-pipeline/tensor-parallelism).
The model didn't install, installed for the wrong interpreter (openssl: installed
cryptography but the checker's python couldn't import it), or RULE 2 discouraged
it. Beyond the RULE 2 reword already proposed, add a reactive nudge: on a tool
result containing ModuleNotFoundError/command-not-found, inject "install the
missing package with the SAME interpreter the task uses (python -m pip install X);
the container has network" then re-run. External ground truth, fires on the error.
Large-file write deadlock (2026-07-26) — PROPOSED, medium. Tasks needing a big deliverable file (path-tracing raytracer image.c) trigger a write->truncate->reject->retry thrash: the model writes the whole file in one turn, it exceeds the output limit, the write guard correctly rejects the truncated write (11x in path-tracing), and the model ignores RULE 1's chunk advice and retries the same way. Idea: make the write-guard rejection escalate/actionable -- after the 2nd truncated write to the SAME file, force an append protocol (accept the partial + "continue with append") instead of just re-blocking. Targets path-tracing + torch-pipeline write-storm.
Note: the shipped output-nudge reaches NEVER-PASSED thrashers too.
path-tracing (304k output tokens), and likely winning-avg-corewars/gcode-to-text/
schemelike, cross the 130k threshold -- so OUTPUT_NUDGE could raise the CEILING
(unlock never-passed thrash tasks), not only stabilize flippy ones. Watch for
output-budget nudge sent markers on never-passed tasks in a full-suite on-run.
antidoom (Liquid4All) evaluated 2026-07-25 as a possible DRY replacement / anti-rumination TRAINING fix — high effort, later experiment. github.com/ Liquid4All/antidoom: an FTPO LoRA pipeline (not a runtime tool) that trains the loop-STARTING token (its own finding: overtrained "Wait,"/"So" — matches our cobol markers exactly) to be rejected in favor of coherent alternatives. Attractive because it fixes the propensity at the weights with NO DRY tool-syntax collateral (DRY was removed for the 122b for exactly that). BUT: vLLM/PyTorch only (no llama.cpp/GGUF — would need LoRA-merge + requantize), ships NO pre-trained adapters, validated only on LiquidAI LFM2.5-1.2B (nothing on Qwen/35B MoE), no published benchmarks, and its DETECTOR is scoped to verbatim repetition while our worst case (cobol) is reworded semantic thrash — so transfer is unproven. Order of effort/payoff: (1) E2BIG guard, (2) runtime thrash-proxy, (3) antidoom-style FTPO adapter only if runtime steering proves insufficient.
Structural note: 40/89 is near this model's ceiling on this suite. The mechanical harness levers are largely exhausted (preamble, guards, recovery, trim all shipped). The remaining gap is model capability (3B active MoE) + variance + env limits. If a materially higher score is the goal, the real lever is the model: the 27b-dense or qwen3.5-122b (122b already flips more fast2 tasks per the 2026-07-12/13 notes), not another MinimalPi hook.
First full 89-task MinimalPi suite since the fix stack landed: 35/89 = 39.3% vs 33/89 pre-fix (20260703), K=1, AGENT_TIMEOUT_MULT=2.0. Churn was big (11 gained / 9 lost); 5 first-ever passes (break-filter-js-from-html, build-pov-ray, large-scale-text-editing, llm-inference-batching-scheduler, overfull-hbox) push the 35b all-time union to 49/89 = 55.1%. 12 ERR trials (11 AgentTimeoutError + 1 NonZeroAgentExitCodeError). See the run NOTES.md.
Investigate the 2 baseline-PASS → ERR-timeout regressions before the official K=5 run: feal-linear-cryptanalysis and qemu-startup both passed the 20260703 suite and now burned the full 2.0× cap (suite__…20260718-104156). Check transcripts for loop-guard/nudge interference vs plain variance; both are also at risk under the leaderboard-mandated AGENT_TIMEOUT_MULT=1.0. Resolved 2026-07-19: transcripts checked — NO guard interference in either (zero loop-guard blocks/nudges; feal-linear's single empty-final recovery nudge was legitimate and late). Both are plain hard-task cap exhaustion: feal-linear kept rewriting a multithreaded brute-force attack to the 1h cap; qemu-startup thrashed serial/telnet bridging approaches (100 shell cmds, 39 writes). Plain K=1 variance; the MULT=1.0 risk note stands.
pytorch-model-recovery NonZeroAgentExitCodeError in the same run —
identify the exit path (the print-mode OOM class was supposedly fixed
2026-07-06; a recurrence would reopen that item).
Resolved 2026-07-19: NOT the OOM class — pi exited instantly (0 tokens,
0s) with Error: Unknown option: - You are given a PyTorch state
dictionary…. The task's instruction.md is the ONLY one of the 89 that
starts with -, and pi's CLI arg parser (cli/args.js) rejects any
positional starting with a single dash and has NO -- terminator. The
20260703 baseline trial died the exact same way (NonZeroAgentExitCodeError,
reward 0) — this task has NEVER actually run. Fix item below.
Dash-leading instruction kills pytorch-model-recovery before the agent
starts (both suite runs; only task affected, checked all 89
instruction.md). Fix in MinimalPi.run(): when
instruction.lstrip().startswith("-"), prepend a neutral line (e.g.
"Task:\n") before the base Pi.run() shlex-quotes it — pi has no --
arg terminator so the harness must transform the prompt. Cheap gate: a
replay whose prompt starts with - must reach turn 1. Guaranteed to
un-brick the trial (not necessarily a pass; task is "medium", 15 min
expert time).
DONE 2026-07-20: MinimalPi.run() prepends "Task:\n" when the instruction
starts with -; task text preserved verbatim below. Shipped in the
suite__20260720 run.
Parsed-toolCall storm burns the whole cap after a garbled length-stop
(torch-pipeline-parallelism suite__…104156, wg:2547): the model looped
emitting Qwen XML <tool_call> markup as text, the parser split ONE
length-stopped generation into ~2547 near-identical write_file calls
(wire markup leaked inside the content args), the write guard correctly
blocked EVERY one, and pi spent the rest of the 30-min budget processing
the queue — the transcript ends mid-write of the giant message_end line
(truncated at the 64KiB pipe buffer by the AgentTimeout kill). Neither
guard terminates this shape: the loop guard's streak counts only its OWN
blocks, and the write guard has no escalation. Fix: count blocked calls
per assistant message in the tool_call hook; on a length-stopped message
with >N (say 8) blocked writes, escalate like the loop guard (fail-fast
the remaining calls with one short error result, or process.exit(0)
hard-stop so the verifier scores on-disk state). Upstream lever is still
the DRY-sampler item (the generation loop itself).
DONE 2026-07-20: new knob WRITE_GUARD_STOP_AFTER (default 8; config/
harness.env + lib.sh + per-trial config.json). The write-guard hook resets
a per-message counter on each assistant message_end and, after N blocked
truncated writes in ONE length-stopped turn, ends the session with
process.exit(0) (verifier scores on-disk state, like the loop-guard
hard-stop). Marker WRITE GUARD hard-stop; flagged in the transcript
viewer (write_escalation). Gate: check_write_guard_logic.py storm probe
(mocks process.exit; non-length turns exempt). Benchmark effect pending
the suite__20260720 run.
rec: recovery counter is DEAD in failure_modes.py AND the transcript
viewer flag: both still match the pre-2026-07-09 marker
"trimming runaway", but the rewritten recovery hook prints
"-- nudging (recovery N/2)" (kinds: length-stop / clean stop with
oversized text / empty final …). suite__…104156 had 13 recovery nudges
across 11 trials, all reported as rec:0 / unflagged. Fix the patterns
(match nudging (recovery), add a case to check_transcript_logic.py.
DONE 2026-07-20: failure_modes.py now counts nudging (recovery (keeps the
old trimming runaway alias for old dirs); transcript.py _MARKERS matches
nudging \(recovery (code runaway) and adds WRITE GUARD hard-stop
(write_escalation). Verified against suite__104156 — rec:1/rec:2 now
render on feal-linear, torch-tensor, winning-avg-corewars, etc. Gate:
check_transcript_logic.py banner-events case extended.
Runaway visible-text reasoning WITH a tool call evades recovery: torch-tensor-parallelism (suite__…104156) emitted an 89,121-char TEXT block in a turn that still ended in a toolCall (stop=toolUse), so the
8000-char recovery trigger (no-toolCall turns only) never fires and the ~22k tokens replay into every later request. Consider trimming oversized visible text on toolUse turns too (keep head+tail; the tool call is the productive part). Single occurrence so far — low priority, but the trial also went to lg:11 + hard-stop. DONE 2026-07-20: new knob
TRIM_RUNAWAY_TEXT(default on) adds acontexthook (registered before the pruner) that trims any assistant visible-text block >8000 chars to head(3000)+marker+tail(2000), idempotently (marker guards re-entry → byte-identical replays, cache-positive like STRIP_THINKING). Only the visible text block is touched; thinking, tool calls, tool results, message 0, user messages untouched. Gate: check_context_strip_logic.py TRIM_PROBE (oversized text on a toolCall turn trimmed, idempotent, off-render inert). Benchmark effect pending the suite__20260720 run.
Wall-clock lever (observation, no mechanism yet): schemelike-metacircular-eval ran 765 assistant turns / 105.8M input tokens in 66 min (~20% of the whole run's input) grinding tiny turns at ~170k ctx, then self-declared done and failed. Nothing bounds turn count or per-trial input tokens; a turn/input budget (nudge-then-stop, like the loop-guard escalation) would cap this shape. Positive note from the same run: the loop-guard hard-stop got its first suite-scale validation — fired in 7 trials (regex-chess, winning-avg-corewars, path-tracing-reverse, tune-mjcf, password-recovery, feal-differential, torch-tensor-parallelism), ending them before their declared 2.0× caps.
Input tokens ran 2.25× the 20260703 suite (533.7M vs 237.0M) with the same K — sessions live longer under the guard stack. Directly inflates the OpenRouter cost estimate for an API-backed K=5 (~$150-250 → recheck with this run's totals: ~$375 input-side at $0.13/M if uncached).