← older: smoke__qwen3.6-35b-a3b__20260704-131520all runs · task boardnewer: smoke__qwen3.6-35b-a3b__20260704-140750 →

smoke__qwen3.6-35b-a3b__20260704-132733

Guard-validation K=5 — 0/5: the guard fired in every trial and every firing KILLED the trial; ctx.compact() is fatal in pi's print mode

What was run

regex-log × 5, COMPACT_AT_TOKENS=15000 — a threshold every regex-log draw crosses mid-run, to finally exercise the turn_end compaction guard.

Result — the guard works exactly as written, and that is the bug

All 5 trials: guard armed, fired once (minimal-pi: context 16000–19401 > 15000 tokens, forcing compaction), then the trial died on the spot — 6–7 turns, ~1.5 min, last stopReason toolUse (mid-task, tools still pending), transcript ends at compaction_start (reason: "manual") with no compaction_end, no error, no further events. pi exited; Harbor ran the verifier against a half-finished workspace. 0/5 on a task that is ~95% at K≥5.

Root cause (from pi 0.73.1 source, @mariozechner/pi-coding-agent)

ctx.compact() → session.compact(), whose first two lines are:

this._disconnectFromAgent();
await this.abort();          // aborts the running agent loop

It is built for the interactive /compact command (abort any streaming, then summarize). Harbor runs pi --print --mode json, where the process lifetime is await session.prompt(...): aborting the loop resolves prompt(), print mode returns, and the process exits — orphaning the compaction that was fired async (void (async …), nobody awaits it). Hence compaction_start as the final event, every time.

Collateral findings (they correct two standing beliefs)

  1. pi's auto-compaction DOES anchor to the declared contextWindow (agent-session.js:1383: this.model?.contextWindow ?? 0), not the server's real -c. The comment in minimal_pi.py/harness.env claiming otherwise is wrong. The real reason it never fired in the 20260703 suite: _checkCompaction runs only at agent_end or before the next user prompt — i.e. never mid-loop in one-shot print mode. No declared-window value can make built-in threshold compaction work mid-run.
  2. The suite's four hard-400 deaths actually DID trigger pi's built-in overflow recovery — 3 of 4 transcripts show compaction_start (reason: "overflow") as their last event. It dies the same way: at agent_end the print-mode process exits before the async compact+retry completes (and at 262k the summarization request itself would overflow llama.cpp anyway). So H6's "catch the 400 → compact/retry" cannot be delegated to pi in print mode either.
  3. pi 0.73.1 is the latest release — no upgrade available.

The viable mechanism (found in the same source)

pi.on("context", handler) — "Fired before each LLM call. Can modify messages." It is awaited inside the agent loop via the agent's transformContext, receives a structuredClone of the messages, returns {messages} to replace them, cannot abort anything, and handler exceptions are swallowed by the runner. Redesign the guard as a context-pruning transform: when the running total exceeds the ceiling, drop/stub the oldest tool results (keep system prompt + task instruction + recent turns + a short "[elided N old tool results]" marker) and return the trimmed array. No summarize call, no abort, and requests physically never reach the 262k cap — which also subsumes the H6 hard-400 fix.

Action items

  1. Until the redesign lands, COMPACT_AT_TOKENS=200000 is ACTIVELY HARMFUL — any suite trial crossing 200k gets executed by its own guard. Set the default to 0 (guard off) or land the context-hook version first.
  2. Rewrite the guard on the context hook; re-run this exact K=5 experiment (threshold 15000) — success = 5 firings, ≥4/5 pass, peaks capped.
  3. Fix the wrong "anchors to server context" comments in minimal_pi.py / harness.env / CLAUDE.md when touching the code.

Run details

modelllama-local/qwen3.6-35b-a3bagentharnesses.minimal_pi:MinimalPithinkingonreasoning budgetdirect (server/none)* — see journal for the authoritative mechanismmaxTokens / contextWindow32768 / 229376agent timeout ×2.0trials5 of 5 — 0 pass · 5 failmean reward0.00tokens (job total)263,348 in / 67,793 outstarted / finished2026-07-04T13:27 / 2026-07-04T13:39wall clock11m58s

Tasks

regex-log — 0/5 passed

#resulttotalagentin/out tokflags
1FAIL2m34s1m39s67893/15251
—
🔍 view
2FAIL2m25s1m29s43960/14399
—
🔍 view
3FAIL2m31s1m34s61461/14795
—
🔍 view
4FAIL2m11s1m15s47933/11651
—
🔍 view
5FAIL2m16s1m20s42101/11697
—
🔍 view