An iterative attempt to make local Qwen models (llama.cpp on a single RTX 3090, no external LLM providers) perform well on terminal-bench 2.0 by tuning the agent harness and the server config, one measured run at a time. Each run below links to a detail page with what was tested, the results, an interpretation and the next steps — read the story top to bottom to follow the iterations.
Chronological — each entry is one run's headline; click through for the full notes (what changed, results, interpretation, next steps).
Human-readable notes on the benchmark runs under runs/. The per-trial JSON
records the model, thinking, max_tokens, llama_port, rewards and tokens —
but not which reasoning-budget mechanism was in effect. That's the whole
point of this file.
Per-run narrative lives in
runs/<job>/NOTES.md(what was run & changed, results, interpretation, next steps) — rendered on each run's page of the Pages site, with the first# headingused as the run's timeline headline. This file stays the cross-run journal: mechanism history and how to read old runs.
llama_port in the trial config tells you the path:8020 = the agent talked directly to llama-swap.8021 = the agent went through the reasoning-cap proxy (a since-removed
host-side proxy that capped reasoning at ~4000 tok/turn and forced an
answer). Any port-8021 run therefore had an effective reasoning budget of 4000.thinking = (none) in the config means the run predates the thinking
control (the early reasoning: false extension bug): reasoning was on but
uncontrolled, so it could loop and burn the whole budget.preamble in the trial config (from 2026-07-04 night onward): the
scaffolding preamble (chunked writes + verified-done rules appended to pi's
system prompt). Absent from a trial's config = the run predates the knob
(no preamble). on/off is recorded per trial, so runs are directly
attributable — unlike the server-side reasoning budget.fast_timeouts in the trial config (from 2026-07-06 onward): per-task
fast-fail agent wall clocks (make fast sets them by default; empty/absent =
uncapped). A trial the cap actually CUT carries
<trial>/agent/fast-timeout.json and a {"type":"fast_timeout",…} line in
pi.txt — its verdict is post-cut verification (scored normally, no
AgentTimeoutError), so it is NOT wall-time- or late-pass-comparable to
uncapped runs; trials that finished under their cap are unaffected. Shown as
cut@Xm by make results, ft:Ns by make failures. First run + task
rationale: runs/EXPERIMENT-20260706-fast-target.md.llama_port=8021.
3. server flag — llama.cpp native --reasoning-budget 4000 set in the
llama-swap config (llama_cpp_on_my_desktop, committed 2026-07-02, server
restarted the same day). This is the current & future mechanism.Future runs use the server flag, so they will show llama_port=8020 and
thinking=on — which looks identical in the logs to the old uncapped
port-8020 runs (e.g. cmpthink-on). The difference (budget enforced
server-side) is invisible to the trial JSON. Tell them apart by date
(anything after the 2026-07-02 server restart has --reasoning-budget 4000) and
by this journal. The budget value lives in the llama-swap config's git history,
not in these logs — if you change it, add a row below noting the new value.
Budget change 2026-07-04 ~19:00: 4000 → 8000 (both qwen bases, uncommitted
in llama_cpp_on_my_desktop at the time of the switch). Motivation: the 4000
force-close was cutting thinking mid-thought and spawning plain-text runaway
loops (2/5 fails in run …180910). Confirmed in effect from run …192605 onward
(think blocks up to ~6–7k tok, impossible under 4000). Rule of thumb: runs
dated before 2026-07-04 19:00 = budget 4000; after = 8000.
| Job (runs/…) | Reasoning budget | thinking | tasks × K | result | notes |
|---|---|---|---|---|---|
smoke__…163559 |
none (uncontrolled) | (none) | 4 × 1 | 4/4 | first smoke; regex-log happened to terminate |
smoke__…164851 |
none (uncontrolled) | (none) | 4 × 1 | 3/4 | exposed the loop: regex-log burned 32k tok on reasoning, failed |
cmpthink-off__…171738 |
none | off | 4 × 1 | 3/4 | thinking OFF: regex-log passes, openssl fails |
cmpthink-on__…172246 |
none | on | 4 × 1 | 3/4 | thinking ON uncapped: regex-log loops/fails, openssl passes |
smoke__…180115 |
proxy 4000 | on | regex-log × 1 | 1/1 | proxy validation; cap fired 2× |
smoke__…181612 |
proxy 4000 | on | 2 × 5 (=10) | 0.80 | regex-log 5/5, openssl 3/5 |
smoke__…191732 |
proxy 4000 | on | regex-log × 1 | errored | single errored trial (ignore) |
smoke__…191746 |
proxy 4000 | on | 2 × 15 (=30) | 0.77 | regex-log 14/15 (93%), openssl 9/15 (60%) |
smoke__qwen3.6-27b__190304 |
proxy 4000 | on | 4 × 1 | 4/4 | dense 27B model through the proxy |
smoke__…213821 |
server flag 4000 | on | 2 × 10 (=20) | 0.75 | first confirmed server-flag run (--reasoning-budget 4000, port 8020). regex-log 8/10, openssl 7/10. Parity with proxy confirmed — see below |
suite__…20260703-003556 |
server flag 4000 | on | 89 × 1 | 0.371 | first full suite, ~23.5 h. See its NOTES.md + the two ANALYSIS-*.md reports |
smoke__…20260704-121319 |
server flag 4000 | on | 3 × 1 | 1/3 | first context-fix run (context_window=229376 + COMPACT_AT_TOKENS=200000 guard): guard active in every trial but never fired (peaks 51–165k), zero hard 400s; both former 400-victims now clean-stop near-misses — trajectory variance, not the fix. See its NOTES.md |
smoke__…20260704-131520 |
server flag 4000 | on | regex-log × 1 | 1/1 | guard-validation attempt at COMPACT_AT_TOKENS=40000 — PASS 2m14s but peaked at 28k, trigger still never fired; retry at 15000 (see NOTES.md) |
smoke__…20260704-132733 |
server flag 4000 | on | regex-log × 5 | 0/5 | guard fired in all 5 — and killed every trial: ctx.compact() aborts the agent loop; in pi --print mode the process exits mid-compaction. Guard must be rebuilt on the context hook; COMPACT_AT_TOKENS=200000 harmful until then. See NOTES.md |
smoke__…20260704-140750 |
server flag 4000 | on | regex-log × 5 | 4/5 | prune-guard (context hook) validated: 87 prunes across 4 trials, all 5 ran full trajectories, no deaths; the fail is a findall-capture-group near-miss, not pruning. Exposed a ~2× chars/4 underestimate on dense content → usage-calibrated ratio added after the run. See NOTES.md |
smoke__…20260704-150817 |
server flag 4000 | on | regex-log × 5 | 4/5 | calibrated guard confirmed: prunes in 4/4 eligible trials, measured ratios 1.84–2.03, peaks capped 11–14k real at limit 15000. Fail = turn-1 reasoning-budget force-close (out 4002 ≈ budget 4000, thinking cut mid-regex, text=>, no tool call) — 2nd sighting of that artifact, unrelated to the guard. See NOTES.md |
smoke__…20260704-152728 |
server flag 4000 | on | circuit-fibsqrt + make-mips × 1 | 0/2 | H6 closed: both remaining 400-victims at stock settings — zero hard 400s, clean ~58 min trajectories, peaks 114–136k (were 242–257k deaths), ratios 1.56/2.53, up to 216 messages stubbed per call. Both fail on merit: mips VM too slow; circuit = B2 oversized-writes (H2 is the next lever). See NOTES.md |
smoke__…20260704-180910 |
server flag 4000 | on | regex-log × 5 | 3/5 | hysteresis guard correct in 5/5 (batched advances, stable stubs, live ratio 1.7→2.45). Both fails = budget force-close → plain-text thinking-continuation runaway (smoking gun: TEXT continues the truncated regex char-for-char) — the force-close artifact CAUSES B1 loops, cost 2/5 tonight. Cache A/B neutral at limit 15k (slack too small); decisive test = make-mips at stock. See NOTES.md |
smoke__…20260704-192605 |
server flag 8000 | on | 4 × 1 | 3/4 | first budget-8000 run (think blocks up to ~5.9k tok confirm the flag took). fix-git, nginx-request-logging, regex-log pass; openssl fail = its known ~60% baseline (near-miss, one assertion away). See NOTES.md |
smoke__…20260704-193517 |
server flag 8000 | on | make-mips × 1 | 0/1 (timeout) | the owed hysteresis cache rerun — but the guard never engaged (peak ~55k ≪ 200k; 19 turns in 1h, AgentTimeoutError right at the wire vs 58 min last time). Cache trivially clean: 93.2% hit, cacheRead monotone, 0 breaks. Measurement still owed. Fails on merit (MIPS VM too slow) either way. See NOTES.md |
smoke__…20260704-203646 |
server flag 8000 | on | regex-log × 5 | 5/5 | budget 8000 validated on the artifact task: zero length-stops, thinking completes naturally at ~6–7k tok (was cut at 4000 → 2/5 runaway fails the night before), trials 1m32s–3m03s. regex-log is 6/6 at 8000 counting …192605. See NOTES.md |
smoke__…20260704-205241 |
server flag 8000 | on | polyglot-rust-c × 3 | 0/3 | expected-fail task, two signals: 2× B2 oversized-write (82–89k-char tool args; needs H2 chunked writes, budget irrelevant) + 1× soft runaway — polyglot maxes the 8000 budget every turn; final turn's cut reasoning continued as 19k chars of TEXT then self-terminated with a clean stop, no tool call → trial ended. Evades the length-only recovery → recovery trigger extended (stop + no tool call + text >8k chars). See NOTES.md |
regex-log — which loops and fails uncapped — passed
~95% (19/20) with the cap. That jump is the evidence.openssl-selfsigned-cert ~60% is NOT a budget issue. Its entire trajectory
is only ~1.7k–3.7k output tokens, so per-turn reasoning never reaches the 4000
cap — the budget never fires for it. 60% is the model's baseline on that task.Conclusion: 4000 fixes the loop without hurting the short task. It hasn't been tuned on breadth — a proper sweep (2000 / 4000 / 8000) should run against the full suite, since different tasks stress reasoning length differently. And these were proxy runs; re-verify parity now that the server flag is live.
This is the first run on the native server budget (--reasoning-budget 4000
in the llama-swap config), replacing the proxy. Because a server-flag run is
indistinguishable from an old uncapped port-8020 run in the trial JSON, parity
was verified directly from the transcripts (agent/pi.txt):
thinking block per trial: max ≈3,141
tok (median ~146), vs ~20,000 tok in the uncapped runs (164851,
cmpthink-on 172246) and ~4,001 tok under the proxy. No reasoning block gets
near the 32k maxTokens ceiling. (The server tops out below 4000 because
llama.cpp nudges the model to close </think> as it nears the budget, rather
than the proxy's hard splice exactly at 4000.)stopReason: length — every turn ended cleanly (toolUse/stop). The
old uncapped 164851 had a length stop; that failure mode is gone.Verdict: proxy → server flag is a clean swap — same bounding, same scores,
no extra process. 4000 stays as the value pending a full-suite breadth sweep.
To re-check on any future run: max thinking-block tokens should stay well under
4000 and stopReason: length count should be 0.