← older: fast__qwen3.6-35b-a3b__20260708-210003all runs · task boardnewer: fast__qwen3.6-27b__20260708-213445

fast__qwen3.6-35b-a3b__20260708-210920

QwenPi (qwen-native tools) — 35b fast, 5/9; tools work flawlessly, no score change

First benchmark run of the QwenPi harness (read_file / write_file / edit / run_shell_command, --no-builtin-tools, delegating to pi's builtins). Knobs otherwise current (loop_guard on, strip off, subagent_timeout 600).

Result 5/9 — identical pass-SET to the closest plain-MinimalPi baseline (fast__qwen3.6-35b-a3b__20260706-113104, also 5/9: fix-git, nginx, openssl, sanitize-git-repo, regex-log PASS; cancel-async, filter-js, query-optimize, sparql FAIL). The tool rename moved nothing on score.

Tool mechanism verdict — clean: 0 schema/unknown-tool errors across all 9 trials; the model used the qwen-named tools fluently and every wrapper delegated to the real builtin (fix-git 18s, nginx/openssl 16s agent). The suspiciously fast fails are NOT tool regressions: cancel-async (17s) made write_file+run_shell_command+read_file and a real solution+summary — the usual hidden-edge-case near-miss; filter-js (26s) likewise (the known parse-reserialize class). regex-log PASSED with heavy tool use and NO format leak.

Takeaway: on this task set the failures are task-logic / hidden-criterion, not tool-surface friction — so matching the surface (which the literature says helps 3-8B models) had nothing to remove here. See the ledger verdict.

Run details

modelllama-local/qwen3.6-35b-a3bagentharnesses.qwen_pi:QwenPithinkingonreasoning budgetdirect (server/none)* — see journal for the authoritative mechanismmaxTokens / contextWindow65536 / 196608agent timeout ×2.0trials9 of 9 — 5 pass · 4 failmean reward0.56tokens (job total)1,962,325 in / 72,533 outstarted / finished2026-07-08T21:09 / 2026-07-08T21:34wall clock25m23s

Tasks

cancel-async-tasks — 0/1 passed

#resulttotalagentin/out tokflags
1FAIL1m06s17s27346/2532
🔍 view

filter-js-from-html — 0/1 passed

#resulttotalagentin/out tokflags
1FAIL3m30s26s42349/4334
🔍 view

fix-git — 1/1 passed

#resulttotalagentin/out tokflags
1PASS56s18s42413/2965
🔍 view

nginx-request-logging — 1/1 passed

#resulttotalagentin/out tokflags
1PASS55s16s36707/2310
🔍 view

openssl-selfsigned-cert — 1/1 passed

#resulttotalagentin/out tokflags
1PASS53s16s38489/2713
🔍 view

query-optimize — 0/1 passed

#resulttotalagentin/out tokflags
1FAIL8m47s2m00s51558/4839
a bash command timed out ×2
🔍 view

regex-log — 1/1 passed

#resulttotalagentin/out tokflags
1PASS3m39s2m27s587924/23917
long reasoning (21,791 chars)
🔍 view

sanitize-git-repo — 1/1 passed

#resulttotalagentin/out tokflags
1PASS1m55s1m17s587497/7280
🔍 view

sparql-university — 0/1 passed

#resulttotalagentin/out tokflags
1FAIL3m38s2m48s548042/21643
long reasoning (24,354 chars)
🔍 view