QwenPi (qwen-native tools) — 35b fast, 5/9; tools work flawlessly, no score change
First benchmark run of the QwenPi harness (read_file / write_file / edit /
run_shell_command, --no-builtin-tools, delegating to pi's builtins).
Knobs otherwise current (loop_guard on, strip off, subagent_timeout 600).
Result 5/9 — identical pass-SET to the closest plain-MinimalPi baseline
(fast__qwen3.6-35b-a3b__20260706-113104, also 5/9: fix-git, nginx, openssl,
sanitize-git-repo, regex-log PASS; cancel-async, filter-js, query-optimize,
sparql FAIL). The tool rename moved nothing on score.
Tool mechanism verdict — clean: 0 schema/unknown-tool errors across all 9
trials; the model used the qwen-named tools fluently and every wrapper
delegated to the real builtin (fix-git 18s, nginx/openssl 16s agent). The
suspiciously fast fails are NOT tool regressions: cancel-async (17s) made
write_file+run_shell_command+read_file and a real solution+summary — the
usual hidden-edge-case near-miss; filter-js (26s) likewise (the known
parse-reserialize class). regex-log PASSED with heavy tool use and NO
format leak.
Takeaway: on this task set the failures are task-logic / hidden-criterion,
not tool-surface friction — so matching the surface (which the literature
says helps 3-8B models) had nothing to remove here. See the ledger verdict.