gemma-4-12b first light — smoke 3/4

First benchmark run of gemma-4-12b (Gemma 4 12B Unified, dense 11.95B, Q6_K + MTP drafter, f16 KV, -c 262144, --reasoning on --reasoning-budget 8000). Harness: MinimalPi (baseline), K=1, no fast-fail caps, all knobs at their config/harness.env defaults (thinking=on, qwen_tools=on, max_tokens 65536).

task	result	total	agent	in/out tok	laguna agent	35b agent
fix-git	PASS	1m13s	39s	116124/3319	FAIL 3m44s	18s
nginx-request-logging	PASS	1m45s	1m09s	111009/7207	7m02s	15s
openssl-selfsigned-cert	PASS	1m18s	44s	100258/4522	4m53s	13s
regex-log	ERR	30m44s	30m00s	0/0	PASS 21m30s	cut@20m

3/4 passed, 1 errored, elapsed 35m01s. Same score as laguna's first light, but the three passes are 6–9× faster in agent time.

The one failure is a RUNAWAY GENERATION, not a hard task

regex-log did not thrash and did not time out on a slow task. The llama-swap log shows a single generation — task 1056 — that started at 19:29:10, ran for the full 30-minute agent budget at ~118 tok/s, emitted 215,206 tokens in one response, and ended only because pi disconnected (recovered from upstream disconnection during streaming). pi never got a message back, which is why agent/pi.txt is 0 bytes and in/out tokens are 0/0.

It was not even context-bounded: 1718 prompt + 215206 output = 216,924 of the 262,144 window, i.e. it still had ~45k to go.

Diagnosis, all probed directly on the box the same day:

• max_tokens IS honoured when sent — max_tokens: 2000 → predicted_n exactly 2000, finish_reason: length. MTP does not overshoot it. So the runaway request simply carried no cap.
• --reasoning-budget IS working — set to 200 as a probe, thinking capped at ~128 tokens. So the 215k tokens were visible content, not thinking. The reasoning budget cannot bound this failure mode.
• No harness guard can bound it either. write_guard, loop_guard, trim_runaway_text, output_nudge and the context pruner all run between turns; none can interrupt a single in-flight generation.

Fix applied (server side, in llama_cpp_on_my_desktop)

llama.cpp's -n/--predict defaults to -1 = infinity, so any client that omits max_tokens can drive any model until the context fills. The gemma-4-12b entry now sets -n 65536 — exactly this repo's own MAX_TOKENS ceiling, so it clamps only requests asking for more than the harness ever intends. Verified: an uncapped would-be-runaway prompt now stops at exactly 65536 with finish_reason: length instead of running to 215k. Worst case goes from unbounded (30 min) to ~9 min.

This is a per-model fix on the one model that demonstrated the problem — every other entry in tools/llama-swap/config.yaml still has the same unbounded exposure. It has simply never been hit, presumably because the other models stop on their own.

Still open: why that request carried no max_tokens when the provider extension declares maxTokens: 65536. The server-side cap makes it survivable either way, but the harness-side question is unanswered and regex-log should be re-run to see whether the capped generation now leaves the agent enough turns to finish.

Model behaviour notes

• Fast, and stays fast at depth. 142–149 tok/s short, 117–119 tok/s on a real 41.6k-token prompt. Prompt-cache hit rates were high on the passes (fix-git: 109k of 116k input tokens served from cache).
• MTP earns its keep, unlike Bonsai's DSpark: matched A/B measured 2.31× short and 2.16× at 41.6k, with draft acceptance holding at ~76% at depth (DSpark decayed to 32–40% by 13k).
• No Laguna-style empty-content pathology — real content 3/3 at max_tokens 256/1024/3000.
• Thinking lands in reasoning_content with no raw tag leak; tool calls come back as proper OpenAI tool_calls (this llama.cpp build has a hand-written gemma4 parser, so it is not relying on the differential autoparser).

Caveat on comparability

qwen_tools=on was left at its default, i.e. gemma-4-12b was given the Qwen-named tool surface (read_file/write_file/edit/run_shell_command). That surface was chosen because it matches what Qwen3.6 was post-trained on; there is no reason to think it matches Gemma 4's own tool post-training. A QWEN_TOOLS=off arm is the obvious first A/B for this model.