Experiment 2026-07-06 — first `make fast` run: fast-fail timeouts + the 5 new quick-fail tasks

Main goal reminder: measure harness/server changes against terminal-bench 2.0 score on the local qwen models — rigorously, not by vibes. The loop is change → benchmark → reward delta → keep/revert.

Why this note

make fast is a NEW run type (AGENT_TODOS idea #2, shipped 2026-07-06): the smoke set + five quick previously-failing tasks, with per-task fast-fail agent wall clocks ON by default. This first run tests two things at once:

The fast-fail mechanism itself, live. The deterministic probe (scripts/check_fast_timeout_logic.py) proves the wrap kills a stub and writes the marker; what it can't prove is behavior in a real trial container. Success = capped trials show cut@… in make results / ft:Ns in make failures, still get VERIFIED (a scored FAIL, not an errored trial), and trials that finish under their cap are untouched.
First scored numbers for the current harness bundle on 9 quick tasks — nothing has been benchmarked since the 2026-07-06 stack change (see confounds below).

What's being tested (task set + per-task caps + baselines)

Model command below runs the MoE first; baselines are the K=1 suite__qwen3.6-35b-a3b__20260703-003556 results unless noted.

Smoke set (canaries + one known preamble flip):

task	cap	baseline	expectation
fix-git	120s	PASS @ 13s agent	must stay PASS, nowhere near cap
nginx-request-logging	120s	PASS @ 14s	must stay PASS
openssl-selfsigned-cert	120s	FAIL pre-preamble; PASS 3/3 with RULE 2 (20260705 batch)	should PASS
regex-log	600s	PASS @ ~3m20s agent (MoE)	must stay PASS. NB dense 27b passed at ~9m agent — 600s is tight there; a 27b `cut@10m` FAIL here is the CAP, not the model

The five new previously-failing tasks (chosen: quick — failing trial <10 min wall, no long single-thread compile/compute phase — and distinct analyzable failure modes):

task	cap	baseline (20260703 suite)	failure mode to analyze
cancel-async-tasks	180s	FAIL @ 39s agent	single hidden-assertion near-miss
sanitize-git-repo	180s	FAIL @ 51s	git hygiene, fast clean-stop fail
sparql-university	240s	FAIL @ 1m06s	correctness near-miss (H4 control: preamble should NOT flip it)
filter-js-from-html	240s	FAIL @ 1m38s	text/JS filtering miss
query-optimize	600s	FAIL @ 3m06s (0.80× vs required ≥0.95×)	real perf miss; flipped PASS once in the 20260705 batch with V=0 — variance suspect, worth a re-read

Caps are ~2–4× the observed passing agent time (never-passed tasks: a generous multiple of the failing time), defined in FAST_TIMEOUTS_DEFAULT (config/harness.env). Worst-case agent time if EVERY task runs to its cap: 2400s = 40 min per model, plus env-build/verify overhead — vs the multi-hour tail these tasks cost uncapped.

Confounded variables vs the 20260703 baselines

The baselines above predate the 2026-07-06 stack change; deltas are a bundle gate, not clean attribution:

pi 0.73.1 → 0.80.2 (removes the hidden 32000 output clamp) + MAX_TOKENS 32768→65536, CONTEXT_WINDOW →196608, COMPACT_AT →180000.
Print-mode stream-tick patch (big-write container OOM fix) — benchmark effect never measured.
Write guard narrowed to truncated-only (stopReason=length).
Server: min-p 0.0, reasoning-budget 8000 (both already active for the 20260705 batch, not for the 20260703 suite).
The fast caps themselves — any trial with a cut@ marker is not wall-time- or late-pass-comparable to uncapped runs. Verdicts of trials that finished under their cap ARE comparable.

Decision rule

Mechanism: any capped trial that ends ERR instead of scored-FAIL, or a missing marker on a visibly cut trial → fix the harness before trusting fast runs.
Canaries (fix-git, nginx, regex-log MoE) must stay PASS; a canary lost AT its cap → the cap is too tight, raise it in FAST_TIMEOUTS_DEFAULT rather than dropping the task.
The five fails are there to be analyzed (make failures, verifier near-miss output), not expected to flip. Any flip is a bonus datapoint for the 0.80.2/64k bundle — K=1, so treat single flips as variance-suspect (query-optimize has form here).

Command

make fast                          # both models, per-task caps on by default
# or one model first:
make fast MODELS=qwen3.6-35b-a3b
# watch live in a 2nd terminal:
make watch

Variants: FAST_TIMEOUTS="default=300" make fast (override the map), FAST_TIMEOUTS=off make fast (same task set, uncapped), FAST_TIMEOUTS="default=600" make suite (caps on any run type).

Results land in

runs/fast__<model>__<stamp>/ — read with make results (cut trials show cut@Xm) and make failures (ft:Ns flag; verdict is post-cut verification). Per-trial marker: <trial>/agent/fast-timeout.json (exit_code 124 = TERM cut, 137 = KILL/OOM — cross-check timing). Write the run's narrative into runs/fast__…/NOTES.md per the RUNS.md convention.