query-optimize - quadratic final join made the verifier itself time out

Status: ERR (VerifierTimeoutError after 900s) in fast__qwen3.6-35b-a3b__20260707-001626 (subagents harness). Not an agent timeout and not an infra flake: the agent finished normally after ~8 min, the verifier started, passed tests 1–2, then hung ~9 minutes inside test_compare_golden_vs_solution_runtime until Harbor killed the exec at the task's 900s verifier cap. VerifierTimeoutError is on the retry exclude list, so it stays an ERR on the board.

True verdict is FAIL, not a near-miss. The solution is semantically correct (tests 1–2 passed; the agent's own diff showed 0 differences) but measured ~140x slower than golden — even with an unlimited verifier budget the <= 1.05 * golden_median assertion fails by two orders of magnitude.

Contrast: the plain-pi baseline on this model failed the same test at 1.36x golden (see the pi harness analysis for this task) — a near-miss. This run regressed to a categorically worse query. The 27b subagents run (fast__qwen3.6-27b__20260706-231402) PASSED this task at 0.99x golden.

Why the verifier ERRORED instead of failing

Measured on the host against the task DB with the verifier's PRAGMAs (sqlite 3.53): golden 0.30s, this run's sol.sql 42.4s per execution. Scaled by the in-container golden time from passing runs (0.67s ≈ 2.2x host), sol ≈ 95s+ per run in the 1-CPU/2GB container. The runtime test executes each query 6 times (warm-up + 5 alternating iterations) ≈ 570s+, but tests 1–2 (including one ~2 min run of the slow original in test 1, plus apt/uv setup) had already burned ~360s of the 900s budget. The timeline confirms it: verifier exec 00:26:36 → last stdout (..) 00:32:32 → killed 00:41:36.

Consistent with the transcript: the agent's two verification commands (each running original + sol once) took 216s each — original ≈ 120s + sol ≈ 95s.

Generalizable: any slow-but-correct solution to this task with per-run cost over roughly 90s converts a scored FAIL into an unscored ERR, because the benchmark loop no longer fits the fixed 900s verifier budget.

The bug: a 3-way join with no usable index

Every CTE in the agent's query is individually fast (base 0.07s, word_stats 0.21s, synset_counts 0.29s on host). The blowup is entirely in the final SELECT, isolated by bisection:

• word_stats JOIN top_synset: 0.26s
• words JOIN word_stats: 0.17s
• words JOIN word_stats JOIN top_synset: 42s

The plan for the 3-way join shows why:

SCAN ws                                                      -- 17,834 rows
SEARCH synset_counts USING AUTOMATIC PARTIAL COVERING INDEX (rn=?)
SEARCH words USING AUTOMATIC COVERING INDEX (wordid=?)

The DB ships with zero indexes, so SQLite builds automatic transient indexes per query. For the top_synset lookup it chose an index on rn alone — but rn = 1 matches all 17,834 top rows, so each of the 17,834 word_stats rows scans all of them and filters by wordid afterwards: ~318M row visits, quadratic in the eligible-word count. The golden query never creates this shape — its final SELECT joins only two materialized CTEs pairwise (word_stats ⋈ synset_sense_counts with the rn = 1 filter inline), and the automatic index lands on wordid.

Notably the agent's redundant fourth join (synset_counts sc joined again on rn = 1 beside top_synset) is NOT the cause — a variant without it still takes 42s. The killer is purely the planner's rn-only automatic index on the 3-way join.

Trajectory

15 tool calls, competent exploration (schema, counts, EXPLAIN QUERY PLAN on the original), then one write of sol.sql and verification:

• Correctness was genuinely verified: row counts matched (502/502) and a full diff of original vs solution output was empty. RULE 2 worked.
• Performance was never measured. The agent compared EXPLAIN QUERY PLAN shapes ("no more correlated scalar subqueries") and declared the query optimized. It never timed sol against anything — not the original, not /tests/golden.sql — on a task whose single scored axis is runtime.
• The red flag was in plain sight: both of its own verification commands took 216 seconds and it drew no conclusion from that.
• Zero subagent delegation. Despite the strengthened "make your very FIRST action a subagent tool call with agent scout" workflow prompt, this trial made no subagent calls at all — pure self-serve. Whatever this trial measures, it is not the subagent workflow; count it accordingly in the delegation-rate A/B.

Verdict attribution

Model optimization + verification miss, same failure class as the plain-pi baseline (optimized against the original, never against golden), but with a worse query and a new harness-visible consequence: the slow solution pushed the verifier past its own timeout, so the board shows ERR where the true verdict is FAIL. No harness mechanism misbehaved (no cap fired, write guard and pruner idle).

Follow-ups

• Same nudge-detector idea as the baseline analysis, sharpened: on tasks with a runtime-threshold verifier, "query-plan looks better" claims without a wall-clock measurement are exactly the claim-vs-evidence gap — here the agent even had the 216s observation and ignored it.
• When reading the board, treat query-optimize ERRs on this harness as FAILs for scoring intuition (slow-but-correct converts FAIL → ERR mechanically).
• Reproduction: extract /app/oewn.sqlite from alexgshaw/query-optimize:20251031, run the sol.sql from the transcript (runs/fast__qwen3.6-35b-a3b__20260707-001626/query-optimize__mAEywRE/agent/pi.txt) with PRAGMA temp_store=MEMORY; PRAGMA cache_size=-100000 on a read-only connection.

2026-07-08 strip-off rerun (7rKKLUm) — SOLVED (verifier 1.0) but recorded ERR via two harness holes

In fast__qwen3.6-35b-a3b__20260708-115953 (loop_guard+nudges on, strip_thinking off) the model finally beat this task — the verifier scored reward 1.0 — yet the board shows ERR AgentTimeoutError at exactly 30m00s. The 30 minutes decompose into three phases, two of them pure harness loss:

1. 17 min (12:04→12:21): a looping planner child. The first planner subagent blocked the root for 1021s, internally repeated a grep until the loop guard blocked it, then died exit 1 ("grep failed … LOOP DETECTED"), work lost — the third occurrence of the child-loop→guard-block→exit-1 shape (see the fix-ocaml-gc 2026-07-08 analysis). NB this arm had the context strip OFF, so child loops are orthogonal to the strip.
2. ~4 min (12:21→12:25): the actual solve. Second planner (fast), worker, own verification (VERIFICATION PASSED: outputs are identical), one reviewer child, clean final answer at 12:25:43 (session span 1306s). Unlike the 07-07 trial, runtime was actually addressed this time — the verifier's <= 1.05x golden runtime test passed.
3. ~8 min (12:25→12:33): post-session hang. The pi process never exited after the final message and Harbor killed it at the 1800s cap → AgentTimeoutError. Second confirmed occurrence (headless-terminal 2026-07-08 was the first); both trials used subagent children and had "State: remembered foreground" in a status result. Harbor ran the verifier anyway, which is why the reward exists behind the ERR.

Scoring note: count this as a PASS when reading the board (reward 1.0 is recorded); the ERR is teardown, not agent failure. Ledger items: subagent child wall-clock cap + post-session hang fix.