Resource-lifted arm — 7 env/CPU-bound tasks at 24 CPU / 32 GB
Purpose: test whether the 7 tasks that ERR'd on the terminal-bench per-task
resource declarations (cpus=1–4, memory=2–4G) in suite__20260720 were blocked by
the cap or by capability. New CPU_OVERRIDE knob (--override-cpus) + existing
MEM_OVERRIDE_MB lifted to 24 CPU / 32 GB for every task. MinimalPi, 35b,
K=1, uncapped agent timeout (declared × 2.0). NON-STANDARD arm: override_cpus
+ override_memory_mb are recorded in config.json; Harbor's leaderboard
static-validation rejects both, so this is NOT leaderboard-comparable and is kept
off the board (BOARD_SET_EXCLUDE).
RESULT: 3/7 PASS (mean 0.43). Elapsed 4h25m.
| task |
before (1c / 2–4G) |
after (24c / 32G) |
| caffe-cifar-10 |
ERR timeout @40m |
PASS @17m |
| mcmc-sampling-stan |
ERR timeout @60m |
PASS @11m |
| qemu-alpine-ssh |
ERR died @2m |
PASS @16m |
| install-windows-3.11 |
ERR OOM (137) @12m |
FAIL @5m (2/4 tests) |
| make-doom-for-mips |
ERR timeout @30m |
ERR timeout @30m |
| train-fasttext |
ERR timeout @2h |
ERR timeout @2h |
| extract-moves-from-video |
ERR timeout @60m |
FAIL @56m |
Applied to the suite that is 40 → 43/89 (48.3%).
Interpretation
- 3 genuine CPU/RAM walls, now solved: caffe-cifar-10 (build+train),
mcmc-sampling-stan (Stan compile+sample), qemu-alpine-ssh (TCG boot + SSH —
the surprise; it had died fast, not timed out).
- install-windows-3.11: the memory bump removed the OOM (exit 137 → clean run
scoring 2/4 tests). What remains is a CAPABILITY wall — it can't drive the
graphical Windows install (
test_qemu_running_with_correct_params +
visual-feedback fail). Resources removed the wall; the GUI install is the ceiling.
- make-doom-for-mips: never resource-bound — 0 loops, 0 cut commands, churned
24M input tokens over 30m without converging. Hard cross-compile reasoning.
- train-fasttext: genuine long compute — each
fasttext supervised run took
600s even at 24 threads; the model ran several variants and its own 600s
timeouts cut 4 of them. CPU helped per-run but its multi-run strategy didn't fit
2h. (The 90s bash default was NOT a factor.)
- extract-moves-from-video: video processing genuinely ~56m; now completes
(no timeout) but produces the wrong output → FAIL.
Bottom line: the lift cleanly separated "blocked by the cap" (3, fixable, +3) from
"the cap was never the problem" (4: capability / hard-reasoning / long-compute /
wrong-output). Resources remove walls; they don't solve tasks.