filter-js-from-html - regex sanitizer passed its own smoke test, failed the corpus

Status: FAIL (2/2 verifier tests) in fast__qwen3.6-35b-a3b__20260706-113104. The fast-fail cap was 240s, but it did not fire: the agent ended normally after ~1m31s, then the Selenium verifier ran for ~3m and failed both checks. Baseline: also FAIL in suite__qwen3.6-35b-a3b__20260703-003556 at ~1m38s agent time. Both runs failed the same two verifier classes, though today's sanitizer was narrower and left more XSS batches alerting.

What the task wants

Create /app/filter.py, taking an HTML file path as argv[1], editing it in place to remove JavaScript while preserving legitimate HTML structure and content. The public wording says not to alter formatting, but the hidden verifier is more specific in practice:

download the html-sanitizer-testbed XSS corpus, add embedded vectors, run /app/filter.py on each vector, concatenate filtered files into batches, and open each batch in headless Chrome;
assert that no batch triggers an alert;
run 12 clean HTML samples through the filter and compare the result with BeautifulSoup(original, "html.parser") output after whitespace removal.

That last check matters: a byte-preserving filter can still fail if it does not produce the verifier's parser-normalized serialization for clean pages.

Trajectory

The model wrote a regex-based sanitizer in its first turn, read it back, made one cleanup edit, then ran only a synthetic smoke test that it generated itself. The smoke test covered common cases:

ordinary <script> blocks;
common event handlers like onclick, onerror, onload;
plain javascript: URLs in href, src, action, and formaction;
safe-looking structure such as tables, forms, SVG, and safe links.

That smoke test passed, and the model declared "All tests pass." It never ran the task verifier or any approximation of the downloaded XSS corpus.

The bug

The final sanitizer is a hand-written regex filter:

remove <script>...</script>;
remove a fixed allowlist-style tuple of known on... event attributes;
blank URL attributes only when the value, after stripping whitespace/control characters, starts with literal javascript:.

This misses too much of the corpus:

encoded or entity-obfuscated JavaScript URLs, e.g. javascript%3A..., javas%63ript..., javascript:..., javascript&colon;..., and newline/entity splits inside javascript;
dangerous non-javascript: schemes and payload containers, especially data:text/html and data:image/svg+xml in iframe, object, embed, img, input, and link;
CSS execution vectors such as expression(...), CSS url(javascript:...), and old browser bindings;
legacy event attributes not in the model's fixed list, e.g. ONCUT, ONROWSDELETE, ONURLFLIP, ONLOSECAPTURE, ONROWSEXIT, ONBEGIN, ONACTIVATE, ONSEEK, and ONREADYSTATECHANGE;
malformed or namespace variants such as <x:script>, <SCR\0IPT>, <SCRIPT/alert(10);/, and malformed iframe/onload constructs;
URL-like attributes outside its list, including cite, usemap, longdesc, and related historical attributes.

The verifier reported alerts in 9 of 28 concatenated batches. The first printed failed batches include exactly these classes: URL-encoded javascript%3A, data: SVG/script payloads, XML namespace script tags, old IE event handlers, CSS expression, malformed iframes, entity-encoded colons, UTF-7 payloads, and nul-byte script spelling.

The second failure is the clean-HTML invariant. Today's fast run did not round-trip clean HTML through BeautifulSoup; it preserved raw input formatting. The verifier expected BeautifulSoup-normalized output. Five clean samples therefore failed on harmless serialization differences such as parser-sorted attributes, <br> becoming <br/>, © becoming a literal copyright symbol, and indentation changes. This is a benchmark/test-contract mismatch with the prose instruction, but it is still a real scored failure: the reference shape is parser-based normalization plus sanitizer behavior, not literal byte preservation.

Verdict attribution

This is a model correctness and verification failure, not a harness artifact. The fast cap did not fire, there was no exception, and the verifier completed. The model solved the easy, visible version of the problem and stopped after its own small synthetic test. It did not discover that the benchmark's security surface is a browser-executed XSS corpus with hostile encodings, malformed HTML, historical event attributes, CSS vectors, and parser-normalized clean HTML expectations.

The preamble did help enough to produce a self-check, but the self-check was too narrow and then got overclaimed as "all tests pass." This is a strong example for the claim-vs-evidence nudge: the claim was based only on a model-authored happy-path fixture, not on the actual verifier or a corpus with adversarial encodings.

Follow-ups

For future attempts, this task needs a corpus-oriented smoke test: at minimum encoded javascript: variants, data: HTML/SVG payloads, CSS expression/url vectors, old on... event names, malformed script/iframe tags, and one clean sample compared against BeautifulSoup serialization.
Update the TODO hypothesis: today's run did not fail because it re-serialized clean HTML; it failed because it did not match the verifier's parser-normalized clean-HTML contract.
Full run narrative: runs/fast__qwen3.6-35b-a3b__20260706-113104/NOTES.md.