← task board · runs · Qwen 3.6 35B A3B · pi

sparql-university - valid SPARQL, empty result set from an uncorrelated subquery

Status: FAIL (2/3 subtests) in fast__qwen3.6-35b-a3b__20260706-113104. The fast-fail cap was 240s, but it did not fire: agent execution ended normally after ~2m04s, then verification failed only test_sparql_query_results. Baseline: also FAIL in the 20260703 suite and 20260705 smoke runs, each as the same one-assertion semantic miss. Earlier attempts returned partial country sets; this latest attempt regressed to an empty result set.

What the task wants

Write /app/solution.sparql for a university RDF graph. The query must return full professors who:

  1. have a role that is a full professor, not an assistant professor;
  2. work in at least one department whose university is in an EU country as of 2025-08-16;
  3. work in at least one department with more than 10 currently enrolled students in classes taught by that department.

The returned row must be:

SELECT ?professorName (GROUP_CONCAT(DISTINCT ?country; separator=", ") AS ?countries)

Important detail: ?countries is all countries where the qualifying professor currently works, not just EU countries and not just countries attached to the large-enrollment department. The hidden fixture checks:

Trajectory

The model did read /app/university_graph.ttl, wrote a 51-line /app/solution.sparql, and made several verification attempts. It did not have rdflib available in the agent container, so its only executed validation was a SPARQL parser check via sparqljs. That proved syntax, not result semantics.

The final narrative then manually traced the public /app graph and concluded that the expected rows were Orfeas Menis, Pepe Attanasio, and Aristotle Tympas. That was already an insufficient check: the verifier runs against /tests/university_graph_test.ttl, where the professor names and one qualifying multi-country case differ. A general query should still pass the hidden fixture, but this one did not execute semantically as intended.

The bug

The verifier output is precise:

Got: set()
Expected: {('Aristotle Tympas', 'GR'),
           ('Giorgos Stamou', 'GR, US'),
           ('Chrysoula Zerva', 'GR, PT'),
           ('Alex Dimakis', 'CH, ES, US')}

The written query parses and runs, but its second FILTER EXISTS contains a subquery that tries to correlate back to the outer ?professor:

FILTER EXISTS {
  SELECT ?d
  WHERE {
    ?professorWorksIn uni:worksIn ?d .
    ...
    FILTER(?professorWorksIn = ?professor)
  }
  GROUP BY ?d
  HAVING(COUNT(DISTINCT ?student) > 10)
}

In rdflib's SPARQL evaluation, that subquery does not bind against the outer ?professor in the way the model expected. The inner filter therefore never produces a binding that can satisfy FILTER EXISTS, so every outer professor row is eliminated and the final result is empty.

The reference solution avoids that trap by binding independent department roles in the outer query:

That shape also handles Alex Dimakis: the EU department is Robotics_UPM (ES), the large-enrollment department is Engineering_Berkeley (US), and the output must still include Physics_ETH (CH).

Verdict attribution

This is a model correctness failure, not a harness or timeout artifact. The file exists, the query is syntactically valid, and the verifier could execute it. The semantic check failed because the model relied on parser-only validation plus manual reasoning over the public graph, then wrote a correlated subquery pattern that rdflib evaluated to no rows.

The preamble did what H4 predicted: it did not move this task. The task needs real result-set validation against a SPARQL engine, not generic "verify before done" scaffolding. Stop treating sparql-university as a likely preamble flip; use it as a stable semantic-query control.

Follow-ups