{"record":{"id":"d5f079a7220f57f8","repo":"abhigyanpatwari/GitNexus","slug":"label-found-recoverable-benchmark-harness-refere","errorCode":null,"errorMessage":"{label} found recoverable benchmark harness references","messagePattern":"(.+?) found recoverable benchmark harness references","errorType":"exception","errorClass":"SandboxError","httpStatus":null,"severity":"critical","filePath":"eval/workflow_bench/sanitized_graph.py","lineNumber":286,"sourceCode":"        raise SandboxError(\"bounded graph-query output was unavailable\")\n    return result.stdout_capture\n\n\ndef _marker_predicate(variable: str) -> str:\n    literals = (\"'\" + marker.replace(\"\\\\\", \"\\\\\\\\\").replace(\"'\", \"\\\\'\") + \"'\" for marker in GRAPH_MARKERS)\n    return \" OR \".join(f\"CAST({variable} AS STRING) CONTAINS {literal}\" for literal in literals)\n\n\ndef _parse_empty_query(raw: bytes, *, label: str) -> None:\n    try:\n        payload = json.loads(raw.decode(\"utf-8\", errors=\"strict\"))\n    except (UnicodeError, json.JSONDecodeError) as exc:\n        raise SandboxError(f\"{label} did not return strict JSON\") from exc\n    if payload == []:\n        return\n    if isinstance(payload, dict) and payload.get(\"row_count\") == 0:\n        return\n    raise SandboxError(f\"{label} found recoverable benchmark harness references\")\n\n\ndef _scrub_and_verify_graph(prefix: Sequence[str]) -> None:\n    node_predicate = _marker_predicate(\"n\")\n    relation_predicate = _marker_predicate(\"r\")\n    node_result = _run_graph_cli(\n        prefix,\n        (\"cypher\", f\"MATCH (n) WHERE {node_predicate} RETURN n LIMIT 1\", \"-r\", \"benchmark-target\", \"--limit\", \"1\"),\n        timeout=GRAPH_QUERY_TIMEOUT_SECONDS,\n        capture_stdout=True,\n    )\n    relation_result = _run_graph_cli(\n        prefix,\n        (\n            \"cypher\",\n            f\"MATCH ()-[r]->() WHERE {relation_predicate} RETURN r LIMIT 1\",\n            \"-r\",\n            \"benchmark-target\",","sourceCodeStart":268,"sourceCodeEnd":304,"githubUrl":"https://github.com/abhigyanpatwari/GitNexus/blob/d540b00184d71a896261ee02670da9a92d59d8f7/eval/workflow_bench/sanitized_graph.py#L268-L304","documentation":"The marker-proof cypher query (MATCH (n) WHERE <markers> RETURN n LIMIT 1, and the relation equivalent) returned a non-empty result: a node or relationship in the built graph still contains one of the GRAPH_MARKERS substrings. The scrubber failed to eliminate all harness-referencing inputs, so shipping the graph would leak benchmark/oracle hints to an arm.","triggerScenarios":"_scrub_and_verify_graph runs after analyze and either the node proof or the relation proof returns a row: scrubbing by path and by content missed a file whose stored or generated content references a marker such as 'eval/workflow_bench' or 'tasks.scenarios.yaml'.","commonSituations":"A new harness token was added to the codebase but not to GRAPH_MARKERS; a file larger than MAX_GRAPH_SCRUB_FILE_BYTES (512 KiB) embedded a marker and was skipped by the scrubber yet still indexed at --max-file-size 512; a generated/minified artifact produced marker text during indexing.","solutions":["Reproduce by running the cypher proof query manually and inspecting the returned node/relation property to find which marker matched.","Either add the leaked token to GRAPH_MARKERS or ensure the offending source is removed/excluded by _scrub_source_references.","Reconcile the scrub skip vs the index cap: files > 512 KiB are skipped by the scrubber but admitted by --max-file-size 512, letting markers slip through - raise the scrub file cap or exclude such files.","Rebuild the sanitized graph from scratch."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"import os\nfrom workflow_bench.sanitized_graph import GRAPH_MARKERS, MAX_GRAPH_SCRUB_FILE_BYTES\n\ndef scan_oversize_files_for_markers(root):\n    \"\"\"The scrubber skips files > 512 KiB; pre-scan them so markers cannot slip through.\"\"\"\n    marker_bytes = [m.encode() for m in GRAPH_MARKERS]\n    leaks = []\n    for dirpath, dirnames, files in os.walk(root):\n        parts = os.path.relpath(dirpath, root).split(os.sep)\n        if parts and parts[0] in {\".git\", \".gitnexus\"}:\n            dirnames[:] = []\n            continue\n        for name in files:\n            p = os.path.join(dirpath, name)\n            try:\n                size = os.path.getsize(p)\n            except OSError:\n                continue\n            if size <= MAX_GRAPH_SCRUB_FILE_BYTES:\n                continue\n            with open(p, \"rb\") as fh:\n                head = fh.read(MAX_GRAPH_SCRUB_FILE_BYTES)\n            if any(m in head for m in marker_bytes):\n                leaks.append(p)\n    if leaks:\n        raise RuntimeError(f\"marker-bearing files exceed scrub size cap: {leaks}\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep GRAPH_MARKERS in lock-step with every harness token added to the codebase.","Reconcile MAX_GRAPH_SCRUB_FILE_BYTES with the analyze --max-file-size so no marker-bearing file is indexed but unscrubbed.","Never disable the marker-proof queries; treat this error as a release blocker."],"tags":["security","benchmark-integrity","sanitization","gitnexus","workflow-bench"],"backgroundTag":null,"analyzedSha":"d540b00184d71a896261ee02670da9a92d59d8f7","analyzedAt":"2026-08-12T19:50:25.132Z","schemaVersion":2},"datasetVersion":"2026-08-12T23:17:12.415Z"}