abhigyanpatwari/GitNexus · critical · SandboxError

{label} found recoverable benchmark harness references

Error message

{label} found recoverable benchmark harness references

What it means

The marker-proof cypher query (MATCH (n) WHERE <markers> RETURN n LIMIT 1, and the relation equivalent) returned a non-empty result: a node or relationship in the built graph still contains one of the GRAPH_MARKERS substrings. The scrubber failed to eliminate all harness-referencing inputs, so shipping the graph would leak benchmark/oracle hints to an arm.

Source

Thrown at eval/workflow_bench/sanitized_graph.py:286

        raise SandboxError("bounded graph-query output was unavailable")
    return result.stdout_capture


def _marker_predicate(variable: str) -> str:
    literals = ("'" + marker.replace("\\", "\\\\").replace("'", "\\'") + "'" for marker in GRAPH_MARKERS)
    return " OR ".join(f"CAST({variable} AS STRING) CONTAINS {literal}" for literal in literals)


def _parse_empty_query(raw: bytes, *, label: str) -> None:
    try:
        payload = json.loads(raw.decode("utf-8", errors="strict"))
    except (UnicodeError, json.JSONDecodeError) as exc:
        raise SandboxError(f"{label} did not return strict JSON") from exc
    if payload == []:
        return
    if isinstance(payload, dict) and payload.get("row_count") == 0:
        return
    raise SandboxError(f"{label} found recoverable benchmark harness references")


def _scrub_and_verify_graph(prefix: Sequence[str]) -> None:
    node_predicate = _marker_predicate("n")
    relation_predicate = _marker_predicate("r")
    node_result = _run_graph_cli(
        prefix,
        ("cypher", f"MATCH (n) WHERE {node_predicate} RETURN n LIMIT 1", "-r", "benchmark-target", "--limit", "1"),
        timeout=GRAPH_QUERY_TIMEOUT_SECONDS,
        capture_stdout=True,
    )
    relation_result = _run_graph_cli(
        prefix,
        (
            "cypher",
            f"MATCH ()-[r]->() WHERE {relation_predicate} RETURN r LIMIT 1",
            "-r",
            "benchmark-target",

View on GitHub (pinned to d540b00184)

Solutions

1. Reproduce by running the cypher proof query manually and inspecting the returned node/relation property to find which marker matched.
2. Either add the leaked token to GRAPH_MARKERS or ensure the offending source is removed/excluded by _scrub_source_references.
3. Reconcile the scrub skip vs the index cap: files > 512 KiB are skipped by the scrubber but admitted by --max-file-size 512, letting markers slip through - raise the scrub file cap or exclude such files.
4. Rebuild the sanitized graph from scratch.

import os
from workflow_bench.sanitized_graph import GRAPH_MARKERS, MAX_GRAPH_SCRUB_FILE_BYTES

def scan_oversize_files_for_markers(root):
    """The scrubber skips files > 512 KiB; pre-scan them so markers cannot slip through."""
    marker_bytes = [m.encode() for m in GRAPH_MARKERS]
    leaks = []
    for dirpath, dirnames, files in os.walk(root):
        parts = os.path.relpath(dirpath, root).split(os.sep)
        if parts and parts[0] in {".git", ".gitnexus"}:
            dirnames[:] = []
            continue
        for name in files:
            p = os.path.join(dirpath, name)
            try:
                size = os.path.getsize(p)
            except OSError:
                continue
            if size <= MAX_GRAPH_SCRUB_FILE_BYTES:
                continue
            with open(p, "rb") as fh:
                head = fh.read(MAX_GRAPH_SCRUB_FILE_BYTES)
            if any(m in head for m in marker_bytes):
                leaks.append(p)
    if leaks:
        raise RuntimeError(f"marker-bearing files exceed scrub size cap: {leaks}")

When it happens

Trigger: _scrub_and_verify_graph runs after analyze and either the node proof or the relation proof returns a row: scrubbing by path and by content missed a file whose stored or generated content references a marker such as 'eval/workflow_bench' or 'tasks.scenarios.yaml'.

Common situations: A new harness token was added to the codebase but not to GRAPH_MARKERS; a file larger than MAX_GRAPH_SCRUB_FILE_BYTES (512 KiB) embedded a marker and was skipped by the scrubber yet still indexed at --max-file-size 512; a generated/minified artifact produced marker text during indexing.

Related errors

• sandbox_copy cannot import prebuilt graph or harness data: {
• sandbox dependency cannot expose prebuilt graph or harness d
• target .gitnexus path must be a real directory before graph
• {label} must be a real non-symlink directory: {path}
• {label} contains an unsafe path component: {relative}

AI-assisted analysis of abhigyanpatwari/GitNexus@d540b00184 (2026-08-12). Data as JSON: /api/errors/d5f079a7220f57f8. Report an issue: GitHub.