{"record":{"id":"87b361c25d05a376","repo":"abhigyanpatwari/GitNexus","slug":"oracle-sanitization-left-the-benchmark-harness-vis","errorCode":null,"errorMessage":"oracle sanitization left the benchmark harness visible","messagePattern":"oracle sanitization left the benchmark harness visible","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"eval/workflow_bench/oracle_assets.py","lineNumber":447,"sourceCode":"    if hidden_tree:\n        forbidden_objects.append((hidden_tree, \"hidden harness tree\"))\n    for forbidden_object, label in forbidden_objects:\n        probe = run_managed(\n            [\"git\", \"-C\", str(root), \"cat-file\", \"-e\", forbidden_object],\n            timeout=60,\n        )\n        if probe.ok:\n            raise ValueError(f\"oracle sanitization left the {label} recoverable\")\n        if probe.state != \"exited\" or probe.returncode not in {1, 128}:\n            raise ValueError(f\"oracle sanitization could not verify removal of the {label}\")\n\n    hidden_listing = _git_checked(\n        root,\n        [\"ls-tree\", \"-r\", \"--name-only\", \"HEAD\", \"--\", HIDDEN_HARNESS_PATH.as_posix()],\n        timeout=60,\n    )\n    if hidden_listing or current.exists() or current.is_symlink():\n        raise ValueError(\"oracle sanitization left the benchmark harness visible\")\n    if _git_checked(root, [\"status\", \"--porcelain=v1\", \"--untracked-files=all\"], timeout=60):\n        raise ValueError(\"oracle sanitization did not produce a clean task snapshot\")\n    if _git_checked(root, [\"rev-parse\", \"--verify\", \"HEAD^{commit}\"], timeout=60) != sanitized_head:\n        raise ValueError(\"oracle sanitization did not retain its parentless task snapshot\")\n    parents = _git_checked(root, [\"show\", \"-s\", \"--format=%P\", \"HEAD\"], timeout=60)\n    if parents:\n        raise ValueError(\"oracle sanitization snapshot unexpectedly retained parent history\")\n    if _git_checked(root, [\"remote\"], timeout=60):\n        raise ValueError(\"oracle sanitization retained a repository remote\")\n    if logs.exists() or logs.is_symlink():\n        raise ValueError(\"oracle sanitization retained reflog metadata\")\n    return sanitized_head\n\n\ndef _write_stage_file(stage_root: Path, item: OracleFileSnapshot) -> None:\n    destination = stage_root.joinpath(*PurePosixPath(item.target).parts)\n    destination.parent.mkdir(parents=True, mode=0o700, exist_ok=True)\n    current = stage_root","sourceCodeStart":429,"sourceCodeEnd":465,"githubUrl":"https://github.com/abhigyanpatwari/GitNexus/blob/d540b00184d71a896261ee02670da9a92d59d8f7/eval/workflow_bench/oracle_assets.py#L429-L465","documentation":"Post-condition: after rewriting HEAD and removing the harness, `git ls-tree -r HEAD -- eval/workflow_bench` must be empty and the worktree path must not exist or be a symlink. Any output or stray path means the harness is still visible to the model.","triggerScenarios":"Triggered when `git rm -r -- eval/workflow_bench` did not actually remove the path from the new commit's tree, or the worktree path re-appeared (untracked file written between sanitize and verify).","commonSituations":"A setup command that recreates eval/workflow_bench after sanitization; git rm --ignore-unmatch silently no-op'd because the path was already absent from the index in a different form; case-insensitive FS quirks leaving a directory entry.","solutions":["Inspect tree: `git -C <clone> ls-tree -r HEAD -- eval/workflow_bench` and worktree: `ls -la <clone>/eval/workflow_bench`.","Remove from index and worktree: `git -C <clone> rm -r --cached eval/workflow_bench && rm -rf <clone>/eval/workflow_bench` then rebuild the parentless commit.","Ensure no setup command writes under eval/workflow_bench between sanitize and task start.","Re-clone on a case-sensitive FS if case collisions are suspected."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"from pathlib import Path\nfrom eval.workflow_bench.oracle_assets import HIDDEN_HARNESS_PATH\nfrom eval.workflow_bench.process_control import run_checked\n\ndef harness_is_invisible(clone: Path) -> bool:\n    out = run_checked([\"git\",\"-C\",str(clone),\"ls-tree\",\"-r\",\"--name-only\",\"HEAD\",\"--\",HIDDEN_HARNESS_PATH.as_posix()], timeout=60).stdout_tail.strip()\n    p = clone / HIDDEN_HARNESS_PATH\n    return not out and not p.exists() and not p.is_symlink()\n","typeGuard":"def is_harness_still_visible(exc: BaseException) -> bool:\n    return isinstance(exc, ValueError) and \"left the benchmark harness visible\" in str(exc)\n","tryCatchPattern":"try:\n    oracle_assets.sanitize_clone_for_hidden_oracles(clone)\nexcept ValueError as exc:\n    quarantine(clone)\n    raise AbortTask(str(exc)) from exc\n","preventionTips":["Do not let task setup commands write under eval/workflow_bench.","Sanitize on a case-sensitive filesystem."],"tags":["git","ls-tree","worktree","data-leak","oracle","sanitization","postcondition","invariant"],"backgroundTag":null,"analyzedSha":"d540b00184d71a896261ee02670da9a92d59d8f7","analyzedAt":"2026-08-12T19:50:25.132Z","schemaVersion":2},"datasetVersion":"2026-08-12T23:17:12.415Z"}