abhigyanpatwari/GitNexus · critical · ValueError

oracle sanitization left the benchmark harness visible

Error message

oracle sanitization left the benchmark harness visible

What it means

Post-condition: after rewriting HEAD and removing the harness, `git ls-tree -r HEAD -- eval/workflow_bench` must be empty and the worktree path must not exist or be a symlink. Any output or stray path means the harness is still visible to the model.

Source

Thrown at eval/workflow_bench/oracle_assets.py:447

    if hidden_tree:
        forbidden_objects.append((hidden_tree, "hidden harness tree"))
    for forbidden_object, label in forbidden_objects:
        probe = run_managed(
            ["git", "-C", str(root), "cat-file", "-e", forbidden_object],
            timeout=60,
        )
        if probe.ok:
            raise ValueError(f"oracle sanitization left the {label} recoverable")
        if probe.state != "exited" or probe.returncode not in {1, 128}:
            raise ValueError(f"oracle sanitization could not verify removal of the {label}")

    hidden_listing = _git_checked(
        root,
        ["ls-tree", "-r", "--name-only", "HEAD", "--", HIDDEN_HARNESS_PATH.as_posix()],
        timeout=60,
    )
    if hidden_listing or current.exists() or current.is_symlink():
        raise ValueError("oracle sanitization left the benchmark harness visible")
    if _git_checked(root, ["status", "--porcelain=v1", "--untracked-files=all"], timeout=60):
        raise ValueError("oracle sanitization did not produce a clean task snapshot")
    if _git_checked(root, ["rev-parse", "--verify", "HEAD^{commit}"], timeout=60) != sanitized_head:
        raise ValueError("oracle sanitization did not retain its parentless task snapshot")
    parents = _git_checked(root, ["show", "-s", "--format=%P", "HEAD"], timeout=60)
    if parents:
        raise ValueError("oracle sanitization snapshot unexpectedly retained parent history")
    if _git_checked(root, ["remote"], timeout=60):
        raise ValueError("oracle sanitization retained a repository remote")
    if logs.exists() or logs.is_symlink():
        raise ValueError("oracle sanitization retained reflog metadata")
    return sanitized_head


def _write_stage_file(stage_root: Path, item: OracleFileSnapshot) -> None:
    destination = stage_root.joinpath(*PurePosixPath(item.target).parts)
    destination.parent.mkdir(parents=True, mode=0o700, exist_ok=True)
    current = stage_root

View on GitHub (pinned to d540b00184)

Solutions

  1. Inspect tree: `git -C <clone> ls-tree -r HEAD -- eval/workflow_bench` and worktree: `ls -la <clone>/eval/workflow_bench`.
  2. Remove from index and worktree: `git -C <clone> rm -r --cached eval/workflow_bench && rm -rf <clone>/eval/workflow_bench` then rebuild the parentless commit.
  3. Ensure no setup command writes under eval/workflow_bench between sanitize and task start.
  4. Re-clone on a case-sensitive FS if case collisions are suspected.
Defensive patterns

Strategy: try-catch

Validate before calling

from pathlib import Path
from eval.workflow_bench.oracle_assets import HIDDEN_HARNESS_PATH
from eval.workflow_bench.process_control import run_checked

def harness_is_invisible(clone: Path) -> bool:
    out = run_checked(["git","-C",str(clone),"ls-tree","-r","--name-only","HEAD","--",HIDDEN_HARNESS_PATH.as_posix()], timeout=60).stdout_tail.strip()
    p = clone / HIDDEN_HARNESS_PATH
    return not out and not p.exists() and not p.is_symlink()

Type guard

def is_harness_still_visible(exc: BaseException) -> bool:
    return isinstance(exc, ValueError) and "left the benchmark harness visible" in str(exc)

Try / catch

try:
    oracle_assets.sanitize_clone_for_hidden_oracles(clone)
except ValueError as exc:
    quarantine(clone)
    raise AbortTask(str(exc)) from exc

Prevention

When it happens

Trigger: Triggered when `git rm -r -- eval/workflow_bench` did not actually remove the path from the new commit's tree, or the worktree path re-appeared (untracked file written between sanitize and verify).

Common situations: A setup command that recreates eval/workflow_bench after sanitization; git rm --ignore-unmatch silently no-op'd because the path was already absent from the index in a different form; case-insensitive FS quirks leaving a directory entry.

Related errors

AI-assisted analysis of abhigyanpatwari/GitNexus@d540b00184 (2026-08-12). Data as JSON: /api/errors/87b361c25d05a376. Report an issue: GitHub.