abhigyanpatwari/GitNexus · error · ValueError

benchmark harness checkout must contain only real directorie

Error message

benchmark harness checkout must contain only real directories

What it means

Raised during the same worktree walk as [340]. A path component of eval/workflow_bench lstat'd successfully but is a symlink or not a directory. The guard prevents a symlink from redirecting the subsequent `git rm -r -- eval/workflow_bench` onto an unrelated location (path-substitution / TOCTOU attack).

Source

Thrown at eval/workflow_bench/oracle_assets.py:307

    if not hidden_tree_result.ok:
        raise ValueError("cannot inspect the clone for committed benchmark harness data")
    hidden_tree = hidden_tree_result.stdout_tail.strip()
    if hidden_tree and (
        len(hidden_tree) not in {40, 64} or any(character not in "0123456789abcdefABCDEF" for character in hidden_tree)
    ):
        raise ValueError("committed benchmark harness is not a single bounded tree")

    current = root.joinpath(*HIDDEN_HARNESS_PATH.parts)
    if hidden_tree:
        parent = root
        for part in HIDDEN_HARNESS_PATH.parts:
            parent /= part
            try:
                metadata = parent.lstat()
            except OSError as exc:
                raise ValueError("committed benchmark harness is missing from the clone checkout") from exc
            if stat.S_ISLNK(metadata.st_mode) or not stat.S_ISDIR(metadata.st_mode):
                raise ValueError("benchmark harness checkout must contain only real directories")
    elif current.exists() or current.is_symlink():
        raise ValueError("untracked benchmark harness data blocks oracle sanitization")

    _git_checked(
        root,
        [
            "rm",
            "-r",
            "--force",
            "--quiet",
            "--ignore-unmatch",
            "--",
            HIDDEN_HARNESS_PATH.as_posix(),
        ],
        timeout=120,
    )
    sanitized_tree = _git_checked(root, ["write-tree"], timeout=60)
    deterministic_git_env = {

View on GitHub (pinned to d540b00184)

Solutions

  1. Inspect `ls -la <clone>/eval` and `<clone>/eval/workflow_bench` for symlinks or files.
  2. Remove the offending entry and restore from HEAD: `git -C <clone> checkout -- eval/workflow_bench` (after deleting the link/file).
  3. Re-clone from a trusted source if the worktree looks crafted.

Example fix

// before: eval/workflow_bench -> /etc/passwd (symlink)
rm <clone>/eval/workflow_bench
git -C <clone> checkout HEAD -- eval/workflow_bench
// after: real directory restored, lstat reports S_ISDIR
Defensive patterns

Strategy: validation

Validate before calling

import stat
from pathlib import Path

def harness_parts_are_real_dirs(clone: Path, parts=("eval", "workflow_bench")) -> bool:
    cur = clone
    for part in parts:
        cur = cur / part
        try:
            st = cur.lstat()
        except OSError:
            return False
        if stat.S_ISLNK(st.st_mode) or not stat.S_ISDIR(st.st_mode):
            return False
    return True

Type guard

def is_unsafe_harness_dir(exc: BaseException) -> bool:
    return isinstance(exc, ValueError) and "must contain only real directories" in str(exc)

Try / catch

try:
    oracle_assets.sanitize_clone_for_hidden_oracles(clone)
except ValueError as exc:
    quarantine(clone)
    raise AbortTask(str(exc)) from exc

Prevention

When it happens

Trigger: Triggered when eval/ or eval/workflow_bench/ in the clone worktree is a symbolic link or a regular file rather than a real directory, even though HEAD tracks it as a tree.

Common situations: A hand-modified clone where someone replaced the directory with a symlink; a crafted malicious clone; leftover artifact from a failed merge that left a file named 'workflow_bench'.

Related errors

AI-assisted analysis of abhigyanpatwari/GitNexus@d540b00184 (2026-08-12). Data as JSON: /api/errors/0885f2054f467123. Report an issue: GitHub.