abhigyanpatwari/GitNexus · error · SandboxError

benchmark harness path must be a real directory before it ca

Error message

benchmark harness path must be a real directory before it can be hidden

What it means

Raised while building the hidden-harness visibility mask: if eval/workflow_bench exists (or is a symlink) inside the worktree, it must be a real directory before the harness masks it with an oracle overlay. lstat (no follow) is used and any S_ISLNK or non-S_ISDIR mode is rejected, mirroring the symlink-guard discipline used across staging. The harness hides itself so the candidate cannot read its tests, but only if the path is a genuine directory.

Source

Thrown at eval/workflow_bench/runner.py:1097

                            except (OSError, SandboxError, ValueError) as exc:
                                asset_snapshot_error = exc
                                raise
                        worktree = make_worktree(repo, task_sha, Path(trees))
                        sanitized_head = sanitize_clone_for_hidden_oracles(worktree)
                        graph_snapshot.materialize(worktree, sanitized_head=sanitized_head)
                        dependency_mounts = stage_task_assets(
                            task,
                            repo=repo,
                            clone=worktree,
                            snapshot=asset_snapshot,
                        )
                        registry_mount = isolated_gitnexus_registry_mount(worktree, Path(trees))
                        hidden_harness = worktree / "eval" / "workflow_bench"
                        oracle_visibility_mounts: list[ReadOnlyMount] = []
                        if hidden_harness.exists() or hidden_harness.is_symlink():
                            hidden_metadata = hidden_harness.lstat()
                            if stat.S_ISLNK(hidden_metadata.st_mode) or not stat.S_ISDIR(hidden_metadata.st_mode):
                                raise SandboxError(
                                    "benchmark harness path must be a real directory before it can be hidden"
                                )
                            oracle_visibility_mounts.append(
                                ReadOnlyMount(
                                    source=oracle_mask,
                                    target=f"{SANDBOX_WORKSPACE}/eval/workflow_bench",
                                )
                            )
                        execution_arm = CANDIDATE_ARMS.get(arm, arm)
                        ce_mounts = ce_plugin_mounts_for_arm(execution_arm, ce_plugin_snapshot)
                        with prepare_sandbox(
                            clone=worktree,
                            claude_bin=args.claude_bin,
                            bwrap_bin=bwrap_bin,
                            read_only_mounts=[
                                *dependency_mounts,
                                *runtime_mounts,
                                registry_mount,

View on GitHub (pinned to d540b00184)

Solutions

Inspect the path: `ls -la <worktree>/eval/workflow_bench` and confirm it is a real directory.
If it is a symlink, remove it and check out the real directory (or restructure the repo so eval/workflow_bench is a directory, not a link).
Use a clean detached worktree so no stale artifact collides with the harness path.
Ensure no setup step replaces the directory with a symlink or file.

Example fix

// before — eval/workflow_bench is a tracked symlink
eval/workflow_bench -> ../shared/workflow_bench
// after — make it a real directory in the repo
git rm eval/workflow_bench
git mv shared/workflow_bench eval/workflow_bench

Defensive patterns

Strategy: validation

Validate before calling

import stat
from pathlib import Path

hh = worktree / 'eval' / 'workflow_bench'
if hh.exists() or hh.is_symlink():
    m = hh.lstat().st_mode
    assert not stat.S_ISLNK(m) and stat.S_ISDIR(m), (
        f'{hh} must be a real directory before it can be hidden'
    )

Try / catch

from .proposer_sandbox import SandboxError

try:
    # harness hiding happens inline during arm setup; catch SandboxError there
    ...
except SandboxError as exc:
    if 'must be a real directory' in str(exc):
        # eval/workflow_bench is a symlink/non-dir; fix the checkout
        raise
    raise

Prevention

Never symlink eval/workflow_bench in the repo.
Use a clean detached worktree per arm.
Ensure no setup step replaces the harness directory with a symlink/file.

When it happens

Trigger: hidden_harness = worktree/eval/workflow_bench; hidden_harness.exists() or .is_symlink() is true, and hidden_harness.lstat().st_mode is a symlink or not S_ISDIR.

Common situations: A tracked symlink at eval/workflow_bench (e.g. pointing at a shared harness checkout); the path exists as a regular file or special node due to a bad checkout or prior run; a monorepo symlinks eval/workflow_bench from elsewhere; filesystem/NFS quirk changed the node type.

Related errors

AI-assisted analysis of abhigyanpatwari/GitNexus@d540b00184 (2026-08-12). Data as JSON: /api/errors/0547c2c778fc4721. Report an issue: GitHub.