{"record":{"id":"0547c2c778fc4721","repo":"abhigyanpatwari/GitNexus","slug":"benchmark-harness-path-must-be-a-real-directory-be","errorCode":null,"errorMessage":"benchmark harness path must be a real directory before it can be hidden","messagePattern":"benchmark harness path must be a real directory before it can be hidden","errorType":"exception","errorClass":"SandboxError","httpStatus":null,"severity":"error","filePath":"eval/workflow_bench/runner.py","lineNumber":1097,"sourceCode":"                            except (OSError, SandboxError, ValueError) as exc:\n                                asset_snapshot_error = exc\n                                raise\n                        worktree = make_worktree(repo, task_sha, Path(trees))\n                        sanitized_head = sanitize_clone_for_hidden_oracles(worktree)\n                        graph_snapshot.materialize(worktree, sanitized_head=sanitized_head)\n                        dependency_mounts = stage_task_assets(\n                            task,\n                            repo=repo,\n                            clone=worktree,\n                            snapshot=asset_snapshot,\n                        )\n                        registry_mount = isolated_gitnexus_registry_mount(worktree, Path(trees))\n                        hidden_harness = worktree / \"eval\" / \"workflow_bench\"\n                        oracle_visibility_mounts: list[ReadOnlyMount] = []\n                        if hidden_harness.exists() or hidden_harness.is_symlink():\n                            hidden_metadata = hidden_harness.lstat()\n                            if stat.S_ISLNK(hidden_metadata.st_mode) or not stat.S_ISDIR(hidden_metadata.st_mode):\n                                raise SandboxError(\n                                    \"benchmark harness path must be a real directory before it can be hidden\"\n                                )\n                            oracle_visibility_mounts.append(\n                                ReadOnlyMount(\n                                    source=oracle_mask,\n                                    target=f\"{SANDBOX_WORKSPACE}/eval/workflow_bench\",\n                                )\n                            )\n                        execution_arm = CANDIDATE_ARMS.get(arm, arm)\n                        ce_mounts = ce_plugin_mounts_for_arm(execution_arm, ce_plugin_snapshot)\n                        with prepare_sandbox(\n                            clone=worktree,\n                            claude_bin=args.claude_bin,\n                            bwrap_bin=bwrap_bin,\n                            read_only_mounts=[\n                                *dependency_mounts,\n                                *runtime_mounts,\n                                registry_mount,","sourceCodeStart":1079,"sourceCodeEnd":1115,"githubUrl":"https://github.com/abhigyanpatwari/GitNexus/blob/d540b00184d71a896261ee02670da9a92d59d8f7/eval/workflow_bench/runner.py#L1079-L1115","documentation":"Raised while building the hidden-harness visibility mask: if eval/workflow_bench exists (or is a symlink) inside the worktree, it must be a real directory before the harness masks it with an oracle overlay. lstat (no follow) is used and any S_ISLNK or non-S_ISDIR mode is rejected, mirroring the symlink-guard discipline used across staging. The harness hides itself so the candidate cannot read its tests, but only if the path is a genuine directory.","triggerScenarios":"hidden_harness = worktree/eval/workflow_bench; hidden_harness.exists() or .is_symlink() is true, and hidden_harness.lstat().st_mode is a symlink or not S_ISDIR.","commonSituations":"A tracked symlink at eval/workflow_bench (e.g. pointing at a shared harness checkout); the path exists as a regular file or special node due to a bad checkout or prior run; a monorepo symlinks eval/workflow_bench from elsewhere; filesystem/NFS quirk changed the node type.","solutions":["Inspect the path: `ls -la <worktree>/eval/workflow_bench` and confirm it is a real directory.","If it is a symlink, remove it and check out the real directory (or restructure the repo so eval/workflow_bench is a directory, not a link).","Use a clean detached worktree so no stale artifact collides with the harness path.","Ensure no setup step replaces the directory with a symlink or file."],"exampleFix":"// before — eval/workflow_bench is a tracked symlink\neval/workflow_bench -> ../shared/workflow_bench\n// after — make it a real directory in the repo\ngit rm eval/workflow_bench\ngit mv shared/workflow_bench eval/workflow_bench","handlingStrategy":"validation","validationCode":"import stat\nfrom pathlib import Path\n\nhh = worktree / 'eval' / 'workflow_bench'\nif hh.exists() or hh.is_symlink():\n    m = hh.lstat().st_mode\n    assert not stat.S_ISLNK(m) and stat.S_ISDIR(m), (\n        f'{hh} must be a real directory before it can be hidden'\n    )","typeGuard":null,"tryCatchPattern":"from .proposer_sandbox import SandboxError\n\ntry:\n    # harness hiding happens inline during arm setup; catch SandboxError there\n    ...\nexcept SandboxError as exc:\n    if 'must be a real directory' in str(exc):\n        # eval/workflow_bench is a symlink/non-dir; fix the checkout\n        raise\n    raise","preventionTips":["Never symlink eval/workflow_bench in the repo.","Use a clean detached worktree per arm.","Ensure no setup step replaces the harness directory with a symlink/file."],"tags":["sandbox","symlink-guard","trust-boundary","harness","oracle"],"backgroundTag":null,"analyzedSha":"d540b00184d71a896261ee02670da9a92d59d8f7","analyzedAt":"2026-08-12T19:50:25.132Z","schemaVersion":2},"datasetVersion":"2026-08-12T23:17:12.415Z"}