abhigyanpatwari/GitNexus · critical · ValueError

oracle stage root changed during verification

Error message

oracle stage root changed during verification

What it means

Raised by _verify_staged_oracle after the benchmark surrenders control to the model: the staged oracle root is re-stat()ed and must still be a real (non-symlink) directory that this harness created. A mismatch means something — the model under test, a concurrent process, or a filesystem race — replaced the stage root. This is a deliberate integrity violation rather than a normal control-flow error.

Source

Thrown at eval/workflow_bench/oracle_assets.py:493

        0o400,
    )
    try:
        view = memoryview(item.payload)
        while view:
            written = os.write(descriptor, view)
            if written <= 0:
                raise OSError("short write while staging oracle")
            view = view[written:]
        os.fchmod(descriptor, 0o400)
        os.fsync(descriptor)
    finally:
        os.close(descriptor)


def _verify_staged_oracle(stage_root: Path, snapshot: TaskOracleSnapshot) -> None:
    root_metadata = stage_root.lstat()
    if stat.S_ISLNK(root_metadata.st_mode) or not stat.S_ISDIR(root_metadata.st_mode):
        raise ValueError("oracle stage root changed during verification")
    for item in snapshot.files:
        relative = PurePosixPath(item.target)
        current = stage_root
        for part in relative.parts[:-1]:
            current /= part
            metadata = current.lstat()
            if stat.S_ISLNK(metadata.st_mode) or not stat.S_ISDIR(metadata.st_mode):
                raise ValueError(f"oracle stage parent changed during verification: {item.target}")
        observed = _read_oracle_file(stage_root, relative)
        if observed != item.payload:
            raise ValueError(f"oracle file changed during verification: {item.target}")


@contextmanager
def staged_task_oracle(worktree: Path, snapshot: TaskOracleSnapshot) -> Iterator[Path]:
    """Materialize a private random oracle root only after the model exits."""

    root = worktree.expanduser().absolute()

View on GitHub (pinned to d540b00184)

Solutions

  1. Inspect stage_root path in the traceback and confirm no external process is mutating the worktree (lsof / ps for sweepers).
  2. Re-run the benchmark against a clean worktree on a local filesystem (tmpfs or ext4/xfs), not NFS or a synced folder.
  3. Audit the agent under test for code that writes inside .wfbench-oracle-* — that is forbidden tampering and must be fixed in the agent.
  4. If running concurrently, serialize staged_task_oracle invocations so each owns its worktree exclusively.
Defensive patterns

Strategy: validation

Validate before calling

# Pre-flight: ensure nothing else can write the worktree before staging.
import os, stat
from pathlib import Path

def assert_quiet_real_dir(worktree: Path) -> None:
    st = worktree.lstat()
    assert stat.S_ISDIR(st.st_mode) and not stat.S_ISLNK(st.st_mode), \
        f'worktree {worktree} is not a real directory'
    # Optional: take an exclusive lock so concurrent runs cannot interfere.
    fd = os.open(worktree, os.O_RDONLY)
    import fcntl
    fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)  # raises if contended

Try / catch

try:
    with staged_task_oracle(worktree, snapshot) as stage:
        run_model(stage)
except ValueError as exc:
    if 'changed during verification' in str(exc):
        log.critical('oracle tampering detected: %s', exc)
        mark_run_invalid()
    else:
        raise

Prevention

When it happens

Trigger: Called via staged_task_oracle() after the `yield` returns, when stage_root.lstat() shows S_ISLNK or not S_ISDIR. Happens if the model (or any sibling process) renamed/replaced/unlinked the .wfbench-oracle-<hex> directory while it had access.

Common situations: An agent under benchmark tries to tamper with oracle files to cheat; a misconfigured cleanup task (cron, IDE watcher, `make clean`) sweeps the worktree; running the benchmark on a network filesystem where directory entries are not stable; a leftover foreground `rm -rf` from a prior run.

Related errors

AI-assisted analysis of abhigyanpatwari/GitNexus@d540b00184 (2026-08-12). Data as JSON: /api/errors/1d9dc6ab71f30722. Report an issue: GitHub.