abhigyanpatwari/GitNexus · critical · ValueError

oracle file changed during verification: {item.target}

Error message

oracle file changed during verification: {item.target}

What it means

Final payload check in _verify_staged_oracle: after the model exits, each staged oracle file is re-read and compared byte-for-byte to item.payload. Any difference — content edit, truncation, replacement, or fs write that did not fsync identically — is treated as oracle tampering and the run is voided.

Source

Thrown at eval/workflow_bench/oracle_assets.py:504

    finally:
        os.close(descriptor)


def _verify_staged_oracle(stage_root: Path, snapshot: TaskOracleSnapshot) -> None:
    root_metadata = stage_root.lstat()
    if stat.S_ISLNK(root_metadata.st_mode) or not stat.S_ISDIR(root_metadata.st_mode):
        raise ValueError("oracle stage root changed during verification")
    for item in snapshot.files:
        relative = PurePosixPath(item.target)
        current = stage_root
        for part in relative.parts[:-1]:
            current /= part
            metadata = current.lstat()
            if stat.S_ISLNK(metadata.st_mode) or not stat.S_ISDIR(metadata.st_mode):
                raise ValueError(f"oracle stage parent changed during verification: {item.target}")
        observed = _read_oracle_file(stage_root, relative)
        if observed != item.payload:
            raise ValueError(f"oracle file changed during verification: {item.target}")


@contextmanager
def staged_task_oracle(worktree: Path, snapshot: TaskOracleSnapshot) -> Iterator[Path]:
    """Materialize a private random oracle root only after the model exits."""

    root = worktree.expanduser().absolute()
    metadata = root.lstat()
    if stat.S_ISLNK(metadata.st_mode) or not stat.S_ISDIR(metadata.st_mode) or root.resolve(strict=True) != root:
        raise ValueError(f"oracle worktree must be a real non-symlink directory: {root}")
    stage_root = root / f".wfbench-oracle-{secrets.token_hex(16)}"
    stage_root.mkdir(mode=0o700)
    stage_root.chmod(0o700)
    primary: BaseException | None = None
    try:
        for item in snapshot.files:
            _write_stage_file(stage_root, item)
        yield stage_root

View on GitHub (pinned to d540b00184)

Solutions

  1. Use the {item.target} from the message and diff the staged file against the expected payload to see exactly what changed.
  2. Confirm no editor, linter, watcher, or sync agent (Dropbox, rsync, IDE) is touching the worktree during the run.
  3. Move the worktree to a local tmpfs/ext4 volume and re-run.
  4. If the agent under test is the cause, fix the agent — this is detected tampering by design.
Defensive patterns

Strategy: validation

Validate before calling

# Snapshot oracle file hashes before staging; the harness already does this
# internally, but you can log them for post-mortem.
import hashlib
from pathlib import Path

def log_expected(stage_root: Path, files):
    for item in files:
        h = hashlib.sha256(item.payload).hexdigest()
        log.debug('oracle %s expected sha256=%s', item.target, h)

Try / catch

try:
    with staged_task_oracle(worktree, snapshot) as stage:
        run_model(stage)
except ValueError as exc:
    msg = str(exc)
    if 'oracle file changed during verification' in msg:
        target = msg.split(': ', 1)[-1]
        log.security('model tampered with oracle file %s', target)
        mark_cheating()
        raise

Prevention

When it happens

Trigger: observed = _read_oracle_file(stage_root, relative); observed != item.payload. The model (or a sibling) wrote to the staged oracle file, or the underlying FS changed the bytes (rare, but possible on CoW/synced folders).

Common situations: Agent under test edits oracle files to influence scoring; a linter/formatter rewrites the directory on save; IDE auto-save touches files; running on a network FS where fsync semantics are weak; pre-existing file with same name was not exclusive-created (should be impossible given O_EXCL, but a kernel bug or overlayfs could cause it).

Related errors

AI-assisted analysis of abhigyanpatwari/GitNexus@d540b00184 (2026-08-12). Data as JSON: /api/errors/7c83aa37ed194404. Report an issue: GitHub.