{"record":{"id":"7c83aa37ed194404","repo":"abhigyanpatwari/GitNexus","slug":"oracle-file-changed-during-verification-item-tar","errorCode":null,"errorMessage":"oracle file changed during verification: {item.target}","messagePattern":"oracle file changed during verification: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"eval/workflow_bench/oracle_assets.py","lineNumber":504,"sourceCode":"    finally:\n        os.close(descriptor)\n\n\ndef _verify_staged_oracle(stage_root: Path, snapshot: TaskOracleSnapshot) -> None:\n    root_metadata = stage_root.lstat()\n    if stat.S_ISLNK(root_metadata.st_mode) or not stat.S_ISDIR(root_metadata.st_mode):\n        raise ValueError(\"oracle stage root changed during verification\")\n    for item in snapshot.files:\n        relative = PurePosixPath(item.target)\n        current = stage_root\n        for part in relative.parts[:-1]:\n            current /= part\n            metadata = current.lstat()\n            if stat.S_ISLNK(metadata.st_mode) or not stat.S_ISDIR(metadata.st_mode):\n                raise ValueError(f\"oracle stage parent changed during verification: {item.target}\")\n        observed = _read_oracle_file(stage_root, relative)\n        if observed != item.payload:\n            raise ValueError(f\"oracle file changed during verification: {item.target}\")\n\n\n@contextmanager\ndef staged_task_oracle(worktree: Path, snapshot: TaskOracleSnapshot) -> Iterator[Path]:\n    \"\"\"Materialize a private random oracle root only after the model exits.\"\"\"\n\n    root = worktree.expanduser().absolute()\n    metadata = root.lstat()\n    if stat.S_ISLNK(metadata.st_mode) or not stat.S_ISDIR(metadata.st_mode) or root.resolve(strict=True) != root:\n        raise ValueError(f\"oracle worktree must be a real non-symlink directory: {root}\")\n    stage_root = root / f\".wfbench-oracle-{secrets.token_hex(16)}\"\n    stage_root.mkdir(mode=0o700)\n    stage_root.chmod(0o700)\n    primary: BaseException | None = None\n    try:\n        for item in snapshot.files:\n            _write_stage_file(stage_root, item)\n        yield stage_root","sourceCodeStart":486,"sourceCodeEnd":522,"githubUrl":"https://github.com/abhigyanpatwari/GitNexus/blob/d540b00184d71a896261ee02670da9a92d59d8f7/eval/workflow_bench/oracle_assets.py#L486-L522","documentation":"Final payload check in _verify_staged_oracle: after the model exits, each staged oracle file is re-read and compared byte-for-byte to item.payload. Any difference — content edit, truncation, replacement, or fs write that did not fsync identically — is treated as oracle tampering and the run is voided.","triggerScenarios":"observed = _read_oracle_file(stage_root, relative); observed != item.payload. The model (or a sibling) wrote to the staged oracle file, or the underlying FS changed the bytes (rare, but possible on CoW/synced folders).","commonSituations":"Agent under test edits oracle files to influence scoring; a linter/formatter rewrites the directory on save; IDE auto-save touches files; running on a network FS where fsync semantics are weak; pre-existing file with same name was not exclusive-created (should be impossible given O_EXCL, but a kernel bug or overlayfs could cause it).","solutions":["Use the {item.target} from the message and diff the staged file against the expected payload to see exactly what changed.","Confirm no editor, linter, watcher, or sync agent (Dropbox, rsync, IDE) is touching the worktree during the run.","Move the worktree to a local tmpfs/ext4 volume and re-run.","If the agent under test is the cause, fix the agent — this is detected tampering by design."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"# Snapshot oracle file hashes before staging; the harness already does this\n# internally, but you can log them for post-mortem.\nimport hashlib\nfrom pathlib import Path\n\ndef log_expected(stage_root: Path, files):\n    for item in files:\n        h = hashlib.sha256(item.payload).hexdigest()\n        log.debug('oracle %s expected sha256=%s', item.target, h)","typeGuard":null,"tryCatchPattern":"try:\n    with staged_task_oracle(worktree, snapshot) as stage:\n        run_model(stage)\nexcept ValueError as exc:\n    msg = str(exc)\n    if 'oracle file changed during verification' in msg:\n        target = msg.split(': ', 1)[-1]\n        log.security('model tampered with oracle file %s', target)\n        mark_cheating()\n        raise","preventionTips":["Lock down file watchers, linters, and IDE auto-save against the worktree.","Move the worktree to tmpfs for the duration of the run.","Audit the model/agent for any write under .wfbench-oracle-*."],"tags":["oracle","toctou","integrity","benchmark","filesystem"],"backgroundTag":null,"analyzedSha":"d540b00184d71a896261ee02670da9a92d59d8f7","analyzedAt":"2026-08-12T19:50:25.132Z","schemaVersion":2},"datasetVersion":"2026-08-12T23:17:12.415Z"}