abhigyanpatwari/GitNexus · critical · SandboxError

Compound Engineering plugin file changed during validation:

Error message

Compound Engineering plugin file changed during validation: {path}

What it means

TOCTOU guard between lstat and the post-open fstat. After opening the path with O_RDONLY|O_NOFOLLOW, _bounded_plugin_bytes compares (st_dev, st_ino) and the regular-file flag of the opened descriptor against the pre-open lstat; any mismatch means the inode was swapped (classic symlink/replace race) and reading would be unsafe. On platforms without O_NOFOLLOW the open itself could have followed a link planted in the window.

Source

Thrown at eval/workflow_bench/runtime_mounts.py:363

        yield from walk(directory, PurePosixPath(name))


def _bounded_plugin_bytes(path: Path) -> tuple[bytes, bool]:
    """Read one stable regular file without following a last-component symlink."""

    try:
        before = path.lstat()
    except OSError as exc:
        raise SandboxError(f"Compound Engineering plugin file is unreadable: {path}: {exc}") from exc
    if stat.S_ISLNK(before.st_mode) or not stat.S_ISREG(before.st_mode):
        raise SandboxError(f"Compound Engineering plugin file must be regular and non-symlink: {path}")
    if before.st_size > MAX_CE_PLUGIN_FILE_BYTES:
        raise SandboxError(f"Compound Engineering plugin file exceeds the per-file limit: {path}")
    descriptor = os.open(path, os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0))
    try:
        opened = os.fstat(descriptor)
        if (opened.st_dev, opened.st_ino) != (before.st_dev, before.st_ino) or not stat.S_ISREG(opened.st_mode):
            raise SandboxError(f"Compound Engineering plugin file changed during validation: {path}")
        chunks: list[bytes] = []
        remaining = MAX_CE_PLUGIN_FILE_BYTES + 1
        while remaining > 0:
            chunk = os.read(descriptor, min(64 * 1024, remaining))
            if not chunk:
                break
            chunks.append(chunk)
            remaining -= len(chunk)
        payload = b"".join(chunks)
        after = os.fstat(descriptor)
    finally:
        os.close(descriptor)
    if len(payload) > MAX_CE_PLUGIN_FILE_BYTES:
        raise SandboxError(f"Compound Engineering plugin file exceeds the per-file limit: {path}")
    identity_before = (opened.st_dev, opened.st_ino, opened.st_size, opened.st_mtime_ns)
    identity_after = (after.st_dev, after.st_ino, after.st_size, after.st_mtime_ns)
    if identity_after != identity_before or len(payload) != after.st_size:
        raise SandboxError(f"Compound Engineering plugin file changed while being copied: {path}")

View on GitHub (pinned to d540b00184)

Solutions

Freeze the plugin source before the run: produce a tarball or a fresh 'cp -a' checkout and point --ce-plugin-dir at the immutable copy.
Stop any file watcher, build daemon, or sync client that touches the plugin tree during the benchmark.
Re-run the snapshot step; transient races usually clear once the writer is quiescent.
On shared/networked filesystems, copy the plugin onto local disk (tmpfs ideal) first to remove cross-host inode jitter.

Example fix

# before: build and bench share the tree
pnpm --filter ce-plugin build && wfbench run --ce-plugin-dir ./packages/ce-plugin
# after: snapshot once, then bench the frozen copy
cp -a ./packages/ce-plugin /tmp/ce-plugin-frozen && wfbench run --ce-plugin-dir /tmp/ce-plugin-frozen

Defensive patterns

Strategy: validation

Validate before calling

from pathlib import Path
import shutil, os

# Freeze the source into a read-only copy so no writer can race the snapshot.
src = Path("ce-plugin")
frozen = Path("/tmp/ce-plugin-frozen")
if frozen.exists():
    shutil.rmtree(frozen)
shutil.copytree(src, frozen, symlinks=False)
for root, dirs, files in os.walk(frozen):
    Path(root).chmod(0o555)
    for f in files:
        Path(root, f).chmod(0o444)
# pass frozen as --ce-plugin-dir

Type guard

import os, stat
from pathlib import Path

def open_stable(path: Path):
    before = path.lstat()
    if stat.S_ISLNK(before.st_mode) or not stat.S_ISREG(before.st_mode):
        return None
    fd = os.open(path, os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0))
    opened = os.fstat(fd)
    if (opened.st_dev, opened.st_ino) != (before.st_dev, before.st_ino):
        os.close(fd)
        return None
    return fd  # caller reads and closes

Try / catch

try:
    snapshot = _build_ce_plugin_snapshot(config, destination_parent)
except SandboxError as exc:
    if "changed during validation" in str(exc):
        log.error("plugin tree mutated mid-snapshot; quiesce writers and retry")
    raise

Prevention

Never pass a live checkout (with active builds/watchers) as --ce-plugin-dir.
Snapshot the plugin into a read-only copy before benchmarking.
Run on local disk, not a synced/networked folder, to avoid inode instability.

When it happens

Trigger: The plugin file was replaced, renamed, or turned into a symlink between the lstat and os.open; a concurrent writer (live build, editor autosave, package manager reinstall) mutated the tree mid-snapshot; a hostile plugin deliberately races the open.

Common situations: Building the plugin and running the benchmark from the same tree; a watcher (tsc --watch, vite) rewriting files; NFS or synced-folder filesystems with deferred inode stability; CI that checks out the plugin while the harness walks it.

Related errors

AI-assisted analysis of abhigyanpatwari/GitNexus@d540b00184 (2026-08-12). Data as JSON: /api/errors/41bd9d15189923f3. Report an issue: GitHub.