{"record":{"id":"2cf11a1924ee914d","repo":"paperclipai/paperclip","slug":"acpx-agent-runtime-executable-changed-while-it","errorCode":null,"errorMessage":"ACPX ${agent} runtime executable changed while it was verified","messagePattern":"ACPX (.+?) runtime executable changed while it was verified","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/paperclip-runner/src/drivers/acpx/installation-integrity.ts","lineNumber":1088,"sourceCode":"    } finally {\n      buffer.fill(0);\n    }\n    const after = await handle.stat({ bigint: true });\n    const lexicalAfter = await lstat(executablePath, { bigint: true }).catch(\n      () => null,\n    );\n    const beforeIdentity = fileIdentity(before);\n    const afterIdentity = fileIdentity(after);\n    if (\n      position !== Number(before.size) ||\n      lexicalAfter === null ||\n      lexicalAfter.isSymbolicLink() ||\n      !lexicalAfter.isFile() ||\n      !sameIdentity(fileIdentity(lexicalBefore), fileIdentity(lexicalAfter)) ||\n      !sameIdentity(fileIdentity(lexicalAfter), afterIdentity) ||\n      !sameIdentity(beforeIdentity, afterIdentity)\n    ) {\n      throw new Error(\n        `ACPX ${agent} runtime executable changed while it was verified`,\n      );\n    }\n    const digest = `sha256:${hash.digest(\"hex\")}`;\n    if (digest !== expectedDigest) {\n      throw new Error(`ACPX ${agent} runtime executable digest mismatch`);\n    }\n    return { handle, identity: afterIdentity };\n  } catch (error) {\n    await handle.close();\n    throw error;\n  }\n}\n\n/** Fail closed where Node cannot atomically refuse a final symlink component. */\nexport function verifiedExecutableOpenFlags(\n  platform: NodeJS.Platform,\n  noFollowFlag: number | undefined,","sourceCodeStart":1070,"sourceCodeEnd":1106,"githubUrl":"https://github.com/paperclipai/paperclip/blob/01ad8584922b5d85292b1723cae71fa0d9b07a19/packages/paperclip-runner/src/drivers/acpx/installation-integrity.ts#L1070-L1106","documentation":"While hashing, the runtime re-stat's the open handle and re-lstat's the path, comparing device/inode identities across three snapshots (lexical before, lexical after, handle after). If any identity differs — or the file disappeared, became a symlink, or could not be fully read (position !== before.size) — the file mutated during verification, so the digest cannot be trusted and this error is thrown. This is a deliberate race-detection guard against binary-swapping attacks.","triggerScenarios":"The executable file is replaced, deleted, resized, or hard-link-swapped by another process between the initial lstat and the post-hash checks; the read loop ends early leaving `position !== Number(before.size)`; or lstat after hashing returns null (file removed).","commonSituations":"Package manager or auto-updater upgrading the binary at the exact moment a run starts; two installers racing on the same install dir; cleanup/AV quarantine replacing the file; container image being rebuilt while the service runs; CI jobs re-linking binaries concurrently.","solutions":["Retry the runtime start once the concurrent install/upgrade process has finished — this is usually a transient race","Serialize runtime upgrades with runtime starts (a lock around install + spawn) so the binary is never swapped mid-verification","Pin the runtime version and disable auto-updaters/AV quarantine on the install directory","Install the binary atomically (write to temp + `rename`) so readers never observe a half-swapped file","Check for competing processes (CI jobs, second service instance) touching the same install path"],"exampleFix":"// before: naive in-place upgrade racing with spawn\ncurl -o /opt/acpx/bin/acpx https://...   # spawn may read mid-write\n// after: atomic install + lock\nacquireLock('acpx-install')\nconst tmp = '/opt/acpx/bin/.acpx.tmp'; writeDownload(tmp); chmod +x tmp\nrenameSync(tmp, '/opt/acpx/bin/acpx')\nreleaseLock(); thenStartRuntime()","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  await startAcppRuntime();\n} catch (e) {\n  if (String(e?.message).includes('changed while it was verified')) {\n    // file was swapped mid-verification; wait for quiescence and retry once\n    await waitForInstallLockRelease();\n    await startAcppRuntime();\n  } else {\n    throw e;\n  }\n}","preventionTips":["Serialize installs and runtime starts with a lock so binaries are never swapped mid-read","Install binaries atomically (write temp + rename)","Disable auto-updaters and AV quarantine on the install directory","Run only one process (or container image version) that owns a given install path"],"tags":["filesystem","security","race-condition","acpx"],"backgroundTag":"internal-invariant-violation","analyzedSha":"01ad8584922b5d85292b1723cae71fa0d9b07a19","analyzedAt":"2026-09-10T03:14:50.855Z","contentChangedAt":"2026-09-10T03:14:50.855Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}