{"record":{"id":"40eabae63a739886","repo":"paperclipai/paperclip","slug":"managed-codex-credential-ownership-was-lost","errorCode":null,"errorMessage":"Managed Codex credential ownership was lost","messagePattern":"Managed Codex credential ownership was lost","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"packages/paperclip-runner/src/drivers/acpx/codex-credentials.ts","lineNumber":436,"sourceCode":"    inheritanceFds = Object.freeze([first, second]) as readonly [\n      number,\n      number,\n    ];\n  } catch (error) {\n    released = true;\n    await Promise.allSettled(servers.map(closeCredentialLeaseServer));\n    throw error;\n  }\n\n  return Object.freeze({\n    assertHeld(): void {\n      if (\n        released ||\n        invalid !== null ||\n        servers.filter((server) => server.listening).length <\n          CREDENTIAL_LEASE_QUORUM\n      ) {\n        throw new Error(\"Managed Codex credential ownership was lost\");\n      }\n    },\n    candidatePorts(): readonly [number, number, number] {\n      return candidatePorts;\n    },\n    inheritanceFds(): readonly [number, number] {\n      this.assertHeld();\n      return inheritanceFds;\n    },\n    async activateLifetimeOwner(pid: number): Promise<void> {\n      this.assertHeld();\n      if (!Number.isSafeInteger(pid) || pid < 1) {\n        throw new Error(\"Managed Codex credential lifetime owner is invalid\");\n      }\n    },\n    async release(): Promise<void> {\n      if (released) return;\n      const outcomes = await Promise.allSettled(","sourceCodeStart":418,"sourceCodeEnd":454,"githubUrl":"https://github.com/paperclipai/paperclip/blob/01ad8584922b5d85292b1723cae71fa0d9b07a19/packages/paperclip-runner/src/drivers/acpx/codex-credentials.ts#L418-L454","documentation":"Managed Codex credentials are guarded by a lease quorum of listening lock servers plus validity checks. assertHeld throws when ownership is no longer provable: the lease was released, a lock server reported the lease invalid, or fewer servers than CREDENTIAL_LEASE_QUORUM report listening. Called by inheritanceFds and activateLifetimeOwner, this means credential inheritance or lifetime-owner activation cannot proceed safely because another process may now own the credential.","triggerScenarios":"Calling inheritanceFds() or activateLifetimeOwner() after the lease was released; a quorum lock server stopped listening; the lease was invalidated by another claimant winning the lifetime-owner election.","commonSituations":"Runner process restarts or fork timing causing the lease to lapse; port conflicts preventing quorum servers from listening; two runner instances contending for the same managed credential; machine sleep/resume expiring the lease.","solutions":["Re-acquire the managed credential lease (re-run the credential acquisition/lock flow) before retrying inheritanceFds or activateLifetimeOwner.","Check that all quorum lock server ports are listening; restart the lease servers and resolve any port conflicts (use candidatePorts() to pick free ports).","Ensure only one runner instance owns the credential at a time; stop the competing process.","Retry promptly after re-acquisition and keep the holder process alive through fork/inheritance to avoid releasing mid-operation."],"exampleFix":"// before\nconst fds = lease.inheritanceFds(); // lease already released -> throws\n// after\nif (!lease.isHeld()) await reAcquireCredentialLease(lease.candidatePorts());\nconst fds = lease.inheritanceFds();","handlingStrategy":"retry","validationCode":"function leaseIsUsable(lease) {\n  return !lease.released && lease.invalid === null &&\n    lease.servers.filter((s) => s.listening).length >= CREDENTIAL_LEASE_QUORUM;\n}\nif (!leaseIsUsable(lease)) await reAcquireLease();","typeGuard":"function holdsQuorum(lease) {\n  return lease.servers.filter((s) => s.listening).length >= CREDENTIAL_LEASE_QUORUM;\n}","tryCatchPattern":"try {\n  const fds = lease.inheritanceFds();\n} catch (err) {\n  if (err.message === \"Managed Codex credential ownership was lost\") {\n    await reAcquireCredentialLease(lease.candidatePorts());\n    // then retry inheritance once\n  } else throw err;\n}","preventionTips":["Keep the lease-holder process alive through fork/inheritance operations.","Pre-check quorum ports are listening before claiming the credential.","Ensure single-instance ownership; use locks to prevent competing runners.","Handle machine sleep/resume by re-validating the lease afterward."],"tags":["credentials","locking","quorum","concurrency"],"backgroundTag":"lock-lease-lost","analyzedSha":"01ad8584922b5d85292b1723cae71fa0d9b07a19","analyzedAt":"2026-09-10T03:14:50.855Z","contentChangedAt":"2026-09-10T03:14:50.855Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}