{"record":{"id":"ded0ad2890521039","repo":"paperclipai/paperclip","slug":"createos-lease-cleanup-must-finish-before-resume","errorCode":null,"errorMessage":"CreateOS lease cleanup must finish before resume.","messagePattern":"CreateOS lease cleanup must finish before resume\\.","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"packages/plugins/sandbox-providers/createos/src/plugin.ts","lineNumber":155,"sourceCode":"      try {\n        lease = await acquire({ ...params, runId: \"probe\" });\n        const result = await execute(new CreateosClient(parseConfig(params.config)), {\n          ...params, lease, command: \"/bin/echo\", args: [\"paperclip-createos-ready\"], cwd: CWD,\n        }, AbortSignal.timeout(parseConfig(params.config).timeoutMs));\n        if (result.timedOut || result.exitCode !== 0 || !result.stdout.includes(\"paperclip-createos-ready\")) throw new Error(\"CreateOS command probe failed.\");\n        return { ok: true, summary: \"CreateOS sandbox creation and command execution succeeded.\" };\n      } catch (error) {\n        return { ok: false, summary: error instanceof Error ? error.message : \"CreateOS probe failed.\" };\n      } finally {\n        // Never leave a reusable probe sandbox behind or hide deletion failure.\n        if (lease?.providerLeaseId) await new CreateosClient(parseConfig(params.config)).destroySandbox(lease.providerLeaseId);\n      }\n    },\n    onEnvironmentAcquireLease: acquire,\n    async onEnvironmentResumeLease(params) {\n      if (!metadataMatches(params, params.leaseMetadata)) throw new Error(\"CreateOS lease does not belong to this environment.\");\n      const scope = key(params, params.providerLeaseId);\n      if (closing.has(scope) || unconfirmedCleanup.has(scope)) throw new Error(\"CreateOS lease cleanup must finish before resume.\");\n      const marker = params.leaseMetadata?.marker;\n      if (typeof marker !== \"string\" || !/^[0-9a-f-]{36}$/.test(marker)) return { providerLeaseId: null, metadata: { expired: true } };\n      const config = parseConfig(params.config);\n      if (params.leaseMetadata?.shape !== config.shape || params.leaseMetadata.rootfs !== config.rootfs || params.leaseMetadata.region !== config.region) {\n        return { providerLeaseId: null, metadata: { expired: true } };\n      }\n      const client = new CreateosClient(config);\n      const signal = AbortSignal.timeout(config.timeoutMs);\n      try {\n        const sandbox = await client.getSandbox(params.providerLeaseId, signal);\n        if ([\"destroyed\", \"failed\"].includes(sandbox.status!)) return { providerLeaseId: null, metadata: { expired: true } };\n        await client.transition(params.providerLeaseId, \"running\", signal);\n        const data = await client.json(`/sandboxes/${params.providerLeaseId}/exec`, \"POST\", {\n          cmd: \"/bin/cat\", args: [MARKER],\n        }, signal);\n        const result = object(data.result);\n        if (result.exit_code !== 0 || result.stdout !== marker) return { providerLeaseId: null, metadata: { expired: true } };\n        return { providerLeaseId: params.providerLeaseId, metadata: { ...params.leaseMetadata, resumedLease: true } };","sourceCodeStart":137,"sourceCodeEnd":173,"githubUrl":"https://github.com/paperclipai/paperclip/blob/3f1d897a7c018d76563a21c6e39c3c9b03933622/packages/plugins/sandbox-providers/createos/src/plugin.ts#L137-L173","documentation":"Thrown by onEnvironmentResumeLease when the lease is currently being cleaned up (scope present in 'closing') or had an unconfirmed cleanup (scope in 'unconfirmedCleanup'). Resuming a lease mid-destroy or with unverifiable prior cleanup would attach work to a sandbox that may disappear or still hold stale processes, so the provider refuses. Wait for cleanup to finish or destroy and re-acquire.","triggerScenarios":"Calling onEnvironmentResumeLease concurrently with release()/stopActive for the same lease (cleanup in progress), or resuming a lease previously released without destroy whose process cleanup was never confirmed (see error 116).","commonSituations":"Agent scheduler resuming work while an environment stop/teardown is still awaiting the CreateOS API; a retry loop resuming immediately after a failed release; reuse-mode pools where a prior pause was unconfirmed.","solutions":["Wait for the concurrent release to complete, then retry resume; the 'closing' guard is transient.","If the lease has unconfirmed cleanup, destroy it (release with destroy=true) and acquire a fresh lease instead of resuming.","Serialize lease lifecycle operations per lease so resume never overlaps release.","Add a small backoff around resume retries to avoid racing the teardown path."],"exampleFix":"// before: immediate resume after teardown request\nstopEnvironment(env);\nawait plugin.onEnvironmentResumeLease(resumeParams); // throws\n// after: await teardown, then resume or re-acquire\nawait stopEnvironment(env);\ntry { await plugin.onEnvironmentResumeLease(resumeParams); }\ncatch { await plugin.onEnvironmentAcquireLease(acquireParams); }","handlingStrategy":"retry","validationCode":"// Track lifecycle state per lease in caller code\nconst releasing = new Set<string>();\nfunction canResume(leaseId: string): boolean {\n  return !releasing.has(leaseId);\n}","typeGuard":"function isCleanupPendingError(e: unknown): boolean {\n  return e instanceof Error && e.message.includes('cleanup must finish before resume');\n}","tryCatchPattern":"try {\n  await plugin.onEnvironmentResumeLease(params);\n} catch (e) {\n  if (isCleanupPendingError(e)) {\n    await sleep(1000);\n    await plugin.onEnvironmentResumeLease(params); // or acquire fresh lease\n  } else throw e;\n}","preventionTips":["Serialize lease lifecycle: never call resume while release/stop is in flight for the same lease","Mark leases as 'destroying' in your scheduler and route resumes to fresh acquires instead","Use exponential backoff on resume retries rather than immediate re-calls","Always destroy (not reuse) leases that previously reported unconfirmed cleanup before reacquiring"],"tags":["concurrency","sandbox","lease","cleanup"],"backgroundTag":"invalid-state-transition","analyzedSha":"3f1d897a7c018d76563a21c6e39c3c9b03933622","analyzedAt":"2026-09-18T08:03:59.046Z","contentChangedAt":"2026-09-18T08:03:59.046Z","schemaVersion":2},"datasetVersion":"2026-09-22T11:17:16.035Z"}