{"record":{"id":"6fe13d89d39af438","repo":"different-ai/openwork","slug":"automation-run-lease-lost","errorCode":"automation_run_lease_lost","errorMessage":"automation_run_lease_lost","messagePattern":"automation_run_lease_lost","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"ee/apps/den-api/src/automations/repository.ts","lineNumber":653,"sourceCode":"\n  async setCloudExecution(input: {\n    runId: string\n    leaseOwner: string\n    engineKind: string\n    receipt: Record<string, unknown>\n    now: number\n  }): Promise<void> {\n    const result: unknown = await db.update(AutomationRunTable).set({\n      engine_kind: input.engineKind,\n      engine_receipt: input.receipt,\n      engine_admitted_at: new Date(input.now),\n      updated_at: new Date(input.now),\n    }).where(and(\n      eq(AutomationRunTable.id, normalizeRunId(input.runId)),\n      eq(AutomationRunTable.lease_owner, input.leaseOwner),\n      eq(AutomationRunTable.status, \"running\"),\n    ))\n    if (!automationUpdateChangedRows(result)) throw new Error(\"automation_run_lease_lost\")\n  }\n\n  async heartbeatCloud(input: { runId: string; leaseOwner: string; leaseMs: number; now: number }): Promise<boolean> {\n    const result: unknown = await db.update(AutomationRunTable).set({\n      heartbeat_at: new Date(input.now),\n      lease_expires_at: new Date(input.now + input.leaseMs),\n      updated_at: new Date(input.now),\n    }).where(and(\n      eq(AutomationRunTable.id, normalizeRunId(input.runId)),\n      eq(AutomationRunTable.lease_owner, input.leaseOwner),\n      eq(AutomationRunTable.status, \"running\"),\n      gt(AutomationRunTable.lease_expires_at, new Date(input.now)),\n    ))\n    return automationUpdateChangedRows(result)\n  }\n\n  async cloudRunState(runId: string): Promise<{ cancelRequested: boolean; receipt: Record<string, unknown> | null } | null> {\n    const rows = await db.select({","sourceCodeStart":635,"sourceCodeEnd":671,"githubUrl":"https://github.com/different-ai/openwork/blob/2b7df46e8ae1517d64c896c7793d2d52ec845669/ee/apps/den-api/src/automations/repository.ts#L635-L671","documentation":"The repository transitions a cloud automation run's state (e.g. completion/cancel) with an UPDATE guarded by id, lease_owner, and status='running'. If zero rows changed — automationUpdateChangedRows(result) is false — the caller no longer holds the lease or the run is no longer running, so the mutation is refused with automation_run_lease_lost to protect against a concurrent worker taking over the run.","triggerScenarios":"Calling a lease-guarded mutation when: the lease expired and another worker claimed the run; heartbeatCloud was not called within leaseMs; the run already left status 'running'; or a different leaseOwner string is supplied than the one that claimed it.","commonSituations":"Long-running job exceeding its lease without heartbeats; two workers racing on the same run; clock skew shortening the effective lease; restart of a worker that reuses a stale runId/leaseOwner.","solutions":["Call heartbeatCloud on a schedule shorter than leaseMs to keep the lease alive","On this error, re-claim via claimCloud (or abandon) and re-run idempotently rather than forcing the update","Ensure the exact leaseOwner string used in claimCloud is reused for subsequent updates","Reduce job duration or increase leaseMs so long runs don't expire mid-flight"],"exampleFix":"// before\nawait repo.completeCloud({ runId, leaseOwner, ... }) // throws if lease expired\n// after\ntry {\n  await repo.completeCloud({ runId, leaseOwner, ... })\n} catch (e) {\n  if (String(e?.message).includes('lease_lost')) {\n    const claim = await repo.claimCloud({ runId, leaseOwner: newOwner, leaseMs, ... })\n    if (!claim) return // run taken elsewhere; skip\n  } else throw e\n}","handlingStrategy":"retry","validationCode":"const claim = await repo.claimCloud({ runId, leaseOwner, leaseMs, maxConcurrency, now })\nif (!claim) throw new Error('cannot mutate: run not claimed by this worker')","typeGuard":"function leaseIsCurrent(claim: { leaseExpiresAt: number } | null, now: number): claim is { leaseExpiresAt: number } {\n  return claim !== null && claim.leaseExpiresAt > now\n}","tryCatchPattern":"try {\n  await repo.completeCloud(input)\n} catch (e) {\n  if (String((e as Error)?.message) === 'automation_run_lease_lost') {\n    // heartbeat missed or lease stolen: re-claim or abandon idempotently\n  } else throw e\n}","preventionTips":["Heartbeat well within leaseMs (e.g. every leaseMs/3)","Reuse the exact leaseOwner from claimCloud for all subsequent calls","Design completion handlers to be idempotent so re-claim + retry is safe","Monitor lease expiry times for long-running tasks"],"tags":["concurrency","locking","distributed-systems"],"backgroundTag":"lock-lease-lost","analyzedSha":"2b7df46e8ae1517d64c896c7793d2d52ec845669","analyzedAt":"2026-09-01T07:59:23.713Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}