different-ai/openwork · error

automation_run_lease_lost

automation_run_lease_lost

Error message

automation_run_lease_lost

What it means

The repository transitions a cloud automation run's state (e.g. completion/cancel) with an UPDATE guarded by id, lease_owner, and status='running'. If zero rows changed — automationUpdateChangedRows(result) is false — the caller no longer holds the lease or the run is no longer running, so the mutation is refused with automation_run_lease_lost to protect against a concurrent worker taking over the run.

Source

Thrown at ee/apps/den-api/src/automations/repository.ts:653

  async setCloudExecution(input: {
    runId: string
    leaseOwner: string
    engineKind: string
    receipt: Record<string, unknown>
    now: number
  }): Promise<void> {
    const result: unknown = await db.update(AutomationRunTable).set({
      engine_kind: input.engineKind,
      engine_receipt: input.receipt,
      engine_admitted_at: new Date(input.now),
      updated_at: new Date(input.now),
    }).where(and(
      eq(AutomationRunTable.id, normalizeRunId(input.runId)),
      eq(AutomationRunTable.lease_owner, input.leaseOwner),
      eq(AutomationRunTable.status, "running"),
    ))
    if (!automationUpdateChangedRows(result)) throw new Error("automation_run_lease_lost")
  }

  async heartbeatCloud(input: { runId: string; leaseOwner: string; leaseMs: number; now: number }): Promise<boolean> {
    const result: unknown = await db.update(AutomationRunTable).set({
      heartbeat_at: new Date(input.now),
      lease_expires_at: new Date(input.now + input.leaseMs),
      updated_at: new Date(input.now),
    }).where(and(
      eq(AutomationRunTable.id, normalizeRunId(input.runId)),
      eq(AutomationRunTable.lease_owner, input.leaseOwner),
      eq(AutomationRunTable.status, "running"),
      gt(AutomationRunTable.lease_expires_at, new Date(input.now)),
    ))
    return automationUpdateChangedRows(result)
  }

  async cloudRunState(runId: string): Promise<{ cancelRequested: boolean; receipt: Record<string, unknown> | null } | null> {
    const rows = await db.select({

View on GitHub (pinned to 2b7df46e8a)

Solutions

1. Call heartbeatCloud on a schedule shorter than leaseMs to keep the lease alive
2. On this error, re-claim via claimCloud (or abandon) and re-run idempotently rather than forcing the update
3. Ensure the exact leaseOwner string used in claimCloud is reused for subsequent updates
4. Reduce job duration or increase leaseMs so long runs don't expire mid-flight

Example fix

// before
await repo.completeCloud({ runId, leaseOwner, ... }) // throws if lease expired
// after
try {
  await repo.completeCloud({ runId, leaseOwner, ... })
} catch (e) {
  if (String(e?.message).includes('lease_lost')) {
    const claim = await repo.claimCloud({ runId, leaseOwner: newOwner, leaseMs, ... })
    if (!claim) return // run taken elsewhere; skip
  } else throw e
}

const claim = await repo.claimCloud({ runId, leaseOwner, leaseMs, maxConcurrency, now })
if (!claim) throw new Error('cannot mutate: run not claimed by this worker')

function leaseIsCurrent(claim: { leaseExpiresAt: number } | null, now: number): claim is { leaseExpiresAt: number } {
  return claim !== null && claim.leaseExpiresAt > now
}

try {
  await repo.completeCloud(input)
} catch (e) {
  if (String((e as Error)?.message) === 'automation_run_lease_lost') {
    // heartbeat missed or lease stolen: re-claim or abandon idempotently
  } else throw e
}

When it happens

Trigger: Calling a lease-guarded mutation when: the lease expired and another worker claimed the run; heartbeatCloud was not called within leaseMs; the run already left status 'running'; or a different leaseOwner string is supplied than the one that claimed it.

Common situations: Long-running job exceeding its lease without heartbeats; two workers racing on the same run; clock skew shortening the effective lease; restart of a worker that reuses a stale runId/leaseOwner.

Related errors

• agent_diagnostics_in_progress
• agent_diagnostics_busy
• automation_revision_changed
• connector_mapping_not_found
• MCP_OAUTH_AUTHORIZATION_CLIENT_CHANGED

AI-assisted analysis of different-ai/openwork@2b7df46e8a (2026-09-01). Data as JSON: /api/errors/6fe13d89d39af438. Report an issue: GitHub.