{"record":{"id":"a996fbb6b2c868a8","repo":"mastra-ai/mastra","slug":"factory-kickoff-lease-was-lost-before-completion","errorCode":null,"errorMessage":"Factory kickoff lease was lost before completion.","messagePattern":"Factory kickoff lease was lost before completion\\.","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"mastracode/factory/src/rules/dispatcher.ts","lineNumber":1008,"sourceCode":"            const observed = await waitForAgentEndOrTimeout(agentEnd, this.#skillCompletionObservationTimeoutMs);\n            if (!observed) {\n              throw new Error('Factory kickoff run terminal event was not observed before timeout.');\n            } else if (endReason === 'error') {\n              throw new Error('Factory kickoff run ended in error.');\n            } else if (endReason === 'aborted') {\n              // Retryable for the same reason as skill decisions: the dominant\n              // cause is the process going away underneath the run, not a\n              // deliberate stop, and a spurious retry is bounded by\n              // MAX_ATTEMPTS while a dead card costs a human a manual nudge.\n              throw new Error('Factory kickoff run was aborted before it finished.');\n            }\n          } finally {\n            unsubscribe();\n          }\n        },\n      );\n      const completed = await this.#storage.completePendingStart(leaseIdentity(record, this.#ownerId), new Date());\n      if (!completed) throw new Error('Factory kickoff lease was lost before completion.');\n    } catch (error) {\n      await this.#storage.failPendingStart({\n        ...leaseIdentity(record, this.#ownerId),\n        now: new Date(),\n        availableAt: retryAt(now, record.attempts),\n        lastError: sanitizeDispatchError(error),\n        failureCode: factoryDispatchFailureCode(error),\n        terminal: record.attempts >= MAX_ATTEMPTS,\n      });\n    }\n  }\n}\n\nexport const FACTORY_DISPATCH_CONSTANTS = {\n  leaseMs: LEASE_MS,\n  pollMs: POLL_MS,\n  batchSize: BATCH_SIZE,\n  maxAttempts: MAX_ATTEMPTS,","sourceCodeStart":990,"sourceCodeEnd":1026,"githubUrl":"https://github.com/mastra-ai/mastra/blob/75dd419e613fe9c39f846ffc500716141b74fda6/mastracode/factory/src/rules/dispatcher.ts#L990-L1026","documentation":"After the run body completes, the dispatcher calls completePendingStart with the lease identity (record + ownerId). If storage returns falsy, the distributed lease on the pending start was lost — another owner took over, or the lease expired/was released — so this dispatcher no longer owns the record and must not mark it complete. The throw routes the error into failPendingStart, which records the sanitized error and schedules the record for retry at availableAt.","triggerScenarios":"completePendingStart returned false: lease expired due to a long run exceeding lease TTL, a competing dispatcher instance with a different ownerId claimed the record, or failover/leader election changed ownership mid-run.","commonSituations":"Multiple dispatcher replicas running against the same storage; runs (including observation waits) exceeding the lease duration; clock skew between instances; manual intervention or a restart releasing the lease.","solutions":["Ensure only one dispatcher owner (ownerId) processes a given record — check for duplicate deployments or overlapping leader election.","Extend the lease/TTL (or renew it) to outlast the longest possible run including skillCompletionObservationTimeoutMs waits.","Check storage for who holds the lease now; if another owner completed it, treat this attempt as superseded and drop the duplicate.","Synchronize clocks / verify lease expiry timestamps aren't skewed across hosts."],"exampleFix":"// before: second dispatcher instance steals an in-flight record\n// two processes share ownerId or run without leader election\n// after: single active owner per record\nconst isLeader = await electLeader();\nif (isLeader) await dispatcher.process(record); // lease stays with one owner","handlingStrategy":"try-catch","validationCode":"const owned = await verifyLeaseOwnership(record, ownerId); // confirm lease before long work\nif (!owned) throw new Error('lease not held; skip processing');","typeGuard":null,"tryCatchPattern":"try {\n  await dispatcher.dispatch(record);\n} catch (e) {\n  if (String(e?.message).includes('lease was lost')) {\n    // another owner took over; drop this attempt, do not re-fail aggressively\n    logger.warn({ record }, 'lease lost; skipping duplicate attempt');\n  } else throw e;\n}","preventionTips":["Run a single active dispatcher owner per record (leader election) across replicas","Set lease TTL longer than the maximum run plus observation timeout, or renew leases mid-run","Synchronize clocks across hosts to avoid premature lease expiry","Alert when completePendingStart returns false — it signals duplicate owners or undersized TTLs"],"tags":["factory","lease","concurrency","distributed"],"backgroundTag":"lease-lost-before-completion","analyzedSha":"75dd419e613fe9c39f846ffc500716141b74fda6","analyzedAt":"2026-08-30T00:15:31.844Z","schemaVersion":2},"datasetVersion":"2026-08-30T03:17:51.788Z"}