mastra-ai/mastra · error

Factory kickoff lease was lost before completion.

Error message

Factory kickoff lease was lost before completion.

What it means

After the run body completes, the dispatcher calls completePendingStart with the lease identity (record + ownerId). If storage returns falsy, the distributed lease on the pending start was lost — another owner took over, or the lease expired/was released — so this dispatcher no longer owns the record and must not mark it complete. The throw routes the error into failPendingStart, which records the sanitized error and schedules the record for retry at availableAt.

Source

Thrown at mastracode/factory/src/rules/dispatcher.ts:1008

            const observed = await waitForAgentEndOrTimeout(agentEnd, this.#skillCompletionObservationTimeoutMs);
            if (!observed) {
              throw new Error('Factory kickoff run terminal event was not observed before timeout.');
            } else if (endReason === 'error') {
              throw new Error('Factory kickoff run ended in error.');
            } else if (endReason === 'aborted') {
              // Retryable for the same reason as skill decisions: the dominant
              // cause is the process going away underneath the run, not a
              // deliberate stop, and a spurious retry is bounded by
              // MAX_ATTEMPTS while a dead card costs a human a manual nudge.
              throw new Error('Factory kickoff run was aborted before it finished.');
            }
          } finally {
            unsubscribe();
          }
        },
      );
      const completed = await this.#storage.completePendingStart(leaseIdentity(record, this.#ownerId), new Date());
      if (!completed) throw new Error('Factory kickoff lease was lost before completion.');
    } catch (error) {
      await this.#storage.failPendingStart({
        ...leaseIdentity(record, this.#ownerId),
        now: new Date(),
        availableAt: retryAt(now, record.attempts),
        lastError: sanitizeDispatchError(error),
        failureCode: factoryDispatchFailureCode(error),
        terminal: record.attempts >= MAX_ATTEMPTS,
      });
    }
  }
}

export const FACTORY_DISPATCH_CONSTANTS = {
  leaseMs: LEASE_MS,
  pollMs: POLL_MS,
  batchSize: BATCH_SIZE,
  maxAttempts: MAX_ATTEMPTS,

View on GitHub (pinned to 75dd419e61)

Solutions

1. Ensure only one dispatcher owner (ownerId) processes a given record — check for duplicate deployments or overlapping leader election.
2. Extend the lease/TTL (or renew it) to outlast the longest possible run including skillCompletionObservationTimeoutMs waits.
3. Check storage for who holds the lease now; if another owner completed it, treat this attempt as superseded and drop the duplicate.
4. Synchronize clocks / verify lease expiry timestamps aren't skewed across hosts.

Example fix

// before: second dispatcher instance steals an in-flight record
// two processes share ownerId or run without leader election
// after: single active owner per record
const isLeader = await electLeader();
if (isLeader) await dispatcher.process(record); // lease stays with one owner

const owned = await verifyLeaseOwnership(record, ownerId); // confirm lease before long work
if (!owned) throw new Error('lease not held; skip processing');

try {
  await dispatcher.dispatch(record);
} catch (e) {
  if (String(e?.message).includes('lease was lost')) {
    // another owner took over; drop this attempt, do not re-fail aggressively
    logger.warn({ record }, 'lease lost; skipping duplicate attempt');
  } else throw e;
}

When it happens

Trigger: completePendingStart returned false: lease expired due to a long run exceeding lease TTL, a competing dispatcher instance with a different ownerId claimed the record, or failover/leader election changed ownership mid-run.

Common situations: Multiple dispatcher replicas running against the same storage; runs (including observation waits) exceeding the lease duration; clock skew between instances; manual intervention or a restart releasing the lease.

Related errors

• DURABLE_AGENT_RECOVER_ALREADY_IN_PROGRESS
• AgentThreadLeaseConflictError(runId, owner)
• MastraFactory.prepare() called twice
• error.message (dynamic; MaterializeError)
• ${initial.code}: ${initial.reason}

AI-assisted analysis of mastra-ai/mastra@75dd419e61 (2026-08-30). Data as JSON: /api/errors/a996fbb6b2c868a8. Report an issue: GitHub.