mastra-ai/mastra · error · MastraError

DURABLE_AGENT_RECOVER_ALREADY_IN_PROGRESS

DURABLE_AGENT_RECOVER_ALREADY_IN_PROGRESS

Error message

DurableAgent "${this.name}" recover(${runId}): another process is already recovering this run.

What it means

DurableAgent#recoveryLease calls #acquireRecoveryLease, which first checks a local in-process claim map (localRecoveryClaims). If this process already holds a claim for the lease key `mastra:durable-agent-recovery:v1:[agentId, runId]`, it throws immediately, treating it as another recovery already in progress. It prevents concurrent recoveries of the same run within the same process.

Source

Thrown at packages/core/src/agent/durable/durable-agent.ts:573

   * through the thread stream. The thread lease cannot provide this guarantee:
   * its owner is the logical runId, so two processes recovering the same run
   * are indistinguishable to an idempotent lease backend.
   */
  async #acquireRecoveryLease(runId: string, abortController: AbortController): Promise<RecoveryLease> {
    const pubsub = this.pubsub;
    const unwrap = (pubsub as { getLeaseProvider?: () => LeaseProvider | undefined }).getLeaseProvider;
    const provider =
      typeof unwrap === 'function'
        ? (unwrap.call(pubsub) ?? NoopLeaseProvider)
        : isLeaseProvider(pubsub)
          ? pubsub
          : NoopLeaseProvider;
    const key = `mastra:durable-agent-recovery:v1:${JSON.stringify([this.id, runId])}`;
    const owner = crypto.randomUUID();

    const localOwner = localRecoveryClaims.get(key);
    if (localOwner) {
      throw new MastraError({
        id: 'DURABLE_AGENT_RECOVER_ALREADY_IN_PROGRESS',
        domain: ErrorDomain.AGENT,
        category: ErrorCategory.USER,
        text: `DurableAgent "${this.name}" recover(${runId}): another process is already recovering this run.`,
        details: { agentName: this.name, runId },
      });
    }
    localRecoveryClaims.set(key, owner);

    const leaseAcquireStartedAt = Date.now();
    let acquired: Awaited<ReturnType<LeaseProvider['acquireLease']>>;
    try {
      acquired = await provider.acquireLease(key, owner, RECOVERY_LEASE_TTL_MS);
    } catch (cause) {
      if (localRecoveryClaims.get(key) === owner) localRecoveryClaims.delete(key);
      throw new MastraError(
        {
          id: 'DURABLE_AGENT_RECOVER_LEASE_ACQUIRE_FAILED',

View on GitHub (pinned to 75dd419e61)

Solutions

Await the in-flight recover() promise instead of starting a second one; serialize recoveries per runId.
Add an in-flight guard (map of runId -> Promise) so concurrent calls share one recovery.
If a previous recover() crashed without releasing its local claim, restart the process or wait for the lease TTL to expire, then retry.

Example fix

// before
runs.forEach(runId => agent.recover(runId)); // may overlap
// after
const inflight = new Map<string, Promise<void>>();
const recoverOnce = (runId: string) => {
  if (!inflight.has(runId)) {
    inflight.set(runId, agent.recover(runId).finally(() => inflight.delete(runId)));
  }
  return inflight.get(runId);
};
await Promise.all(runs.map(recoverOnce));

Defensive patterns

Strategy: retry

Validate before calling

if (inflightRecoveries.has(runId)) return inflightRecoveries.get(runId); // dedupe before calling

Try / catch

try {
  await agent.recover(runId);
} catch (e) {
  if (String(e?.id) === 'DURABLE_AGENT_RECOVER_ALREADY_IN_PROGRESS') {
    await waitForInflightRecovery(runId); // await the existing attempt, don't start another
  } else throw e;
}

Prevention

Keep one in-flight recover() per runId per process (promise map).
Avoid retry loops that re-enter recover() while the previous attempt is still running.
Ensure job schedulers don't dispatch the same runId twice within one process.

When it happens

Trigger: Calling agent.recover(runId) twice concurrently in the same process for the same run — e.g. overlapping recover() invocations before the first finishes and releases its claim, or a retry loop firing while a previous attempt is still running.

Common situations: A retry wrapper re-invoking recover() on timeout while the original call is still in flight; a job scheduler dispatching the same recovery to two workers in one process; event handlers firing recover() multiple times for one run.

Related errors

AI-assisted analysis of mastra-ai/mastra@75dd419e61 (2026-08-30). Data as JSON: /api/errors/24c7f686d09d8583. Report an issue: GitHub.