TencentCloud/TencentDB-Agent-Memory · error

[skill-agent-queue] tasks-mutex wait timeout for ${key}

Error message

[skill-agent-queue] tasks-mutex wait timeout for ${key}

What it means

MemoryCore's in-process skill agent task queue guards per-key task execution with an in-memory mutex map. withTasksMutex busy-waits (polling every 10ms) until the key's lock is free, up to a deadline; if the deadline passes it throws this timeout error. It indicates another task is holding the same mutex key for too long or a holder leaked/never released.

Source

Thrown at MemoryCore/src/core/skill/conversation-add/agent-task-queue.ts:337

    fn: () => Promise<T>,
  ): Promise<T> {
    const key = `mutex:${serializeAgentTuple(tuple)}`;
    const deadline = Date.now() + opts.waitDeadlineMs;
    while (true) {
      const now = Date.now();
      const cur = this.tasksMutex.get(key);
      if (!cur || cur.expireAt <= now) {
        const token = randomUUID();
        this.tasksMutex.set(key, { token, expireAt: now + opts.lockTtlMs });
        try {
          return await fn();
        } finally {
          const held = this.tasksMutex.get(key);
          if (held && held.token === token) this.tasksMutex.delete(key);
        }
      }
      if (Date.now() > deadline) {
        throw new Error(`[skill-agent-queue] tasks-mutex wait timeout for ${key}`);
      }
      await sleep(10);
    }
  }

  // ── extract lock ──

  async acquireExtractLock(tuple: AgentTuple, ttlMs: number): Promise<ExtractLockHandle | null> {
    const key = serializeAgentTuple(tuple);
    const now = Date.now();
    const cur = this.extractLocks.get(key);
    if (cur && cur.expireAt > now) return null;
    const token = randomUUID();
    this.extractLocks.set(key, { token, expireAt: now + ttlMs });
    return { key, token };
  }

  async renewExtractLock(handle: ExtractLockHandle, ttlMs: number): Promise<boolean> {

View on GitHub (pinned to 3efcd317b8)

Solutions

1. Identify what task holds the mutex for the key (log acquisition/release in withTasksMutex) and fix or shorten it
2. Check for re-entrant calls: the same logical task must not call withTasksMutex with the same key recursively
3. Increase the wait deadline if the critical section legitimately takes long
4. Ensure the finally block always deletes the lock for the correct token; guard against token mismatch leaks
5. Add monitoring/metrics on mutex wait duration to catch contention early

Example fix

// before: nested lock on same key causes self-deadlock -> timeout
await withTasksMutex(key, async () => {
  await withTasksMutex(key, async () => { /* never runs */ });
});
// after: hoist the outer lock, do nested work inside one critical section
await withTasksMutex(key, async () => {
  await doWork();
  await doMoreWork();
});

// best-effort pre-check for in-process lock contention
const deadline = Date.now() + waitMs;
// wrap the call and classify:
const isMutexTimeout = (e: unknown) =>
  e instanceof Error && e.message.startsWith("[skill-agent-queue] tasks-mutex wait timeout");

function isTasksMutexTimeout(e: unknown): e is Error & { message: string } {
  return e instanceof Error && /tasks-mutex wait timeout for /.test(e.message);
}

try {
  await queue.withTasksMutex(key, () => runTask());
} catch (e) {
  if (isTasksMutexTimeout(e)) {
    logger.warn({ key }, "tasks-mutex contention; retrying after backoff");
    await sleep(100 + Math.random() * 200);
    return retryOnce();
  }
  throw e;
}

When it happens

Trigger: Calling an operation wrapped in withTasksMutex(key) while another concurrent task holds the same key's lock beyond the deadline — e.g. a long-running skill agent task, a deadlock where the current async context re-enters withTasksMutex for the same key, or a holder that threw before its finally-block deleted the entry (stale token mismatch).

Common situations: High concurrency on the same conversation/skill key; a stuck async task blocking the queue; test environments with fake timers making sleep(10) never advance while Date.now() does; lock leakage after a crashed holder whose cleanup ran with a different token.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• KernelFetchError(504 on timeout, 502 otherwise) with dynamic
• ${TAG} ${path} fetch failed: ${(err as Error).message}

AI-assisted analysis of TencentCloud/TencentDB-Agent-Memory@3efcd317b8 (2026-09-01). Data as JSON: /api/errors/012646b10cd8c07a. Report an issue: GitHub.