santifer/career-ops · error · LockTimeoutError

pipeline lock timeout: ${lockDir} held > ${timeoutMs}ms

Error message

pipeline lock timeout: ${lockDir} held > ${timeoutMs}ms

What it means

Thrown as LockTimeoutError by acquireWithRecovery() in pipeline-lock.mjs when Date.now() exceeds the deadline while the lock directory still cannot be created (it keeps hitting EEXIST and is not eligible for stale recovery). It means another holder kept the lock for the entire configured timeoutMs window.

Source

Thrown at pipeline-lock.mjs:165

        // otherwise disable stale recovery forever. The guard normally lives
        // for milliseconds, so an old one is judged by the same age rule.
        if (lockCanRecover(recoverGuardDir, staleMs)) {
          rmSync(recoverGuardDir, { recursive: true, force: true });
        }
      }

      if (hasRecoverGuard) {
        try {
          if (lockCanRecover(lockDir, staleMs)) {
            rmSync(lockDir, { recursive: true, force: true });
            continue; // retry acquisition immediately, still holding the guard's decision
          }
        } finally {
          rmSync(recoverGuardDir, { recursive: true, force: true });
        }
      }

      if (Date.now() > deadline) throw new LockTimeoutError(lockDir, timeoutMs);
      await sleep(retryMs);
      continue;
    }

    // Acquired. Record ownership; an owner-less lock would block every future
    // acquirer until the age-out, so clean up if the stamp can't be written.
    try {
      writeFileSync(join(lockDir, 'owner.json'), JSON.stringify({
        pid: process.pid,
        token,
        started_at: new Date().toISOString(),
        pipeline: pipelinePath,
      }, null, 2));
    } catch (ownerErr) {
      rmSync(lockDir, { recursive: true, force: true });
      throw ownerErr;
    }

View on GitHub (pinned to 9b17a8ac97)

Solutions

1. Wait for the in-flight operation to finish, then retry.
2. If no operation is actually running, remove the stale lock directory shown in the message (or raise staleMs so lockCanRecover reclaims it).
3. Increase the timeoutMs passed to acquireWithRecovery for large batches.
4. Ensure only one pipeline-mutating process runs at a time per tracker.

Example fix

// before
const lock = await acquireWithRecovery(lockDir, { timeoutMs: 5_000 });
// after: give large batches enough headroom and let stale recovery work
const lock = await acquireWithRecovery(lockDir, { timeoutMs: 120_000, staleMs: 60_000 });

null

/** Narrows a caught error to the lock-timeout class. */
function isLockTimeout(e) {
  return e instanceof Error && /pipeline lock timeout/.test(e.message);
}

import { LockTimeoutError } from './pipeline-lock.mjs';
try {
  const lock = await acquireWithRecovery(lockDir, { timeoutMs, staleMs });
} catch (e) {
  if (e instanceof LockTimeoutError || /lock timeout/.test(e.message)) {
    // wait and retry, or escalate if a holder is genuinely stuck
  } else throw e;
}

When it happens

Trigger: A long-running pipeline process holds the lock legitimately longer than the configured timeoutMs; a crashed process left a non-stale lock (owner.json newer than staleMs) that recovery cannot reclaim; two concurrent operations contending with too-short a timeout; a hung process never releasing.

Common situations: Running scan and pipeline concurrently on the same tracker; a prior run was killed -9 leaving owner.json; timeoutMs set too low for a large pipeline batch; a frozen/suspended process holding the lock across a sleep/hibernate.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• LOCK_TIMEOUT
• Could not claim ${count} report slot(s) after ${MAX_RETRIES}
• Cannot verify tracker lock ownership at ${lockDir}
• ${msg}
• Image failed to decode within 10s (unreadable or corrupt fil

AI-assisted analysis of santifer/career-ops@9b17a8ac97 (2026-08-13). Data as JSON: /api/errors/0cd6c9adae001ff1. Report an issue: GitHub.