santifer/career-ops · error · LockTimeoutError

portal-health lock timeout: ${lockDir} held > ${timeoutMs}ms

Error message

portal-health lock timeout: ${lockDir} held > ${timeoutMs}ms

What it means

portal-health-lock.mjs implements a cross-process advisory lock (a directory-based mkdir lock, same idiom as the tracker lock) to serialize writes to data/portal-health.tsv between scan.mjs appenders and read-modify-write cleanups. acquire() loops retrying mkdir until it wins; if the deadline (default 8000ms, configurable via timeoutMs) expires while the lock is still held by another (live) process, it throws a LockTimeoutError. Stale locks (dead owner PID, or aged out) are reclaimed automatically before this fires — so this error specifically means a LIVE process held the lock longer than the budget.

Source

Thrown at portal-health-lock.mjs:160

        // otherwise disable stale recovery forever. The guard normally lives
        // for milliseconds, so an old one is judged by the same age rule.
        if (lockCanRecover(recoverGuardDir, staleMs)) {
          rmSync(recoverGuardDir, { recursive: true, force: true });
        }
      }

      if (hasRecoverGuard) {
        try {
          if (lockCanRecover(lockDir, staleMs)) {
            rmSync(lockDir, { recursive: true, force: true });
            continue; // retry acquisition immediately
          }
        } finally {
          rmSync(recoverGuardDir, { recursive: true, force: true });
        }
      }

      if (Date.now() > deadline) throw new LockTimeoutError(lockDir, timeoutMs);
      await sleep(retryMs);
      continue;
    }

    // Acquired. Record ownership; an owner-less lock would block every future
    // acquirer until the age-out, so clean up if the stamp can't be written.
    try {
      writeFileSync(join(lockDir, 'owner.json'), JSON.stringify({
        pid: process.pid,
        token,
        started_at: new Date().toISOString(),
        file: filePath,
      }, null, 2));
    } catch (ownerErr) {
      rmSync(lockDir, { recursive: true, force: true });
      throw ownerErr;
    }

View on GitHub (pinned to 9b17a8ac97)

Solutions

1. Wait and retry — if a scan is legitimately in progress, it will release the lock; the next acquire succeeds.
2. Check for a hung scan process: `ps aux | grep scan.mjs` and, if it is genuinely stuck (not making progress), kill it so the lock's owner PID dies and stale-reclaim kicks in.
3. As a last resort, remove the lock directory manually: `rm -rf data/portal-health.tsv.lock` AND `data/portal-health.tsv.lock.recover` — only when you are certain no scan is running, since manually deleting a live lock can interleave writers.
4. If contention is expected, raise timeoutMs at the call site (the lock API is caller-configurable for exactly this).

Example fix

# Check for a live holder first
ps aux | grep -E 'scan\.mjs|portal-health'

# If none is genuinely running, remove the stale lock dirs
rm -rf data/portal-health.tsv.lock data/portal-health.tsv.lock.recover

# Then retry the scan

// Detect a likely-stuck lock before acquiring, so you can warn rather than block.
import { existsSync, readFileSync, statSync } from 'fs';
function lockLooksStale(lockDir, staleMs = 30000) {
  if (!existsSync(lockDir)) return false;
  try {
    const owner = JSON.parse(readFileSync(`${lockDir}/owner.json`, 'utf-8'));
    if (owner?.pid && !processExists(owner.pid)) return true; // dead owner
    const age = Date.now() - statSync(lockDir).mtimeMs;
    return age > staleMs;
  } catch {
    return Date.now() - statSync(lockDir).mtimeMs > staleMs;
  }
}
function processExists(pid) { try { process.kill(pid, 0); return true; } catch { return false; } }

/** @param {unknown} e @returns {e is import('portal-health-lock.mjs').LockTimeoutError} */
function isLockTimeout(e) {
  return e instanceof Error && e.name === 'LockTimeoutError' && typeof e.lockDir === 'string';
}

import { acquire, LockTimeoutError } from './portal-health-lock.mjs';
try {
  const release = await acquire(filePath, { timeoutMs: 10000 });
  try { /* ...read-modify-write portal-health.tsv... */ }
  finally { await release(); }
} catch (err) {
  if (err instanceof LockTimeoutError) {
    console.warn(`portal-health lock busy (${err.lockDir}); skipping this write rather than blocking.`);
  } else throw err;
}

When it happens

Trigger: Two or more concurrent processes both writing data/portal-health.tsv (e.g. two `scan` runs launched in parallel, or a scan running while a cleanup/repair job holds the lock) and the first holds it past timeoutMs. Also possible if the holder crashed in a way that left owner.json but the PID is reused by an unrelated long-running process (PID-liveness false positive).

Common situations: Overlapping scheduled scans (cron + manual run); a long-running scan stuck on a slow network fetch while holding the lock; a previous scan killed with SIGKILL whose PID got reused by another process, making a dead lock look live so it never auto-reclaims.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• LOCK_TIMEOUT
• cannot read ${rel || '.'}: ${err.message}
• refusing to hash non-regular file: ${childRel}
• Validation failed (${err.message}) and revert also failed ($
• ROW_NOT_FOUND

AI-assisted analysis of santifer/career-ops@9b17a8ac97 (2026-08-13). Data as JSON: /api/errors/eb81131efe9a20fb. Report an issue: GitHub.