{"record":{"id":"0cd6c9adae001ff1","repo":"santifer/career-ops","slug":"pipeline-lock-timeout-lockdir-held-timeoutms-ms","errorCode":null,"errorMessage":"pipeline lock timeout: ${lockDir} held > ${timeoutMs}ms","messagePattern":"pipeline lock timeout: (.+?) held > (.+?)ms","errorType":"exception","errorClass":"LockTimeoutError","httpStatus":null,"severity":"error","filePath":"pipeline-lock.mjs","lineNumber":453,"sourceCode":"        err.message += `, last mkdir error=${lastContentionError.code} on ${lastContentionError.path ?? '?'}`;\n      }\n    } catch { /* diagnosis must never mask the timeout it describes */ }\n    return err;\n  };\n\n  // A fresh install may not have data/ yet — plugins.mjs's cmdRun calls\n  // appendToPipeline with no directory pre-creation, so create it here rather\n  // than letting mkdirSync(lockDir) throw a raw ENOENT.\n  mkdirSync(dirname(lockDir), { recursive: true });\n\n  for (;;) {\n    // The ceiling is tested FIRST, on its own, on every pass. Nesting it inside\n    // the per-holder deadline made it conditional on a branch that may never\n    // run: a caller whose maxWaitMs is below timeoutMs never reaches the inner\n    // check, a re-arm pushes the next look a whole window away, and the reclaim\n    // fast-path continues straight past both. A bound that holds only when\n    // another bound happens to fire is not a bound.\n    if (Date.now() > hardDeadline) throw buildTimeoutError(maxWaitMs);\n\n    try {\n      mkdirSync(lockDir);\n    } catch (err) {\n      if (!isMkdirContention(err)) throw err;\n      lastContentionError = err;\n      noteWaiting();\n\n      // Serialize stale-reclaim behind a second atomic guard so only one\n      // caller can be inside the decide-then-delete window at a time.\n      let hasRecoverGuard = false;\n      try {\n        mkdirSync(recoverGuardDir);\n        hasRecoverGuard = true;\n      } catch (guardErr) {\n        if (!isMkdirContention(guardErr)) throw guardErr;\n        lastContentionError = guardErr;\n        // An EPERM/EACCES here says the guard directory is mid-flight, not that","sourceCodeStart":435,"sourceCodeEnd":471,"githubUrl":"https://github.com/santifer/career-ops/blob/aac998c7ed7248ea853b720ceeb1fdbeb322fc5d/pipeline-lock.mjs#L435-L471","documentation":"pipeline-lock.mjs's acquirePipelineLock throws this timeout error when the pipeline lock directory still cannot be acquired after the caller's hard wait ceiling: Date.now() exceeds the hard deadline computed from maxWaitMs, so instead of waiting forever on another process's lock it gives up. The ceiling is checked unconditionally at the top of every retry pass so it always bounds the wait.","triggerScenarios":"Calling acquirePipelineLock (directly or via any pipeline/batch operation) while another process holds the lock dir and its holder deadline hasn't expired; running two concurrent pipeline workers; a crashed process leaving the lock dir in place within the staleness window.","commonSituations":"Launching a batch worker while an interactive scan/pipeline run is still going; an earlier killed process's lock not yet considered stale; NFS/filesystem where mkdir contention is slow.","solutions":["Find and wait for (or kill) the process holding the lock; the message names the lock dir","If the holder is dead, remove the stale lock directory manually once confirmed orphaned","Increase maxWaitMs if concurrent runs are legitimate and you just need a longer wait","Serialize your runs — don't start a second pipeline command until the first exits"],"exampleFix":"// before\nawait acquirePipelineLock(lockDir, { maxWaitMs: 1000 }); // too short under contention\n// after\nawait acquirePipelineLock(lockDir, { maxWaitMs: 30000 });","handlingStrategy":"retry","validationCode":"// before acquiring, check if lock exists and how old it is\ntry { const st = statSync(lockDir); console.log('lock held, age ms:', Date.now() - st.mtimeMs); } catch {}\nawait acquirePipelineLock(lockDir, { maxWaitMs: 30000 });","typeGuard":"const lockLooksStale = (st, maxAgeMs) => Date.now() - st.mtimeMs > maxAgeMs;","tryCatchPattern":"try {\n  await acquirePipelineLock(lockDir, { maxWaitMs: 30000 });\n} catch (e) {\n  if (/pipeline lock timeout/.test(e.message)) {\n    // inspect holder, wait or clean stale dir, then retry once\n  } else throw e;\n}","preventionTips":["Serialize pipeline runs — one worker at a time","Set maxWaitMs generously relative to expected run length","Never hard-kill a process mid-run; let it release the lock","Check for a live holder before manually deleting a lock dir"],"tags":["locking","concurrency","timeout"],"backgroundTag":"request-timeout","analyzedSha":"aac998c7ed7248ea853b720ceeb1fdbeb322fc5d","analyzedAt":"2026-09-16T06:35:29.214Z","contentChangedAt":"2026-09-16T06:35:29.214Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}