{"record":{"id":"3b65bb00bdd493e9","repo":"ruvnet/ruflo","slug":"timed-out-acquiring-repo-supervisor-lock","errorCode":null,"errorMessage":"timed out acquiring repo-supervisor lock","messagePattern":"timed out acquiring repo-supervisor lock","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"v3/@claude-flow/cli/src/services/repo-supervisor.ts","lineNumber":121,"sourceCode":"      try {\n        const fd = fs.openSync(lockFile, fs.constants.O_CREAT | fs.constants.O_EXCL | fs.constants.O_WRONLY, 0o600);\n        fs.writeSync(fd, String(process.pid));\n        fs.closeSync(fd);\n        try {\n          return fn();\n        } finally {\n          try { fs.unlinkSync(lockFile); } catch { /* already gone */ }\n        }\n      } catch (e) {\n        if ((e as NodeJS.ErrnoException).code !== 'EEXIST') throw e;\n        try {\n          const st = fs.lstatSync(lockFile);\n          if (Date.now() - st.mtimeMs > LOCK_STALE_MS) {\n            fs.unlinkSync(lockFile);\n            continue;\n          }\n        } catch { /* raced — retry */ }\n        if (Date.now() > deadline) throw new Error('timed out acquiring repo-supervisor lock');\n        await delay(25);\n      }\n    }\n  }\n\n  private readRecord(repositoryId: string): SupervisorRecord | null {\n    const file = this.fileFor(repositoryId);\n    assertNotSymlink(file);\n    if (!fs.existsSync(file)) return null;\n    try {\n      const raw = JSON.parse(fs.readFileSync(file, 'utf-8'));\n      if (raw && typeof raw.pid === 'number' && typeof raw.lastHeartbeat === 'number') {\n        return raw as SupervisorRecord;\n      }\n    } catch { /* corrupt — treat as absent */ }\n    return null;\n  }\n","sourceCodeStart":103,"sourceCodeEnd":139,"githubUrl":"https://github.com/ruvnet/ruflo/blob/fa13ee4ad60ac2090b1480656eb233521790d640/v3/@claude-flow/cli/src/services/repo-supervisor.ts#L103-L139","documentation":"Thrown by RepoSupervisorRegistry.withLock() when an O_CREAT|O_EXCL lockfile (<record>.json.lock) cannot be created within a hard 2-second deadline (25ms retry loop). A lock left by a crashed process is auto-reclaimed once older than LOCK_STALE_MS (10s), so this error means another daemon held a *fresh* lock for the entire 2s window — the election body (fn) runs while holding the lock, so a slow or contended critical section on a peer extends hold time.","triggerScenarios":"Many worktrees of the same repository starting daemons simultaneously (each ticks the election); the peer inside withLock does slow fs work on a cold NFS/lazy network home; the Node event loop of the lock holder is blocked, delaying its unlink; a just-crashed process left a lock younger than 10s.","commonSituations":"Monorepos checked out as 10+ git worktrees with a daemon in each; RUFLO_AI_BUDGET_DIR on NFS where fs.writeSync/unlinkSync latency is high; CI machines fan-out starting daemons at the same instant; the 2s deadline being shorter than a peer's legitimate critical section under heavy load.","solutions":["Simplest: do nothing this tick — the daemon retries election on its next lifecycle tick; contention resolves itself once one daemon wins.","Stagger daemon startup (jittered delays) so N worktrees don't all contest the lock in the same 2s window.","Move RUFLO_AI_BUDGET_DIR (and thus the registry) to local disk instead of NFS/network home to make the critical section fast.","If it persists: lsof/stat the .lock file — a fresh mtime means a live peer is slow (reduce concurrent daemons); an old mtime means the 10s stale-reclaim will clear it within seconds, so just wait."],"exampleFix":"// before: N worktrees start daemons at once\nawait Promise.all(worktrees.map(w => startDaemon(w))); // several throw 'timed out acquiring repo-supervisor lock'\n\n// after: jittered startup lets one daemon win, others participate\nawait Promise.all(worktrees.map(async (w, i) => {\n  await new Promise(r => setTimeout(r, i * 250));\n  await startDaemon(w); // losers just retry next tick\n}));","handlingStrategy":"retry","validationCode":"import * as fs from 'fs';\n\n// best-effort pre-check: is the lock free or stale (reclaimable) right now?\nfunction lockAcquirable(lockFile: string, staleMs = 10_000): boolean {\n  try { return Date.now() - fs.lstatSync(lockFile).mtimeMs > staleMs; }\n  catch (e) { return (e as NodeJS.ErrnoException).code === 'ENOENT'; }\n}","typeGuard":null,"tryCatchPattern":"try {\n  return await registry.tryAcquireSupervision(worktreeRoot);\n} catch (e) {\n  if (e instanceof Error && e.message === 'timed out acquiring repo-supervisor lock') {\n    // Contention, not corruption: another worktree's daemon holds a fresh lock.\n    // Skip this tick — the next lifecycle tick (~60s) retries election.\n    return { isSupervisor: false, record: null };\n  }\n  throw e;\n}","preventionTips":["Stagger daemon startup across worktrees (jitter) so N daemons don't contest the lock in the same 2s window.","Keep RUFLO_AI_BUDGET_DIR on local disk — NFS latency stretches the lock-holder's critical section past the 2s deadline.","Ensure daemons exit cleanly so lockfiles are unlinked; crashed locks self-heal after 10s via stale reclaim.","Design election calls to be idempotent and skippable: losing this tick is always safe."],"tags":["concurrency","file-lock","timeout","repo-supervisor","election"],"backgroundTag":"file-lock-timeout","analyzedSha":"fa13ee4ad60ac2090b1480656eb233521790d640","analyzedAt":"2026-08-18T21:34:22.708Z","contentChangedAt":"2026-08-18T21:34:22.708Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}