{"record":{"id":"9fb2a9a1a1eaaf41","repo":"abhigyanpatwari/GitNexus","slug":"workload-lock-cleanup-failed-lockpath-acquisition-refused","errorCode":null,"errorMessage":"Workload-lock cleanup failed: ${lockPath}. Acquisition refused; see RUNBOOK.md for quiesced recovery.","messagePattern":"Workload-lock cleanup failed: (.+?)\\. Acquisition refused; see RUNBOOK\\.md for quiesced recovery\\.","errorType":"exception","errorClass":"AggregateError","httpStatus":null,"severity":"critical","filePath":"gitnexus/src/storage/index-lock.ts","lineNumber":669,"sourceCode":"      }\n      permissionWaitSince = null;\n      permissionError = undefined;\n    } catch (error) {\n      if (!createdMain && (error as NodeJS.ErrnoException).code === 'EPERM') {\n        // A releasing owner may leave the main file delete-pending on Windows.\n        // Release our guard in finally and retry; never reclaim an unreadable file.\n        permissionWaitSince ??= Date.now();\n        permissionError = error;\n        holder = null;\n        if (Date.now() >= Math.min(startedAt + timeoutMs, permissionWaitSince + GUARD_TIMEOUT_MS)) {\n          throw error;\n        }\n      } else {\n        if (createdMain) {\n          try {\n            if (readRecord(lockPath)?.token === me.token) unlinkSync(lockPath);\n          } catch (cleanupError) {\n            throw new AggregateError(\n              [error, cleanupError],\n              `Workload-lock cleanup failed: ${lockPath}. Acquisition refused; see RUNBOOK.md for quiesced recovery.`,\n            );\n          }\n        }\n        throw error;\n      }\n    } finally {\n      releaseAcquisitionGuard(guardPath, me, createdMain, lockPath);\n    }\n    const waited = Date.now() - startedAt;\n\n    if (holder) {\n      // Live holder → wait.\n      lastLiveHolder = holder;\n      if (!announcedWait) {\n        announcedWait = true;\n        opts.onWaitStart?.(holder);","sourceCodeStart":651,"sourceCodeEnd":687,"githubUrl":"https://github.com/abhigyanpatwari/GitNexus/blob/ac9a4e9abd8fd3058c070b72c23402a4f887929a/gitnexus/src/storage/index-lock.ts#L651-L687","documentation":"This AggregateError is thrown by acquireViaFile when a non-permission error occurred while trying to claim the workload lock while the guard was held, and the attempt's own token-exact analyze.lock record then failed to unlink during rollback. The acquisition is refused (no handle is returned) but stale lock state may remain under .gitnexus/, so recovery requires quiescing writers and following RUNBOOK.md. Both the original error and the cleanup error are surfaced via AggregateError.errors.","triggerScenarios":"In the catch block of the guarded section: an error other than a retryable EPERM (e.g. guard verification failure, write failure) with createdMain=true, and the rollback readRecord/unlinkSync on analyze.lock throws (EACCES, ENOENT-then-race, EIO, read-only remount).","commonSituations":"Disk filling up or filesystem going read-only mid-acquisition; permissions on .gitnexus/ changed while analyze ran; concurrent deletion of the lock file by a tool or another user; flaky network mounts dropping unlinks; interrupted mounts or dying disks.","solutions":["Quiesce all writers, inspect both errors in AggregateError.errors, fix the underlying filesystem fault (space, permissions, writability), then follow RUNBOOK.md to clear leftover analyze.lock/guard and retry.","Verify .gitnexus/ is writable by the running user (ls -ld, touch a test file) and that the mount is not read-only (mount | grep ro).","Free disk space if ENOSPC appears in the wrapped errors.","Exclude .gitnexus/ from sync/AV tools and move off NFS if unlink races persist.","As a last resort with no writers active, remove the whole .gitnexus/ slot and re-run analyze to rebuild."],"exampleFix":"// before (mid-run read-only remount leaves the lock)\n$ npx gitnexus analyze\n# AggregateError: Workload-lock cleanup failed: .gitnexus/analyze.lock ...\n\n// after (check writability first, then retry)\n$ mount | grep $(df .gitnexus --output=source | tail -1)   # ensure rw\n$ touch .gitnexus/.writetest && rm .gitnexus/.writetest\n$ npx gitnexus analyze","handlingStrategy":"try-catch","validationCode":"// Pre-flight: writable dir, free space, rw mount\nimport { accessSync, constants, statfsSync } from 'node:fs';\naccessSync('.gitnexus', constants.W_OK);\nconst { bavail, bsize } = statfsSync('.gitnexus');\nif (bavail * bsize < 10 * 1024 * 1024) throw new Error('.gitnexus nearly full');","typeGuard":"const isWorkloadCleanupAggregate = (e: unknown): e is AggregateError & { errors: [unknown, unknown] } =>\n  e instanceof AggregateError &&\n  typeof e.message === 'string' &&\n  e.message.startsWith('Workload-lock cleanup failed');","tryCatchPattern":"try {\n  await acquireIndexLock(lockDir);\n} catch (err) {\n  if (isWorkloadCleanupAggregate(err)) {\n    const [error, cleanupError] = err.errors;\n    console.error('acquisition refused; quiesce writers, fix fs fault, see RUNBOOK.md', error, cleanupError);\n    return; // never write without a verified lock\n  }\n  throw err;\n}","preventionTips":["Ensure disk space and a writable (rw) mount before long analyze runs.","Keep .gitnexus/ on a local filesystem, off NFS, and excluded from sync/AV tools.","Treat every AggregateError as fail-closed: quiesce, inspect err.errors, run RUNBOOK recovery.","Watch for permission changes on .gitnexus/ mid-run (CI runners switching users)."],"tags":["filesystem","locking","cleanup","io"],"backgroundTag":"lock-cleanup-failed","analyzedSha":"ac9a4e9abd8fd3058c070b72c23402a4f887929a","analyzedAt":"2026-09-15T23:29:44.066Z","contentChangedAt":"2026-09-15T23:29:44.066Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}