{"record":{"id":"11adb99891115794","repo":"KeygraphHQ/shannon","slug":"git-checkpoint-failed","errorCode":"GIT_CHECKPOINT_FAILED","errorMessage":"Git command failed after ${maxRetries} retries","messagePattern":"Git command failed after (.+?) retries","errorType":"exception","errorClass":"PentestError","httpStatus":null,"severity":"error","filePath":"apps/worker/src/services/git-manager.ts","lineNumber":227,"sourceCode":"      return result;\n    } catch (error) {\n      const errMsg = error instanceof Error ? error.message : String(error);\n\n      if (isGitLockError(errMsg) && attempt < maxRetries) {\n        const delay = 2 ** (attempt - 1) * 1000;\n        // executeGitCommandWithRetry is also called outside activity context\n        // (e.g., from resume logic), so we use console.warn as a fallback here\n        console.warn(\n          `Git lock conflict during ${description} (attempt ${attempt}/${maxRetries}). Retrying in ${delay}ms...`,\n        );\n        await new Promise((resolve) => setTimeout(resolve, delay));\n        continue;\n      }\n\n      throw error;\n    }\n  }\n  throw new PentestError(\n    `Git command failed after ${maxRetries} retries`,\n    'filesystem',\n    true, // Retryable - transient git lock issues\n    { maxRetries, description },\n    ErrorCode.GIT_CHECKPOINT_FAILED,\n  );\n}\n\n// Two-phase reset: hard reset (tracked files) + clean (untracked files).\n// When paths is provided, the untracked clean is scoped to those paths so a\n// failing agent's rollback can't delete a concurrent sibling agent's scratch.\nexport async function rollbackGitWorkspace(\n  sourceDir: string,\n  reason: string = 'retry preparation',\n  logger: ActivityLogger,\n  paths?: readonly string[],\n): Promise<GitOperationResult> {\n  // Skip git operations if not a git repository","sourceCodeStart":209,"sourceCodeEnd":245,"githubUrl":"https://github.com/KeygraphHQ/shannon/blob/1ae0a142f8525410a688f0309fd003cc5b1d92de/apps/worker/src/services/git-manager.ts#L209-L245","documentation":"Thrown by executeGitCommandWithRetry when a git invocation (checkpoint commit, reset, clean, rev-list) cannot complete within the retry budget. The function retries only on git lock contention (index.lock, 'unable to lock', 'Another git process', etc.) using exponential backoff (2^(attempt-1) * 1000ms, default 5 attempts); any non-lock git error is re-thrown immediately. The PentestError is marked retryable=true with code GIT_CHECKPOINT_FAILED. Note: on the final retry attempt the raw underlying exec error is actually re-thrown at the `throw error` line, so this specific PentestError message is a defensive terminal guard for the loop-exit path.","triggerScenarios":"Calling executeGitCommandWithRetry (directly or via rollbackGitWorkspace / saveCheckpoint / findLatestCommit / restoreGitCheckpoint) while another git process or a sibling agent holds the repo's index.lock for longer than the entire backoff window (sum of 1s+2s+4s+8s = ~15s across 5 attempts). Also when concurrent parallel agents in the vuln/exploit phases checkpoint into the same private deliverables git repo and the global git lock context (withGitRepoLock) serializes but the OS-level .git/index.lock is stale or held.","commonSituations":"A crashed prior worker left a stale .git/index.lock in the workspace deliverables repo. Running two scans against the same repo/workspace simultaneously. Filesystem (NFS/Docker overlay) latency making git lock acquisition exceed the backoff window. The repo lives on a slow bind-mount inside the worker container.","solutions":["Inspect the workspace deliverables .git dir for a stale index.lock and remove it: rm -f <repo>/.git/index.lock (and any *.lock), then resume.","Ensure only one scan targets a given workspace/repo at a time; parallel agents already share an in-process lock but two CLI invocations do not.","Increase maxRetries where the call site accepts it, or widen the backoff window if the backing filesystem is known-slow.","Re-run from a clean workspace (./shannon stop --clean) if the git repo state is irrecoverably corrupted.","Check container mount performance; move the repo off a network/overlay filesystem onto local disk."],"exampleFix":"// before: stale lock blocks all retries\n//   ls <repo>/.git/index.lock  -> file exists from a crashed worker\n// after: clear the stale lock, then resume\n//   rm -f <repo>/.git/index.lock\n//   ./shannon start -u <url> -r <repo> -w <same-workspace>","handlingStrategy":"retry","validationCode":"// Before checkpointing, confirm no stale lock and that the repo is a git repo\nimport { existsSync } from 'node:fs';\nconst lockPath = path.join(repoPath, '.git', 'index.lock');\nif (existsSync(lockPath)) {\n  const stat = await fs.stat(lockPath);\n  const ageMs = Date.now() - stat.mtimeMs;\n  if (ageMs > 60_000) await fs.remove(lockPath); // stale > 1min\n}\nconst isRepo = await isGitRepository(repoPath);","typeGuard":"// Guard a git command invocation: inputs are a non-empty args array against a real git repo\nfunction canRunGit(commandArgs: string[], sourceDir: string): boolean {\n  return Array.isArray(commandArgs) && commandArgs.length > 0 &&\n    commandArgs.every((a) => typeof a === 'string') &&\n    existsSync(path.join(sourceDir, '.git'));\n}","tryCatchPattern":"try {\n  await executeGitCommandWithRetry(['git', 'commit', '-m', 'cp'], dir, 'checkpoint');\n} catch (e) {\n  if (e instanceof PentestError && e.code === ErrorCode.GIT_CHECKPOINT_FAILED) {\n    // transient lock exhaustion — clear stale lock and retry once, else surface to Temporal\n    await fs.remove(path.join(dir, '.git', 'index.lock')).catch(() => {});\n  }\n  throw e;\n}","preventionTips":["Never run two scans against the same workspace/repo concurrently; the in-process git lock does not cross CLI invocations.","Periodically reap stale .git/index.lock files older than 1 minute in long-lived workspaces.","Keep the repo on local disk inside the container, not a slow network/overlay mount.","For parallel-agent phases, rely on withGitRepoLock to serialize checkpoint writes within one worker."],"tags":["git","filesystem","concurrency","retry","checkpoint","lock-contention"],"backgroundTag":null,"analyzedSha":"1ae0a142f8525410a688f0309fd003cc5b1d92de","analyzedAt":"2026-08-12T17:40:03.583Z","schemaVersion":2},"datasetVersion":"2026-08-12T18:17:37.767Z"}