{"record":{"id":"4c657639b9783bfe","repo":"abhigyanpatwari/GitNexus","slug":"worker-i-crashed-during-startup-respawning-slo","errorCode":null,"errorMessage":"Worker ${i} crashed during startup; respawning slot (self-heal attempt ${attempt + 1}/${STARTUP_RESTART_BUDGET}).","messagePattern":"Worker (.+?) crashed during startup; respawning slot \\(self-heal attempt (.+?)/(.+?)\\)\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"gitnexus/src/core/ingestion/workers/worker-pool.ts","lineNumber":1310,"sourceCode":"            { workerIndex: i, attempt, err: msg, deterministic: deterministicStartupDetected },\n            deterministicStartupDetected\n              ? `Worker ${i} hit a deterministic startup crash-loop; dropping slot without further retries.`\n              : `Worker ${i} did not report ready after ${attempt + 1} attempt(s); dropping slot.`,\n          );\n          return;\n        }\n        // Transient: jittered backoff, then respawn the slot and retry.\n        await abortableSleep(\n          startupBackoffMs(attempt),\n          () => terminated || deterministicStartupDetected,\n          pendingStartupTimers,\n        );\n        if (terminated || deterministicStartupDetected) {\n          initialReadinessFailures.push(msg);\n          activeSlots.delete(i);\n          return;\n        }\n        logger.warn(\n          { workerIndex: i, attempt: attempt + 1 },\n          `Worker ${i} crashed during startup; respawning slot (self-heal attempt ${attempt + 1}/${STARTUP_RESTART_BUDGET}).`,\n        );\n        workers[i] = spawnAndCapture(workerUrl);\n      }\n    }\n  };\n\n  // First dispatch awaits this; it settles every slot's bounded retry loop in\n  // parallel and drops the unrecoverable ones before any dispatch can fire.\n  const initialReadyGate: Promise<void> = Promise.allSettled(\n    workers.map((_, i) => bringSlotReady(i)),\n  ).then(() => undefined);\n\n  /**\n   * Guards the one-dispatch-at-a-time contract. The dispatch machinery keeps\n   * its jobs/busy-slot/in-flight state per call, so two concurrent dispatches\n   * hand the same slots out twice: both stall, and the failure surfaces only","sourceCodeStart":1292,"sourceCodeEnd":1328,"githubUrl":"https://github.com/abhigyanpatwari/GitNexus/blob/0d1aed942f0e8b5d3bac27519fff441aceea722d/gitnexus/src/core/ingestion/workers/worker-pool.ts#L1292-L1328","documentation":"Each pool slot runs a bounded startup retry loop: if a worker's readiness fails (crash during startup), the slot backs off with jitter (abortable on termination or deterministic-failure detection) and respawns, up to STARTUP_RESTART_BUDGET (2) self-heal attempts. This warn announces each respawn; unrecoverable slots are recorded in initialReadinessFailures and dropped before the first dispatch.","triggerScenarios":"Worker thread crashing before its readiness handshake — native grammar loading failure, OOM at boot, ABI-incompatible tree-sitter native module, or transient resource pressure; the loop logs the respawn attempt and re-spawns via spawnAndCapture.","commonSituations":"Fresh installs where optional vendored grammars got source-built against a mismatched toolchain, containers with tight memory at boot, transient cgroup/OOM pressure on CI, or flaky worker script loading under heavy FS contention.","solutions":["Check the worker's stderr/first-failure message (collected into initialReadinessFailures) to identify whether the crash is native-load, OOM, or transient","Verify the install: matching prebuilt grammar binaries, or a C/C++ toolchain (python3, make, g++) present when source-building; consider GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1 for optional languages","Reduce boot memory pressure: lower GITNEXUS_WORKER_POOL_SIZE / GITNEXUS_WORKER_HEAP_MB per the overcommit guidance","If failures are deterministic, the slot is dropped automatically — read the surfaced readiness error rather than retrying blindly"],"exampleFix":"# before\nnpm install   # optional grammar source-build mismatch → worker crashes at startup\n# after\nexport GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1\nnpm install && npx gitnexus analyze   # optional grammars skipped, startup clean","handlingStrategy":"retry","validationCode":"// De-risk startup before invoking analyze:\nconst missing = ['python3', 'make', 'g++'].filter((t) => !hasTool(t));\nif (missing.length > 0) process.env.GITNEXUS_SKIP_OPTIONAL_GRAMMARS = '1';\nif (!process.env.GITNEXUS_WORKER_POOL_SIZE) {\n  process.env.GITNEXUS_WORKER_POOL_SIZE = String(Math.min(4, os.cpus().length));\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Pin one CLI version for install and run so native grammar binaries match","Free boot memory pressure: smaller pool/heap on constrained containers","If readiness failures are deterministic, read the surfaced failure message instead of relying on the 2-attempt self-heal"],"tags":["worker-pool","startup","crash-recovery","native-modules","install"],"backgroundTag":"worker-crash-restart","analyzedSha":"0d1aed942f0e8b5d3bac27519fff441aceea722d","analyzedAt":"2026-08-20T23:29:22.980Z","contentChangedAt":"2026-08-20T23:29:22.980Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}