{"record":{"id":"d396b0f4da5c2023","repo":"abhigyanpatwari/GitNexus","slug":"worker-workerindex-died-respawning-slot-attem","errorCode":null,"errorMessage":"Worker ${workerIndex} died; respawning slot (attempt ${respawnCount[workerIndex]}/${poolOptions.maxRespawnsPerSlot}).","messagePattern":"Worker (.+?) died; respawning slot \\(attempt (.+?)/(.+?)\\)\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"gitnexus/src/core/ingestion/workers/worker-pool.ts","lineNumber":1805,"sourceCode":"              maxRespawns: poolOptions.maxRespawnsPerSlot,\n              reason,\n            },\n            `Worker ${workerIndex} exceeded respawn budget; dropping slot.`,\n          );\n          await removeWorkerFromSlot(workerIndex, removalMode, reason);\n          activeSlots.delete(workerIndex);\n          if (activeSlots.size === 0) {\n            tripBreaker(\n              new WorkerPoolDispatchError(\n                `${reason}. All ${size} worker slot(s) exhausted their respawn budget.`,\n                quarantine.snapshot(),\n              ),\n            );\n            return;\n          }\n          return;\n        }\n        logger.warn(\n          {\n            workerIndex,\n            respawnCount: respawnCount[workerIndex],\n            reason,\n            excludePaths,\n          },\n          `Worker ${workerIndex} died; respawning slot (attempt ${respawnCount[workerIndex]}/${poolOptions.maxRespawnsPerSlot}).`,\n        );\n        const respawned = await replaceWorker(workerIndex, removalMode, reason);\n        if (!respawned) {\n          activeSlots.delete(workerIndex);\n          if (activeSlots.size === 0) {\n            tripBreaker(\n              new WorkerPoolDispatchError(\n                `${reason}. Replacement worker startup failed and no slots remain.`,\n                quarantine.snapshot(),\n              ),\n            );","sourceCodeStart":1787,"sourceCodeEnd":1823,"githubUrl":"https://github.com/abhigyanpatwari/GitNexus/blob/0d1aed942f0e8b5d3bac27519fff441aceea722d/gitnexus/src/core/ingestion/workers/worker-pool.ts#L1787-L1823","documentation":"The per-death counterpart of the respawn-budget warn: whenever a worker dies and both the per-slot consecutive-failure breaker and the respawn budget still have headroom, the slot logs this warn (workerIndex, respawnCount/maxRespawns, reason, excludePaths) and calls replaceWorker to respawn it. Dispatch continues on the new worker.","triggerScenarios":"Any worker exit event (crash, kill, native abort avoided earlier) that reaches the respawn branch: consecutiveFailuresPerSlot is under the threshold, respawnCount is at or below maxRespawnsPerSlot, so the slot is replaced and the attempt counter is announced.","commonSituations":"Occasional worker OOMs on huge files, sporadic native grammar crashes, CI runners with memory spikes — each death produces one warn and a fresh worker; frequent repeats escalate to the budget-exceeded warn and eventually the breaker.","solutions":["Treat a single occurrence as normal self-healing; correlate the reason field with the job/file being parsed at death time","If repeats cluster on a file, expect quarantine (unattributable) or exclusion (attributed) to stop the bleeding — then address that file","For heap-cap reasons, raise GITNEXUS_WORKER_HEAP_MB or reduce GITNEXUS_WORKER_POOL_SIZE to stop systematic OOM","Track respawnCount in the warns: approaching maxRespawnsPerSlot means the root cause must be fixed before slots start dropping"],"exampleFix":"# before\nexport GITNEXUS_WORKER_HEAP_MB=256   # repeated heap-cap deaths → constant respawn warns\n# after\nexport GITNEXUS_WORKER_HEAP_MB=1024  # deaths stop, respawn warns stop","handlingStrategy":"retry","validationCode":"// Watch respawn pressure and intervene before slots drop:\nconst warns = parseWorkerWarns(logText).filter((w) => w.msg.startsWith('Worker ') && w.msg.includes('died; respawning'));\nconst nearBudget = warns.filter((w) => w.respawnCount >= w.maxRespawns - 1);\nif (nearBudget.length > 0) failFastWithReasons(nearBudget.map((w) => w.reason));","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Correlate each death's reason with the file being parsed at that moment","Fix systematic OOM with GITNEXUS_WORKER_HEAP_MB / smaller pools instead of tolerating respawn churn","Track respawnCount trends across runs — growth always precedes slot exhaustion"],"tags":["worker-pool","crash-recovery","respawn","self-healing"],"backgroundTag":"worker-crash-restart","analyzedSha":"0d1aed942f0e8b5d3bac27519fff441aceea722d","analyzedAt":"2026-08-20T23:29:22.980Z","contentChangedAt":"2026-08-20T23:29:22.980Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}