{"record":{"id":"bc4202a76a3b2026","repo":"abhigyanpatwari/GitNexus","slug":"worker-workerindex-replacement-failed-to-come-o","errorCode":null,"errorMessage":"Worker ${workerIndex} replacement failed to come online; dropping slot.","messagePattern":"Worker (.+?) replacement failed to come online; dropping slot\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"gitnexus/src/core/ingestion/workers/worker-pool.ts","lineNumber":1474,"sourceCode":"          retireWorkerAfterTimeout(existing, workerIndex, reason);\n          return;\n        }\n        await existing.terminate().catch(() => undefined);\n      };\n\n      const replaceWorker = async (\n        workerIndex: number,\n        mode: WorkerRemovalMode = 'terminate',\n        reason = 'replacing worker',\n      ): Promise<boolean> => {\n        await removeWorkerFromSlot(workerIndex, mode, reason);\n        if (stopped) return false;\n        const replacement = spawnAndCapture(workerUrl);\n        try {\n          await waitForWorkerReady(replacement, poolOptions.workerReadyTimeoutMs);\n        } catch (err) {\n          await replacement.terminate().catch(() => undefined);\n          logger.warn(\n            { workerIndex, error: err instanceof Error ? err.message : String(err) },\n            `Worker ${workerIndex} replacement failed to come online; dropping slot.`,\n          );\n          return false;\n        }\n        if (stopped) {\n          await replacement.terminate().catch(() => undefined);\n          return false;\n        }\n        workers[workerIndex] = replacement;\n        // U12: bump the slot generation atomically with the worker swap so\n        // any late event from the OLD worker that somehow slipped past\n        // cleanup() carries a stale generation and short-circuits in the\n        // handler guard below. Increment AFTER `workers[workerIndex]` is\n        // updated so observers (getStats) see the new pair consistently.\n        slotGenerations[workerIndex]++;\n        return true;\n      };","sourceCodeStart":1456,"sourceCodeEnd":1492,"githubUrl":"https://github.com/abhigyanpatwari/GitNexus/blob/aac7515d2a8c50a1f8f923c6fb77218b333560d6/gitnexus/src/core/ingestion/workers/worker-pool.ts#L1456-L1492","documentation":"Emitted by the ingestion WorkerPool when a replacement worker thread (spawned after a crash/removal via replaceWorker) fails to post its ready signal within poolOptions.workerReadyTimeoutMs, or errors during boot. The pool terminates the half-started replacement and drops the slot, returning false from replaceWorker. Indexing continues with one fewer parallel lane, so throughput degrades but the run does not abort.","triggerScenarios":"replaceWorker() is invoked internally after removeWorkerFromSlot (worker crash, OOM kill, recycle); spawnAndCapture(workerUrl) creates the new Worker but waitForWorkerReady(replacement, workerReadyTimeoutMs) throws. Typical causes: worker entry module taking longer than the ready timeout to load (cold FS, CI CPU throttling), a module-level exception in the worker bundle, or thread-spawn resource exhaustion (EMFILE, memory pressure).","commonSituations":"CI runners with constrained CPU where worker boot routinely exceeds the default ready timeout; slow network filesystems hosting node_modules; a worker bundle broken by a dependency update; heavy parallel `gitnexus analyze` runs on machines with few cores; Node major-version upgrades changing worker startup timing.","solutions":["Raise the ready timeout: set GITNEXUS_WORKER_READY_TIMEOUT_MS to a larger value (e.g. 120000) or pass workerReadyTimeoutMs in WorkerPoolOptions — the env var and option override DEFAULT_WORKER_READY_TIMEOUT_MS.","Check the log lines immediately before this warn for the original removeWorkerFromSlot reason (crash stack, OOM) — fixing that root cause stops the replacement churn.","Verify the worker entry script imports cleanly: run `node -e \"import('<workerUrl>')\"` style smoke load to surface module-level syntax/dependency errors.","Reduce pool size (GITNEXUS_WORKER_POOL_SIZE) or free system memory/CPU so the new thread can start within the budget."],"exampleFix":"// before\nconst pool = new WorkerPool({ size: 8 }); // default ready timeout too small on throttled CI\n\n// after\nconst pool = new WorkerPool({\n  size: 8,\n  workerReadyTimeoutMs: 120_000, // or: export GITNEXUS_WORKER_READY_TIMEOUT_MS=120000\n});","handlingStrategy":"retry","validationCode":"// Pre-flight: confirm the worker entry module loads in this environment\nimport { pathToFileURL } from 'node:url';\nawait import(pathToFileURL(workerUrl).href); // throws early with the real module error\nprocess.env.GITNEXUS_WORKER_READY_TIMEOUT_MS ??= '120000'; // widen boot budget","typeGuard":null,"tryCatchPattern":"// replaceWorker resolves boolean, not throws — branch, don't catch\nconst ok = await pool.replaceWorker?.(i);\nif (!ok) {\n  await backoff();           // brief pause, then retry the slot fill\n  await pool.refillSlot(i);  // or resize/add to re-request a worker\n}","preventionTips":["Set GITNEXUS_WORKER_READY_TIMEOUT_MS generously on throttled CI/low-core machines.","Smoke-import the worker entry file after dependency upgrades to catch broken bundles before a run.","Monitor pool size / warn rate for 'replacement failed' and alert when slots repeatedly drop.","Cap GITNEXUS_WORKER_POOL_SIZE below physical core count on loaded machines so boot isn't starved."],"tags":["worker-threads","timeout","ingestion","concurrency","degraded-throughput"],"backgroundTag":"worker-startup-timeout","analyzedSha":"aac7515d2a8c50a1f8f923c6fb77218b333560d6","analyzedAt":"2026-08-20T23:29:22.980Z","schemaVersion":2},"datasetVersion":"2026-08-22T14:17:55.899Z"}