abhigyanpatwari/GitNexus · warning

Worker ${workerIndex} replacement failed to come online; dro

Error message

Worker ${workerIndex} replacement failed to come online; dropping slot.

What it means

Emitted by the ingestion WorkerPool when a replacement worker thread (spawned after a crash/removal via replaceWorker) fails to post its ready signal within poolOptions.workerReadyTimeoutMs, or errors during boot. The pool terminates the half-started replacement and drops the slot, returning false from replaceWorker. Indexing continues with one fewer parallel lane, so throughput degrades but the run does not abort.

Source

Thrown at gitnexus/src/core/ingestion/workers/worker-pool.ts:1474

          retireWorkerAfterTimeout(existing, workerIndex, reason);
          return;
        }
        await existing.terminate().catch(() => undefined);
      };

      const replaceWorker = async (
        workerIndex: number,
        mode: WorkerRemovalMode = 'terminate',
        reason = 'replacing worker',
      ): Promise<boolean> => {
        await removeWorkerFromSlot(workerIndex, mode, reason);
        if (stopped) return false;
        const replacement = spawnAndCapture(workerUrl);
        try {
          await waitForWorkerReady(replacement, poolOptions.workerReadyTimeoutMs);
        } catch (err) {
          await replacement.terminate().catch(() => undefined);
          logger.warn(
            { workerIndex, error: err instanceof Error ? err.message : String(err) },
            `Worker ${workerIndex} replacement failed to come online; dropping slot.`,
          );
          return false;
        }
        if (stopped) {
          await replacement.terminate().catch(() => undefined);
          return false;
        }
        workers[workerIndex] = replacement;
        // U12: bump the slot generation atomically with the worker swap so
        // any late event from the OLD worker that somehow slipped past
        // cleanup() carries a stale generation and short-circuits in the
        // handler guard below. Increment AFTER `workers[workerIndex]` is
        // updated so observers (getStats) see the new pair consistently.
        slotGenerations[workerIndex]++;
        return true;
      };

View on GitHub (pinned to aac7515d2a)

Solutions

1. Raise the ready timeout: set GITNEXUS_WORKER_READY_TIMEOUT_MS to a larger value (e.g. 120000) or pass workerReadyTimeoutMs in WorkerPoolOptions — the env var and option override DEFAULT_WORKER_READY_TIMEOUT_MS.
2. Check the log lines immediately before this warn for the original removeWorkerFromSlot reason (crash stack, OOM) — fixing that root cause stops the replacement churn.
3. Verify the worker entry script imports cleanly: run `node -e "import('<workerUrl>')"` style smoke load to surface module-level syntax/dependency errors.
4. Reduce pool size (GITNEXUS_WORKER_POOL_SIZE) or free system memory/CPU so the new thread can start within the budget.

Example fix

// before
const pool = new WorkerPool({ size: 8 }); // default ready timeout too small on throttled CI

// after
const pool = new WorkerPool({
  size: 8,
  workerReadyTimeoutMs: 120_000, // or: export GITNEXUS_WORKER_READY_TIMEOUT_MS=120000
});

// Pre-flight: confirm the worker entry module loads in this environment
import { pathToFileURL } from 'node:url';
await import(pathToFileURL(workerUrl).href); // throws early with the real module error
process.env.GITNEXUS_WORKER_READY_TIMEOUT_MS ??= '120000'; // widen boot budget

// replaceWorker resolves boolean, not throws — branch, don't catch
const ok = await pool.replaceWorker?.(i);
if (!ok) {
  await backoff();           // brief pause, then retry the slot fill
  await pool.refillSlot(i);  // or resize/add to re-request a worker
}

When it happens

Trigger: replaceWorker() is invoked internally after removeWorkerFromSlot (worker crash, OOM kill, recycle); spawnAndCapture(workerUrl) creates the new Worker but waitForWorkerReady(replacement, workerReadyTimeoutMs) throws. Typical causes: worker entry module taking longer than the ready timeout to load (cold FS, CI CPU throttling), a module-level exception in the worker bundle, or thread-spawn resource exhaustion (EMFILE, memory pressure).

Common situations: CI runners with constrained CPU where worker boot routinely exceeds the default ready timeout; slow network filesystems hosting node_modules; a worker bundle broken by a dependency update; heavy parallel `gitnexus analyze` runs on machines with few cores; Node major-version upgrades changing worker startup timing.

Related errors

• Circuit '${key}' is open; retry in ${Math.ceil(retryAfterMs
• Request failed after retries (HTTP ${response.status})
• Timed out after ${waitedMs}ms waiting for another gitnexus a
• Timed out after ${waitedMs}ms waiting for another gitnexus a
• cpp range-binding: parse timed out, skipping file

AI-assisted analysis of abhigyanpatwari/GitNexus@aac7515d2a (2026-08-20). Data as JSON: /api/errors/bc4202a76a3b2026. Report an issue: GitHub.