{"record":{"id":"a20c72b1d66e568f","repo":"abhigyanpatwari/GitNexus","slug":"worker-pool-may-overcommit-memory-size-workers","errorCode":null,"errorMessage":"Worker pool may overcommit memory: ${size} workers × ${workerHeapCapMb}MB heap cap exceeds 60% of the ${effectiveMb}MB available to this process. Reduce GITNEXUS_WORKER_POOL_SIZE or set GITNEXUS_WORKER_HEAP_MB.","messagePattern":"Worker pool may overcommit memory: (.+?) workers × (.+?)MB heap cap exceeds 60% of the (.+?)MB available to this process\\. Reduce GITNEXUS_WORKER_POOL_SIZE or set GITNEXUS_WORKER_HEAP_MB\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"gitnexus/src/core/ingestion/workers/worker-pool.ts","lineNumber":1083,"sourceCode":"  const parsedFileStoreStoragePath = options?.parsedFileStoreStoragePath;\n  const durableParsedFileStoragePath = options?.durableParsedFileStoragePath;\n  // CFG/PDG opt-in (#2081 M1) — carried in workerData alongside the store paths.\n  const pdg = options?.pdg === true;\n  const pdgMaxFunctionLines = options?.pdgMaxFunctionLines;\n  const workerStoreData =\n    parsedFileStoreStoragePath || durableParsedFileStoragePath || pdg\n      ? { parsedFileStoreStoragePath, durableParsedFileStoragePath, pdg, pdgMaxFunctionLines }\n      : undefined;\n  const workerHeapCapMb = resolveWorkerHeapCapMb(size);\n  // The 512MB per-worker floor exists so a worker can parse anything real,\n  // but on a very small container a large pool of floored workers can still\n  // overcommit total memory (#2649 review). Behavior is unchanged — deaths\n  // are attributed and quarantine converges — but say so up front, with the\n  // two levers, instead of letting the operator discover it from worker OOMs.\n  const poolCommitMb = workerHeapCapMb * size;\n  const effectiveMb = Math.floor(effectiveRamBytes() / (1024 * 1024));\n  if (poolCommitMb > 0.6 * effectiveMb) {\n    logger.warn(\n      { poolSize: size, workerHeapCapMb, effectiveMb },\n      `Worker pool may overcommit memory: ${size} workers × ${workerHeapCapMb}MB heap cap exceeds 60% of the ${effectiveMb}MB available to this process. Reduce GITNEXUS_WORKER_POOL_SIZE or set GITNEXUS_WORKER_HEAP_MB.`,\n    );\n  }\n  // #2649 stall probe: test seam wins; production uses the heartbeat tracker.\n  const stallTracker = options?.stallMsProbe\n    ? { read: options.stallMsProbe, stop: (): void => undefined }\n    : startHeartbeatStallTracker();\n  const spawnWorker =\n    options?.workerFactory ??\n    ((url: URL) =>\n      new Worker(url, {\n        // Piped (not inherited) stdio: stderr for crash capture (#1741),\n        // stdout because inherited stdout triggers silent startup crashes on\n        // some hosts (see forwardWorkerStdout).\n        stdout: true,\n        stderr: true,\n        workerData: workerStoreData,","sourceCodeStart":1065,"sourceCodeEnd":1101,"githubUrl":"https://github.com/abhigyanpatwari/GitNexus/blob/0d1aed942f0e8b5d3bac27519fff441aceea722d/gitnexus/src/core/ingestion/workers/worker-pool.ts#L1065-L1101","documentation":"At pool construction the per-worker heap cap (512MB floor, or GITNEXUS_WORKER_HEAP_MB) times the pool size is compared against process-available RAM. If the committed total exceeds 60% of effective memory, the pool warns about potential overcommit (#2649 review): behavior is unchanged (deaths stay attributed, quarantine converges) but the operator is told up front instead of discovering it via worker OOMs.","triggerScenarios":"Creating the worker pool with size × workerHeapCapMb > 0.6 × effectiveMb — e.g. a large GITNEXUS_WORKER_POOL_SIZE on a small container, or the 512MB floor applying across many workers on a low-RAM box; effectiveRamBytes() determines the denominator.","commonSituations":"CI containers with 1–2GB RAM and default pool sizing, self-hosted runners where pool size was copied from a beefier machine, or GITNEXUS_WORKER_HEAP_MB raised without accounting for worker count.","solutions":["Reduce GITNEXUS_WORKER_POOL_SIZE (or pass --workers N) so size × heap ≤ ~60% of available RAM","Set GITNEXUS_WORKER_HEAP_MB explicitly to size the per-worker cap for your container rather than relying on the 512MB floor","Run on/allocate more memory if the parallelism is genuinely needed for indexing throughput","Treat subsequent worker OOM deaths (heap-cap messages) as confirmation — raise memory or lower parallelism, not retries"],"exampleFix":"# before (2GB container)\nexport GITNEXUS_WORKER_POOL_SIZE=8   # 8 × 512MB = 4096MB > 60% of 2048MB → warn\n# after\nexport GITNEXUS_WORKER_POOL_SIZE=2\nexport GITNEXUS_WORKER_HEAP_MB=512   # 1024MB ≤ 1228MB budget → no warn","handlingStrategy":"validation","validationCode":"// Pre-flight the commit before constructing the pool:\nconst size = Number(process.env.GITNEXUS_WORKER_POOL_SIZE ?? defaultPoolSize());\nconst heapMb = Number(process.env.GITNEXUS_WORKER_HEAP_MB ?? 512);\nconst availableMb = Math.floor(os.totalmem() / 1024 / 1024); // or cgroup limit\nif (size * heapMb > 0.6 * availableMb) {\n  process.env.GITNEXUS_WORKER_POOL_SIZE = String(Math.max(1, Math.floor((0.6 * availableMb) / heapMb)));\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Size pools against the cgroup/container limit, not host RAM, in CI","Set GITNEXUS_WORKER_HEAP_MB explicitly instead of inheriting the 512MB floor blindly","Treat any later heap-cap worker death as a sizing bug, not a retry candidate"],"tags":["worker-pool","memory","configuration","resource-planning"],"backgroundTag":"memory-overcommit","analyzedSha":"0d1aed942f0e8b5d3bac27519fff441aceea722d","analyzedAt":"2026-08-20T23:29:22.980Z","contentChangedAt":"2026-08-20T23:29:22.980Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}