{"record":{"id":"d4523abf300d1016","repo":"abhigyanpatwari/GitNexus","slug":"analyze-stopped-before-running-out-of-memory-ma","errorCode":null,"errorMessage":"Analyze stopped before running out of memory: ${Math.round(heapUsedNow / 1024 / 1024)}MB of the ${Math.round(heapLimitNow / 1024 / 1024)}MB Node heap in use at parse chunk ${chunkIdx + 1}/${numChunks} (#2649). ${heapPressureRemedy(heapLimitNow)}","messagePattern":"Analyze stopped before running out of memory: (.+?)MB of the (.+?)MB Node heap in use at parse chunk (.+?)/(.+?) \\(#2649\\)\\. (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"gitnexus/src/core/ingestion/pipeline-phases/parse-impl.ts","lineNumber":1272,"sourceCode":"        handleWorkerStartupFailure(err);\n      }\n      pendingRound = { entries: started.entries, missResults };\n    };\n\n    for (let chunkIdx = 0; chunkIdx < numChunks; chunkIdx++) {\n      if (heapProbeEveryN > 0 && chunkIdx > 0 && chunkIdx % heapProbeEveryN === 0) {\n        logHeapProbe(\n          `parse-chunk-${chunkIdx}`,\n          `nodes=${graph.nodeCount} parsedFiles=${allParsedFiles.length}`,\n        );\n      }\n      // #2649 mid-loop heap guard: fail actionably BEFORE V8 enters the\n      // ineffective-mark-compact death spiral (which also falsely times out\n      // healthy workers). The pool is torn down by this function's finally.\n      const heapUsedNow = process.memoryUsage().heapUsed;\n      const heapLimitNow = v8.getHeapStatistics().heap_size_limit;\n      if (shouldAbortForHeapPressure(heapUsedNow, heapLimitNow)) {\n        throw new Error(\n          `Analyze stopped before running out of memory: ${Math.round(heapUsedNow / 1024 / 1024)}MB of the ` +\n            `${Math.round(heapLimitNow / 1024 / 1024)}MB Node heap in use at parse chunk ${chunkIdx + 1}/${numChunks} (#2649). ` +\n            heapPressureRemedy(heapLimitNow),\n        );\n      }\n      const chunkPaths = chunks[chunkIdx];\n      // Start wall-clock for the per-chunk throughput log emitted at end\n      // of this iteration. The gate is computed once above; here we just\n      // sample the clock if the gate is on. Computed when either\n      // NODE_ENV=development OR the operator passed `--verbose`\n      // (GITNEXUS_VERBOSE) — the previous `isDev`-only gate meant\n      // operators running `gitnexus analyze --verbose` in production\n      // never saw the log (M3 from PR #1693 review).\n      const chunkStartMs: number | null = verboseThroughputLog ? Date.now() : null;\n\n      const chunkContentPromise = chunkContentPromises[chunkIdx];\n      if (!chunkContentPromise) {\n        throw new Error(`Missing prefetched parse chunk ${chunkIdx + 1}/${numChunks}`);","sourceCodeStart":1254,"sourceCodeEnd":1290,"githubUrl":"https://github.com/abhigyanpatwari/GitNexus/blob/0d1aed942f0e8b5d3bac27519fff441aceea722d/gitnexus/src/core/ingestion/pipeline-phases/parse-impl.ts#L1254-L1290","documentation":"Mid-loop heap guard in parse-impl.ts (#2649): between parse chunks, if heapUsed exceeds 0.92 of V8's heap_size_limit (shouldAbortForHeapPressure), analyze aborts with heap numbers and a remedy instead of entering V8's ineffective-mark-compact death spiral (2s+ GC pauses that also falsely idle-timeout healthy workers). The remedy (heapPressureRemedy) branches: if the current heap limit sits well below what the RAM-aware auto-sizer would grant (an inherited --max-old-space-size / NODE_OPTIONS pin), drop the pin — GitNexus sizes itself; otherwise the machine is the ceiling and only .gitnexusignore exclusions or bigger hardware help. GITNEXUS_MEMORY=off disables the abort (proceed-at-own-risk).","triggerScenarios":"Analyzing a very large repo with a small pinned heap (e.g. NODE_OPTIONS=--max-old-space-size=2048), inside a container whose cgroup memory limit is far below host RAM, or on a repo whose generated/vendored content inflates node counts past the heap at parse chunk N of M.","commonSituations":"CI images that globally export NODE_OPTIONS with a heap pin; Docker/Kubernetes memory limits; monorepos with vendored node_modules, generated protobuf/graphql code, or huge minified bundles; cgroup-limited build agents on big hosts (the auto-sizer honors the cgroup limit, not raw totalmem).","solutions":["If the message says the machine has more memory available: re-run without the --max-old-space-size pin (remove it from NODE_OPTIONS / node flags) — GitNexus auto-sizes its heap to the machine","If the machine is at its ceiling: add .gitnexusignore entries excluding generated/vendored directories (node_modules, dist, fixtures) to shrink the graph","Run analyze on a machine (or container) with more memory","Last resort only: GITNEXUS_MEMORY=off declines the abort, but the process will likely die to V8 OOM or appear hung instead"],"exampleFix":"# before\n$ NODE_OPTIONS=--max-old-space-size=2048 gitnexus analyze .\n# after\n$ unset NODE_OPTIONS\ngitnexus analyze .","handlingStrategy":"validation","validationCode":"import v8 from 'node:v8';\nimport os from 'node:os';\n\nconst limit = v8.getHeapStatistics().heap_size_limit;\nconst effectiveRam = Number(fs.readFileSync('/sys/fs/cgroup/memory.max', 'utf8')) || os.totalmem(); // honor cgroup limits\nconst autoCap = effectiveRam * 0.75;\nif (limit < autoCap * 0.9) {\n  console.warn('A --max-old-space-size / NODE_OPTIONS heap pin is below the auto-sized cap — unset it before analyze');\n}\nif (projectedHeapNeed > limit * 0.92) {\n  console.warn('Repo likely exceeds heap — add .gitnexusignore excludes or run on a larger machine');\n}","typeGuard":null,"tryCatchPattern":"try {\n  await runAnalyze(repoPath, options);\n} catch (err) {\n  if (err instanceof Error && err.message.startsWith('Analyze stopped before running out of memory')) {\n    // Read the remedy in the message: drop the heap pin, or add .gitnexusignore entries, then re-run once.\n    // Do NOT set GITNEXUS_MEMORY=off in automation — that trades a clean abort for a V8 OOM/hang.\n  }\n  throw err;\n}","preventionTips":["Do not pin --max-old-space-size in CI images or NODE_OPTIONS — GitNexus sizes its heap from effective (cgroup-aware) RAM","Keep .gitnexusignore current: exclude vendored, generated, and minified directories from indexing","Set container memory limits with the biggest repo you index in mind; the guard honors cgroup limits, not host RAM"],"tags":["memory","heap","out-of-memory","analyze","performance","parse"],"backgroundTag":"out-of-memory","analyzedSha":"0d1aed942f0e8b5d3bac27519fff441aceea722d","analyzedAt":"2026-08-20T23:29:22.980Z","contentChangedAt":"2026-08-20T23:29:22.980Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}