{"record":{"id":"53a0ed6edfc61d1c","repo":"abhigyanpatwari/GitNexus","slug":"analyze-stopped-before-running-out-of-memory-he","errorCode":null,"errorMessage":"Analyze stopped before running out of memory: ${heapUsedMB}MB of the ${heapLimitMB}MB Node heap in use at parse chunk ${chunkIdx + 1}/${numChunks} (#2649). ${heapPressureRemedy}","messagePattern":"Analyze stopped before running out of memory: (.+?)MB of the (.+?)MB Node heap in use at parse chunk (.+?)/(.+?) \\(#2649\\)\\. (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"gitnexus/src/core/ingestion/pipeline-phases/parse-impl.ts","lineNumber":971,"sourceCode":"        }\n      }\n      await applyChunkResults(chunkWorkerData, p.chunkIdx, p.chunkFiles, p.chunkStartMs);\n    };\n\n    for (let chunkIdx = 0; chunkIdx < numChunks; chunkIdx++) {\n      if (heapProbeEveryN > 0 && chunkIdx > 0 && chunkIdx % heapProbeEveryN === 0) {\n        logHeapProbe(\n          `parse-chunk-${chunkIdx}`,\n          `nodes=${graph.nodeCount} parsedFiles=${allParsedFiles.length}`,\n        );\n      }\n      // #2649 mid-loop heap guard: fail actionably BEFORE V8 enters the\n      // ineffective-mark-compact death spiral (which also falsely times out\n      // healthy workers). The pool is torn down by this function's finally.\n      const heapUsedNow = process.memoryUsage().heapUsed;\n      const heapLimitNow = v8.getHeapStatistics().heap_size_limit;\n      if (shouldAbortForHeapPressure(heapUsedNow, heapLimitNow)) {\n        throw new Error(\n          `Analyze stopped before running out of memory: ${Math.round(heapUsedNow / 1024 / 1024)}MB of the ` +\n            `${Math.round(heapLimitNow / 1024 / 1024)}MB Node heap in use at parse chunk ${chunkIdx + 1}/${numChunks} (#2649). ` +\n            heapPressureRemedy(heapLimitNow),\n        );\n      }\n      const chunkPaths = chunks[chunkIdx];\n      // Start wall-clock for the per-chunk throughput log emitted at end\n      // of this iteration. The gate is computed once above; here we just\n      // sample the clock if the gate is on. Computed when either\n      // NODE_ENV=development OR the operator passed `--verbose`\n      // (GITNEXUS_VERBOSE) — the previous `isDev`-only gate meant\n      // operators running `gitnexus analyze --verbose` in production\n      // never saw the log (M3 from PR #1693 review).\n      const chunkStartMs: number | null = verboseThroughputLog ? Date.now() : null;\n\n      const chunkContentPromise = chunkContentPromises[chunkIdx];\n      if (!chunkContentPromise) {\n        throw new Error(`Missing prefetched parse chunk ${chunkIdx + 1}/${numChunks}`);","sourceCodeStart":953,"sourceCodeEnd":989,"githubUrl":"https://github.com/abhigyanpatwari/GitNexus/blob/d540b00184d71a896261ee02670da9a92d59d8f7/gitnexus/src/core/ingestion/pipeline-phases/parse-impl.ts#L953-L989","documentation":"A mid-loop heap guard that aborts the parse phase BEFORE V8 enters its ineffective-mark-compact death spiral. It samples `process.memoryUsage().heapUsed` against `v8.getHeapStatistics().heap_size_limit` at each chunk boundary and throws when heap use exceeds 92% (`HEAP_ABORT_FRACTION`) of the limit. The throw is intentional: above ~95% V8 produces multi-second GC pauses that also falsely idle-timeout healthy workers. The worker pool is torn down by the enclosing `finally`. The abort can be declined by setting `GITNEXUS_MEMORY=off`.","triggerScenarios":"Analyzing a repo whose combined AST/node set does not fit in the process heap; the guard fires at a chunk boundary (chunkIdx>0 region) when `shouldAbortForHeapPressure` returns true. Most common when `--max-old-space-size` (or a NODE_OPTIONS pin) caps the heap below what the machine actually has, or when the repo contains very large generated/vendored trees.","commonSituations":"A CI runner with a 2GB NODE_OPTIONS cap on a monorepo with a huge `node_modules`/`vendor` dir. A container whose cgroup memory limit is well below host RAM. A repo that bundles generated code (protobuf stubs, auto-generated grammar files) that bloats the node count.","solutions":["If the message says 'This machine has more memory available' — remove the `--max-old-space-size` pin from NODE_OPTIONS / node flags so the RAM-aware auto-sizer grants a larger heap.","If the message says 'at its memory ceiling' — exclude generated/vendored directories via a `.gitnexusignore` file to shrink the parseable set.","Run on a host (or container with a higher cgroup limit) with more RAM.","As a last resort, set `GITNEXUS_MEMORY=off` to suppress the guard (proceed at your own risk — expect long GC stalls or a hard OOM)."],"exampleFix":"// before — CI pins the heap below machine RAM\nNODE_OPTIONS='--max-old-space-size=2048' gitnexus analyze big-monorepo\n# → Analyze stopped before running out of memory: 1888MB of the 2048MB Node heap...\n\n// after — drop the pin; let gitnexus size itself\ngitnexus analyze big-monorepo","handlingStrategy":"validation","validationCode":"// Estimate heap need and warn before running analyze\nimport { projectParseHeapNeedBytes } from 'gitnexus/dist/core/ingestion/pipeline-phases/parse-impl.js';\nimport v8 from 'node:v8';\n\nconst heapLimit = v8.getHeapStatistics().heap_size_limit;\nconst projected = projectParseHeapNeedBytes(parseableFileCount);\nif (projected > heapLimit * 0.92) {\n  throw new Error(\n    `Projected parse heap (${Math.round(projected/1024/1024)}MB) exceeds 92% of the ` +\n    `heap limit (${Math.round(heapLimit/1024/1024)}MB). Raise --max-old-space-size ` +\n    `or add a .gitnexusignore.`,\n  );\n}","typeGuard":null,"tryCatchPattern":"try {\n  await analyze(repo, options);\n} catch (err) {\n  if (/Analyze stopped before running out of memory/.test(err.message)) {\n    // The message names the remedy (drop the pin, or .gitnexusignore, or more RAM).\n    // Act on it, then re-run. Do NOT retry unchanged.\n    console.error(err.message);\n    process.exit(3);\n  }\n  throw err;\n}","preventionTips":["Do not pin --max-old-space-size below the machine's RAM — let the RAM-aware auto-sizer work.","Keep a .gitnexusignore for generated/vendored trees on large monorepos.","Watch container cgroup memory limits — they cap the heap below host RAM."],"tags":["memory","parsing","heap","oom","worker-pool","autopilot"],"backgroundTag":null,"analyzedSha":"d540b00184d71a896261ee02670da9a92d59d8f7","analyzedAt":"2026-08-12T19:50:25.132Z","schemaVersion":2},"datasetVersion":"2026-08-12T23:17:12.415Z"}