{"record":{"id":"7ccb154c2daa404e","repo":"abhigyanpatwari/GitNexus","slug":"copy-failed-for-table-retrymsg-slice-0-200","errorCode":null,"errorMessage":"COPY failed for ${table}: ${retryMsg.slice(0, 200)}${remedy ? ` ${remedy}` : ''}","messagePattern":"COPY failed for (.+?): (.+?)(.+?)` : ''\\}","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"gitnexus/src/core/lbug/lbug-adapter.ts","lineNumber":1053,"sourceCode":"  log: (message: string) => void,\n  totalSteps: number,\n): Promise<void> => {\n  let stepsDone = 0;\n  for (const [table, { csvPath, rows }] of nodeFileEntries) {\n    stepsDone++;\n    log(`Loading nodes ${stepsDone}/${totalSteps}: ${table} (${rows.toLocaleString()} rows)`);\n\n    if (!(await stagingCsvExists(csvPath))) throw missingStagingCsvError(table, csvPath, rows);\n\n    const copyQuery = getCopyQuery(table, normalizeCopyPath(csvPath));\n    await copyCsvWithRetry(targetConn, copyQuery, (retryErr) => {\n      const retryMsg = retryErr instanceof Error ? retryErr.message : String(retryErr);\n      // Pool exhaustion gets a remedy (#2631): the raw binder text gives the\n      // operator nothing to act on, and on non-4K-page hosts (Ascend aarch64,\n      // Apple Silicon) the pool bills up to pageSize/4KiB x faster than the\n      // sizing was calibrated for — name the knob and the mechanism.\n      const remedy = bufferPoolExhaustionRemedy(retryMsg);\n      throw new Error(\n        `COPY failed for ${table}: ${retryMsg.slice(0, 200)}${remedy ? ` ${remedy}` : ''}`,\n      );\n    });\n  }\n};\n\n/**\n * Persist a KnowledgeGraph: stream CSVs, then bulk-COPY nodes (overlapped with\n * relationship emit — see the body) and relationships.\n *\n * NOT TRANSACTIONAL (#2226). Each `COPY` commits independently and there is no\n * surrounding transaction, so a failure partway through — a node `COPY` that\n * throws at the FK barrier, a relationship `COPY` failure, or a `pdgEmitManifest`\n * collision raised after node rows have already committed in the overlap path —\n * leaves a partially-loaded DB. The caller surfaces the error; recovery is a\n * `--force` re-analyze (a full rebuild), not a partial retry. Callers must not\n * assume the DB is either fully loaded or untouched after a rejection.\n */","sourceCodeStart":1035,"sourceCodeEnd":1071,"githubUrl":"https://github.com/abhigyanpatwari/GitNexus/blob/0d1aed942f0e8b5d3bac27519fff441aceea722d/gitnexus/src/core/lbug/lbug-adapter.ts#L1035-L1071","documentation":"lbug-adapter.ts line 1066 (bulk load): after streaming a table's staging CSV, it is COPY'd into LadybugDB; when copyCsvWithRetry's retries are exhausted the callback throws `COPY failed for <table>` with the engine message (first 200 chars). A distinct class — buffer pool exhaustion (matched by 'buffer pool is full' / 'unable to allocate memory', #2631) — gets an appended remedy naming the effective pool and GITNEXUS_LBUG_BUFFER_POOL_SIZE, plus a note that hosts with non-4KiB OS pages (aarch64/Ascend 64KiB kernels, Apple Silicon) bill pool memory in larger granules — up to 16x faster budget use than x86.","triggerScenarios":"COPYing a multi-million-row node/relationship table when the LadybugDB buffer pool is exhausted: pool left small via GITNEXUS_LBUG_BUFFER_POOL_SIZE, host RAM constrained by a container limit, or a 64KiB-page kernel amplifying granule billing; also unreadable/moved staging CSV or schema-type mismatch in rows.","commonSituations":"Large-repo analyze on ARM/Apple Silicon/aarch64 servers; CI containers with tight memory; operators who previously lowered the buffer pool env var; concurrent memory-heavy processes during load.","solutions":["Raise the pool: set GITNEXUS_LBUG_BUFFER_POOL_SIZE=<bytes> (e.g. 4294967296 for 4 GiB), or to 0 to restore LadybugDB's native 80%-of-RAM default, then re-run analyze","Free memory: stop other heavy processes on the host or raise the container memory limit so the pool can actually allocate","Shrink the graph with .gitnexusignore excludes for vendored/generated dirs if the host cannot afford a bigger pool","If the embedded error is not pool exhaustion, check the staging CSV exists and rows match the table schema, then re-run"],"exampleFix":"# before\n$ gitnexus analyze .   # COPY failed: buffer pool is full\n# after\n$ GITNEXUS_LBUG_BUFFER_POOL_SIZE=4294967296 gitnexus analyze .","handlingStrategy":"retry","validationCode":"import os from 'node:os';\n\nconst pageSize = os.pageSize ?? 4096; // aarch64/Apple Silicon hosts often 65536\nconst poolBytes = Number(process.env.GITNEXUS_LBUG_BUFFER_POOL_SIZE ?? 0);\nif (pageSize > 4096 && poolBytes > 0 && poolBytes < 4 * 1024 ** 3) {\n  console.warn(`Non-4K page host (${pageSize / 1024} KiB): raise GITNEXUS_LBUG_BUFFER_POOL_SIZE (or set 0 for 80%-of-RAM) before big COPYs`);\n}","typeGuard":null,"tryCatchPattern":"try {\n  await persistGraph(sinkResults);\n} catch (err) {\n  if (err instanceof Error && err.message.startsWith('COPY failed for ')) {\n    // if the message names the buffer pool: set GITNEXUS_LBUG_BUFFER_POOL_SIZE=<bytes> (or 0) and re-run analyze;\n    // otherwise inspect the embedded engine error (staging CSV / schema) before retrying\n  }\n  throw err;\n}","preventionTips":["On 64KiB-page hosts (Ascend/aarch64, Apple Silicon), size the pool up-front or set GITNEXUS_LBUG_BUFFER_POOL_SIZE=0 for the native 80%-of-RAM default","Run big analyzes with enough free RAM (or container limit) for the pool to allocate what it bills","Exclude heavy vendored dirs via .gitnexusignore to keep node/relationship COPY sizes within budget"],"tags":["ladybugdb","copy","bulk-load","memory","buffer-pool","arm64"],"backgroundTag":"buffer-pool-exhaustion","analyzedSha":"0d1aed942f0e8b5d3bac27519fff441aceea722d","analyzedAt":"2026-08-20T23:29:22.980Z","contentChangedAt":"2026-08-20T23:29:22.980Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}