abhigyanpatwari/GitNexus · critical

COPY failed for ${table}: ${retryMsg.slice(0, 200)}${remedy

Error message

COPY failed for ${table}: ${retryMsg.slice(0, 200)}${remedy ? ` ${remedy}` : ''}

What it means

lbug-adapter.ts line 1066 (bulk load): after streaming a table's staging CSV, it is COPY'd into LadybugDB; when copyCsvWithRetry's retries are exhausted the callback throws `COPY failed for <table>` with the engine message (first 200 chars). A distinct class — buffer pool exhaustion (matched by 'buffer pool is full' / 'unable to allocate memory', #2631) — gets an appended remedy naming the effective pool and GITNEXUS_LBUG_BUFFER_POOL_SIZE, plus a note that hosts with non-4KiB OS pages (aarch64/Ascend 64KiB kernels, Apple Silicon) bill pool memory in larger granules — up to 16x faster budget use than x86.

Source

Thrown at gitnexus/src/core/lbug/lbug-adapter.ts:1053

  log: (message: string) => void,
  totalSteps: number,
): Promise<void> => {
  let stepsDone = 0;
  for (const [table, { csvPath, rows }] of nodeFileEntries) {
    stepsDone++;
    log(`Loading nodes ${stepsDone}/${totalSteps}: ${table} (${rows.toLocaleString()} rows)`);

    if (!(await stagingCsvExists(csvPath))) throw missingStagingCsvError(table, csvPath, rows);

    const copyQuery = getCopyQuery(table, normalizeCopyPath(csvPath));
    await copyCsvWithRetry(targetConn, copyQuery, (retryErr) => {
      const retryMsg = retryErr instanceof Error ? retryErr.message : String(retryErr);
      // Pool exhaustion gets a remedy (#2631): the raw binder text gives the
      // operator nothing to act on, and on non-4K-page hosts (Ascend aarch64,
      // Apple Silicon) the pool bills up to pageSize/4KiB x faster than the
      // sizing was calibrated for — name the knob and the mechanism.
      const remedy = bufferPoolExhaustionRemedy(retryMsg);
      throw new Error(
        `COPY failed for ${table}: ${retryMsg.slice(0, 200)}${remedy ? ` ${remedy}` : ''}`,
      );
    });
  }
};

/**
 * Persist a KnowledgeGraph: stream CSVs, then bulk-COPY nodes (overlapped with
 * relationship emit — see the body) and relationships.
 *
 * NOT TRANSACTIONAL (#2226). Each `COPY` commits independently and there is no
 * surrounding transaction, so a failure partway through — a node `COPY` that
 * throws at the FK barrier, a relationship `COPY` failure, or a `pdgEmitManifest`
 * collision raised after node rows have already committed in the overlap path —
 * leaves a partially-loaded DB. The caller surfaces the error; recovery is a
 * `--force` re-analyze (a full rebuild), not a partial retry. Callers must not
 * assume the DB is either fully loaded or untouched after a rejection.
 */

View on GitHub (pinned to 0d1aed942f)

Solutions

1. Raise the pool: set GITNEXUS_LBUG_BUFFER_POOL_SIZE=<bytes> (e.g. 4294967296 for 4 GiB), or to 0 to restore LadybugDB's native 80%-of-RAM default, then re-run analyze
2. Free memory: stop other heavy processes on the host or raise the container memory limit so the pool can actually allocate
3. Shrink the graph with .gitnexusignore excludes for vendored/generated dirs if the host cannot afford a bigger pool
4. If the embedded error is not pool exhaustion, check the staging CSV exists and rows match the table schema, then re-run

Example fix

# before
$ gitnexus analyze .   # COPY failed: buffer pool is full
# after
$ GITNEXUS_LBUG_BUFFER_POOL_SIZE=4294967296 gitnexus analyze .

import os from 'node:os';

const pageSize = os.pageSize ?? 4096; // aarch64/Apple Silicon hosts often 65536
const poolBytes = Number(process.env.GITNEXUS_LBUG_BUFFER_POOL_SIZE ?? 0);
if (pageSize > 4096 && poolBytes > 0 && poolBytes < 4 * 1024 ** 3) {
  console.warn(`Non-4K page host (${pageSize / 1024} KiB): raise GITNEXUS_LBUG_BUFFER_POOL_SIZE (or set 0 for 80%-of-RAM) before big COPYs`);
}

try {
  await persistGraph(sinkResults);
} catch (err) {
  if (err instanceof Error && err.message.startsWith('COPY failed for ')) {
    // if the message names the buffer pool: set GITNEXUS_LBUG_BUFFER_POOL_SIZE=<bytes> (or 0) and re-run analyze;
    // otherwise inspect the embedded engine error (staging CSV / schema) before retrying
  }
  throw err;
}

When it happens

Trigger: COPYing a multi-million-row node/relationship table when the LadybugDB buffer pool is exhausted: pool left small via GITNEXUS_LBUG_BUFFER_POOL_SIZE, host RAM constrained by a container limit, or a 64KiB-page kernel amplifying granule billing; also unreadable/moved staging CSV or schema-type mismatch in rows.

Common situations: Large-repo analyze on ARM/Apple Silicon/aarch64 servers; CI containers with tight memory; operators who previously lowered the buffer pool env var; concurrent memory-heavy processes during load.

Related errors

• ${source}: branch name is too long (max ${BRANCH_MAX_LENGTH}
• ${source} must be a boolean (true/false).
• [lbug-load] node COPY also failed while relationship emit wa
• LadybugDB not initialized for repo "${repoId}". Call initLbu
• withRetry: maxAttempts must be >= 1, got ${opts.maxAttempts}

AI-assisted analysis of abhigyanpatwari/GitNexus@0d1aed942f (2026-08-20). Data as JSON: /api/errors/7ccb154c2daa404e. Report an issue: GitHub.