rohitg00/agentmemory · error · Error

too_many_chunks_skipped

too_many_chunks_skipped

Error message

too_many_chunks_skipped: ${skipped}/${chunks.length} chunks failed to parse after retry

What it means

The summarize pipeline splits a session into chunks, summarizes each, and tolerates a fraction of unparseable chunks up to MAX_SKIP_RATIO. If more chunks than allowed failed to parse after a retry, produceSummaryXml throws 'too_many_chunks_skipped' rather than returning a summary built from too little data.

Source

Thrown at src/functions/summarize.ts:155

    await Promise.all(
      batch.map(async (chunk, j) => {
        const idx = batchStart + j;
        partialByIdx[idx] = await summarizeChunkWithRetry(
          provider,
          chunk,
          sessionId,
          project,
          idx,
          chunks.length,
        );
      }),
    );
  }

  const skipped = partialByIdx.filter((p) => p === null).length;
  const partials = partialByIdx.filter((p): p is SessionSummary => p !== null);

  if (skipped > Math.floor(chunks.length * MAX_SKIP_RATIO)) {
    throw new Error(
      `too_many_chunks_skipped: ${skipped}/${chunks.length} chunks failed to parse after retry`,
    );
  }
  if (skipped > 0) {
    logger.warn("Summarize chunks partially skipped", {
      sessionId,
      skipped,
      total: chunks.length,
    });
  }

  const reduceInput = partials.map((p) => {
    const originalIdx = partialByIdx.indexOf(p);
    return {
      title: p.title,
      narrative: p.narrative,
      keyDecisions: p.keyDecisions,

View on GitHub (pinned to e04ba88819)

Solutions

Retry the summarize call — transient provider failures may clear.
Reduce chunk size / batch fewer observations per run so fewer chunks fail.
Check provider health, quota, and API key; fix rate limiting (backoff, higher plan).
Inspect logs to see which chunks failed and why (parse vs empty response).

Example fix

// before
await sdk.trigger({ function_id: "mem::summarize", payload: { sessionId: bigSessionId } });
// after
try {
  await sdk.trigger({ function_id: "mem::summarize", payload: { sessionId: bigSessionId } });
} catch (e) {
  if (String(e).includes("too_many_chunks_skipped")) await retryWithBackoff(() => summarize(sessionId));
}

Defensive patterns

Strategy: retry

Validate before calling

// estimate chunk count and provider health before summarizing large sessions
const estChunks = Math.ceil(observations.length / CHUNK_SIZE);
if (estChunks > 20) summarizeInBatches(observations);

Try / catch

try {
  await sdk.trigger({ function_id: "mem::summarize", payload: { sessionId } });
} catch (e) {
  if (String(e).includes("too_many_chunks_skipped")) await retryWithBackoff(() => summarize(sessionId), 3);
  else throw e;
}

Prevention

Summarize large sessions in smaller batches to reduce per-run failure blast radius.
Monitor provider rate limits and add backoff between summarize runs.
Fix provider auth/quota issues before bulk summarization.
Alert on this error since the session summary is skipped entirely.

When it happens

Trigger: registerSummarizeFunction → produced → produceSummaryXml: partialByIdx contains nulls for more than floor(chunks.length * MAX_SKIP_RATIO) entries after retry, i.e. the provider failed to produce parseable summary XML for too many chunks.

Common situations: Sessions with many observations hitting provider rate limits mid-run; flaky provider returning empty/garbage output repeatedly; token limits truncating responses; misconfigured provider API key causing repeated failures.

Understand the failure class

Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

AI-assisted analysis of rohitg00/agentmemory@e04ba88819 (2026-08-30). Data as JSON: /api/errors/af8e54c565545934. Report an issue: GitHub.