{"record":{"id":"af8e54c565545934","repo":"rohitg00/agentmemory","slug":"too-many-chunks-skipped","errorCode":"too_many_chunks_skipped","errorMessage":"too_many_chunks_skipped: ${skipped}/${chunks.length} chunks failed to parse after retry","messagePattern":"too_many_chunks_skipped: (.+?)/(.+?) chunks failed to parse after retry","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"src/functions/summarize.ts","lineNumber":155,"sourceCode":"    await Promise.all(\n      batch.map(async (chunk, j) => {\n        const idx = batchStart + j;\n        partialByIdx[idx] = await summarizeChunkWithRetry(\n          provider,\n          chunk,\n          sessionId,\n          project,\n          idx,\n          chunks.length,\n        );\n      }),\n    );\n  }\n\n  const skipped = partialByIdx.filter((p) => p === null).length;\n  const partials = partialByIdx.filter((p): p is SessionSummary => p !== null);\n\n  if (skipped > Math.floor(chunks.length * MAX_SKIP_RATIO)) {\n    throw new Error(\n      `too_many_chunks_skipped: ${skipped}/${chunks.length} chunks failed to parse after retry`,\n    );\n  }\n  if (skipped > 0) {\n    logger.warn(\"Summarize chunks partially skipped\", {\n      sessionId,\n      skipped,\n      total: chunks.length,\n    });\n  }\n\n  const reduceInput = partials.map((p) => {\n    const originalIdx = partialByIdx.indexOf(p);\n    return {\n      title: p.title,\n      narrative: p.narrative,\n      keyDecisions: p.keyDecisions,","sourceCodeStart":137,"sourceCodeEnd":173,"githubUrl":"https://github.com/rohitg00/agentmemory/blob/e04ba88819c365c9acf9d6661ea802143e728bd6/src/functions/summarize.ts#L137-L173","documentation":"The summarize pipeline splits a session into chunks, summarizes each, and tolerates a fraction of unparseable chunks up to MAX_SKIP_RATIO. If more chunks than allowed failed to parse after a retry, produceSummaryXml throws 'too_many_chunks_skipped' rather than returning a summary built from too little data.","triggerScenarios":"registerSummarizeFunction → produced → produceSummaryXml: partialByIdx contains nulls for more than floor(chunks.length * MAX_SKIP_RATIO) entries after retry, i.e. the provider failed to produce parseable summary XML for too many chunks.","commonSituations":"Sessions with many observations hitting provider rate limits mid-run; flaky provider returning empty/garbage output repeatedly; token limits truncating responses; misconfigured provider API key causing repeated failures.","solutions":["Retry the summarize call — transient provider failures may clear.","Reduce chunk size / batch fewer observations per run so fewer chunks fail.","Check provider health, quota, and API key; fix rate limiting (backoff, higher plan).","Inspect logs to see which chunks failed and why (parse vs empty response)."],"exampleFix":"// before\nawait sdk.trigger({ function_id: \"mem::summarize\", payload: { sessionId: bigSessionId } });\n// after\ntry {\n  await sdk.trigger({ function_id: \"mem::summarize\", payload: { sessionId: bigSessionId } });\n} catch (e) {\n  if (String(e).includes(\"too_many_chunks_skipped\")) await retryWithBackoff(() => summarize(sessionId));\n}","handlingStrategy":"retry","validationCode":"// estimate chunk count and provider health before summarizing large sessions\nconst estChunks = Math.ceil(observations.length / CHUNK_SIZE);\nif (estChunks > 20) summarizeInBatches(observations);","typeGuard":null,"tryCatchPattern":"try {\n  await sdk.trigger({ function_id: \"mem::summarize\", payload: { sessionId } });\n} catch (e) {\n  if (String(e).includes(\"too_many_chunks_skipped\")) await retryWithBackoff(() => summarize(sessionId), 3);\n  else throw e;\n}","preventionTips":["Summarize large sessions in smaller batches to reduce per-run failure blast radius.","Monitor provider rate limits and add backoff between summarize runs.","Fix provider auth/quota issues before bulk summarization.","Alert on this error since the session summary is skipped entirely."],"tags":["llm","summarization","chunking","provider-failure"],"backgroundTag":"llm-output-parse-failed","analyzedSha":"e04ba88819c365c9acf9d6661ea802143e728bd6","analyzedAt":"2026-08-30T01:07:40.754Z","schemaVersion":2},"datasetVersion":"2026-08-30T03:17:51.788Z"}