{"record":{"id":"b632aa18e91f4ce5","repo":"can1357/oh-my-pi","slug":"archive-is-too-large-to-read-in-memory-formatby","errorCode":null,"errorMessage":"Archive is too large to read in memory (${formatBytes(size)} > ${formatBytes(limits.maxInMemorySize)} limit)","messagePattern":"Archive is too large to read in memory \\((.+?) > (.+?) limit\\)","errorType":"exception","errorClass":"ArchiveError","httpStatus":null,"severity":"error","filePath":"packages/utils/src/ar/limits.ts","lineNumber":41,"sourceCode":"\tmaxLinkDepth: number;\n}\n\nexport const DEFAULT_ARCHIVE_LIMITS: ArchiveLimits = {\n\tmaxEntries: 1_000_000,\n\tmaxInMemorySize: 256 * 1024 * 1024,\n\tmaxIndexSize: 64 * 1024 * 1024,\n\tmaxMemberSize: 64 * 1024 * 1024,\n\tmaxPathBytes: 4096,\n\tmaxLinkDepth: 40,\n};\n\n/** Reject an archive that would be fully materialized beyond `maxInMemorySize`. */\nexport function assertInMemorySize(size: number, limits: ArchiveLimits): void {\n\tif (!Number.isSafeInteger(size) || size < 0) {\n\t\tthrow new ArchiveError(\"Archive is too large to read safely\");\n\t}\n\tif (size > limits.maxInMemorySize) {\n\t\tthrow new ArchiveError(\n\t\t\t`Archive is too large to read in memory (${formatBytes(size)} > ${formatBytes(limits.maxInMemorySize)} limit)`,\n\t\t);\n\t}\n}\n\n/** Reject archive metadata (index/header) beyond `maxIndexSize`. */\nexport function assertIndexSize(size: number, limits: ArchiveLimits, what: string): void {\n\tif (!Number.isSafeInteger(size) || size < 0) {\n\t\tthrow new ArchiveError(`Invalid archive: ${what} has an invalid size`);\n\t}\n\tif (size > limits.maxIndexSize) {\n\t\tthrow new ArchiveError(\n\t\t\t`Archive ${what} is too large (${formatBytes(size)} > ${formatBytes(limits.maxIndexSize)} limit)`,\n\t\t);\n\t}\n}\n\n/**","sourceCodeStart":23,"sourceCodeEnd":59,"githubUrl":"https://github.com/can1357/oh-my-pi/blob/969062200754ea02cfac922e5ebb8c608c079e15/packages/utils/src/ar/limits.ts#L23-L59","documentation":"assertInMemorySize rejects archives that would be fully loaded into memory beyond the configured limit (default maxInMemorySize is 256 MiB in DEFAULT_ARCHIVE_LIMITS). The message reports both the measured size and the limit via formatBytes so you can see by how much the archive overshoots.","triggerScenarios":"Calling a materializing reader (readArj, readCabArchive, readCpio, readDeb -> decompressDebTar, etc.) with default limits on an archive (or a decompressed deb control/data tar member) larger than limits.maxInMemorySize — e.g. a 500 MiB cpio or a deb whose data.tar decompresses past 256 MiB.","commonSituations":"Reading large system images/backup archives (big cpio/cab), .deb packages with huge data payloads, CI machines where callers assumed smaller artifacts, or decompression bombs whose inflated size crosses the limit.","solutions":["Pass custom limits raising maxInMemorySize for this call: { limits: { ...DEFAULT_ARCHIVE_LIMITS, maxInMemorySize: 2 * 1024 * 1024 * 1024 } } (mind actual available RAM)","Stream/extract the archive with a native tool (cpio, cabextract, dpkg-deb) instead of materializing it in JS","Split the archive or read only needed members via targeted tooling","Check for decompression bombs if the size seemed unexpectedly large; reject untrusted inputs"],"exampleFix":"// before\nconst entries = await readCpio(bigBuffer); // >256MiB -> ArchiveError\n// after\nimport { DEFAULT_ARCHIVE_LIMITS } from '.../ar/limits';\nconst entries = await readCpio(bigBuffer, {\n  limits: { ...DEFAULT_ARCHIVE_LIMITS, maxInMemorySize: 1024 * 1024 * 1024 },\n});","handlingStrategy":"try-catch","validationCode":"import { stat } from 'node:fs/promises';\nimport { DEFAULT_ARCHIVE_LIMITS } from '.../ar/limits';\nconst { size } = await stat(path);\nif (size > DEFAULT_ARCHIVE_LIMITS.maxInMemorySize) {\n  throw new Error(`archive ${size} bytes exceeds in-memory limit; stream instead`);\n}","typeGuard":"null","tryCatchPattern":"try {\n  entries = await readCpio(source);\n} catch (err) {\n  if (err instanceof ArchiveError && err.message.startsWith('Archive is too large to read in memory')) {\n    const m = err.message.match(/\\((.*?) > (.*?) limit\\)/); // parse size vs limit\n    // raise limits explicitly or switch to native streaming tools\n  } else throw err;\n}","preventionTips":["Check file size against limits.maxInMemorySize before reading","Pass explicit limits tuned to your deployment's RAM for large artifacts","Use streaming/native tools (cpio, cabextract, dpkg-deb) for anything near 256 MiB","Watch for decompression bombs: compressed size can be tiny while inflated size crosses the limit"],"tags":["archive","limits","memory","size-limit"],"backgroundTag":"archive-size-limit","analyzedSha":"969062200754ea02cfac922e5ebb8c608c079e15","analyzedAt":"2026-08-31T10:29:35.737Z","schemaVersion":2},"datasetVersion":"2026-08-31T14:17:45.589Z"}