{"record":{"id":"cc9c18e69e08b6c7","repo":"can1357/oh-my-pi","slug":"remote-archive-is-too-large-to-buffer-without-rang-cc9c18","errorCode":null,"errorMessage":"Remote archive is too large to buffer without range support (> ${cap} bytes)","messagePattern":"Remote archive is too large to buffer without range support \\(> (.+?) bytes\\)","errorType":"exception","errorClass":"ArchiveError","httpStatus":null,"severity":"error","filePath":"packages/utils/src/ar/source.ts","lineNumber":102,"sourceCode":" * one bounded full download. Wrap with {@link cachingByteSource} to coalesce\n * the many small header reads format parsers issue.\n */\nexport async function httpByteSource(url: string | URL, options: HttpByteSourceOptions = {}): Promise<ByteSource> {\n\tconst doFetch = options.fetch ?? fetch;\n\tconst headers = { ...options.headers, range: \"bytes=0-0\" };\n\tconst probe = await doFetch(url, { headers });\n\tif (probe.status === 200) {\n\t\t// No range support: buffer the whole body once, bounded.\n\t\tconst cap = options.maxFallbackBytes ?? HTTP_FALLBACK_CAP;\n\t\tconst declared = Number(probe.headers.get(\"content-length\") ?? 0);\n\t\tif (declared > cap) {\n\t\t\tthrow new ArchiveError(\n\t\t\t\t`Remote archive is too large to buffer without range support (${declared} > ${cap} bytes)`,\n\t\t\t);\n\t\t}\n\t\tconst bytes = new Uint8Array(await probe.arrayBuffer());\n\t\tif (bytes.byteLength > cap) {\n\t\t\tthrow new ArchiveError(`Remote archive is too large to buffer without range support (> ${cap} bytes)`);\n\t\t}\n\t\treturn memoryByteSource(bytes);\n\t}\n\tif (probe.status !== 206) {\n\t\tawait probe.body?.cancel();\n\t\tthrow new ArchiveError(`Remote archive request failed (HTTP ${probe.status})`);\n\t}\n\tawait probe.body?.cancel();\n\t// `Content-Range: bytes 0-0/12345` carries the total size.\n\tconst contentRange = probe.headers.get(\"content-range\");\n\tconst total = contentRange ? Number(/\\/(\\d+)$/.exec(contentRange)?.[1]) : Number.NaN;\n\tif (!Number.isSafeInteger(total) || total < 0) {\n\t\tthrow new ArchiveError(\"Remote archive did not report a valid size in Content-Range\");\n\t}\n\treturn {\n\t\tsize: total,\n\t\tasync read(start, end) {\n\t\t\tassertValidRange(start, end);","sourceCodeStart":84,"sourceCodeEnd":120,"githubUrl":"https://github.com/can1357/oh-my-pi/blob/969062200754ea02cfac922e5ebb8c608c079e15/packages/utils/src/ar/source.ts#L84-L120","documentation":"httpByteSource() probes a remote archive with a Range request; when the server ignores Range and returns 200, it must download the entire body into memory as a fallback. This throw fires after that full download when the buffered byte count exceeds the cap (maxFallbackBytes, default 256 MiB). It exists to prevent unbounded memory use when a server lacks range support. The declared Content-Length variant (line 97) is checked first; this one catches servers that lie about or omit Content-Length.","triggerScenarios":"Calling httpByteSource(url) against a server that responds 200 (not 206) to 'Range: bytes=0-0' and whose actual body exceeds maxFallbackBytes (default 268435456 bytes) after arrayBuffer() completes — including when Content-Length was absent or understated so the pre-download check at line 95 did not fire.","commonSituations":"Targeting archive mirrors/CDNs that strip or ignore Range headers (some S3-compatible proxies, misconfigured nginx without gzip/range handling); downloads routed through transforms that buffer responses; very large archives (hundreds of MB+) on legacy HTTP servers; developers assuming range support on a plain static host that streams full bodies.","solutions":["Host the archive on a server/CDN with HTTP Range support (S3, most CDNs, nginx with proper range handling) so responses are 206 instead of 200","Raise options.maxFallbackBytes if the memory cost is acceptable: httpByteSource(url, { maxFallbackBytes: 1024**3 })","Download the archive to disk first (fetch + Bun.write) and use fileByteSource(path) instead of httpByteSource","Verify with curl -r 0-0 -i <url> that the server returns 206 and a Content-Range header; fix proxy/gateway config if it returns 200"],"exampleFix":"// before: fails on a 400MB archive from a range-ignoring server\nconst src = await httpByteSource(\"https://mirror.example/big.tar\");\n// after: pre-download to disk, then read locally\nconst res = await fetch(\"https://mirror.example/big.tar\");\nawait Bun.write(\"/tmp/big.tar\", res);\nconst src = fileByteSource(\"/tmp/big.tar\");","handlingStrategy":"validation","validationCode":"const head = await fetch(url, { method: \"HEAD\" });\nconst size = Number(head.headers.get(\"content-length\") ?? 0);\nconst acceptsRanges = head.headers.get(\"accept-ranges\");\nif (!acceptsRanges?.includes(\"bytes\") && size > 256 * 1024 * 1024) {\n\tthrow new Error(`archive too large (${size}B) for a server without range support; download to disk instead`);\n}","typeGuard":null,"tryCatchPattern":"try {\n\tconst src = await httpByteSource(url);\n} catch (err) {\n\tif (err instanceof ArchiveError && err.message.includes(\"too large to buffer\")) {\n\t\t// fall back to full download + fileByteSource\n\t} else throw err;\n}","preventionTips":["Prefer hosts with confirmed Accept-Ranges: bytes support for remote archives","HEAD-check content-length and accept-ranges before opening a remote archive","Set maxFallbackBytes explicitly to your app's memory budget rather than relying on the 256 MiB default","For archives over a few hundred MB, always download to disk first"],"tags":["http","memory-limit","remote-archive","range-requests"],"backgroundTag":"response-too-large-to-buffer","analyzedSha":"969062200754ea02cfac922e5ebb8c608c079e15","analyzedAt":"2026-08-31T10:29:35.737Z","schemaVersion":2},"datasetVersion":"2026-08-31T14:17:45.589Z"}