{"record":{"id":"d66965a0e55675b9","repo":"can1357/oh-my-pi","slug":"remote-archive-is-too-large-to-buffer-without-rang","errorCode":null,"errorMessage":"Remote archive is too large to buffer without range support (${declared} > ${cap} bytes)","messagePattern":"Remote archive is too large to buffer without range support \\((.+?) > (.+?) bytes\\)","errorType":"exception","errorClass":"ArchiveError","httpStatus":null,"severity":"error","filePath":"packages/utils/src/ar/source.ts","lineNumber":96,"sourceCode":"const HTTP_FALLBACK_CAP = 256 * 1024 * 1024;\n\n/**\n * A {@link ByteSource} over HTTP(S) range requests, so remote archives can be\n * indexed and read member-by-member without downloading the whole file.\n * Probes with `Range: bytes=0-0`; servers without range support fall back to\n * one bounded full download. Wrap with {@link cachingByteSource} to coalesce\n * the many small header reads format parsers issue.\n */\nexport async function httpByteSource(url: string | URL, options: HttpByteSourceOptions = {}): Promise<ByteSource> {\n\tconst doFetch = options.fetch ?? fetch;\n\tconst headers = { ...options.headers, range: \"bytes=0-0\" };\n\tconst probe = await doFetch(url, { headers });\n\tif (probe.status === 200) {\n\t\t// No range support: buffer the whole body once, bounded.\n\t\tconst cap = options.maxFallbackBytes ?? HTTP_FALLBACK_CAP;\n\t\tconst declared = Number(probe.headers.get(\"content-length\") ?? 0);\n\t\tif (declared > cap) {\n\t\t\tthrow new ArchiveError(\n\t\t\t\t`Remote archive is too large to buffer without range support (${declared} > ${cap} bytes)`,\n\t\t\t);\n\t\t}\n\t\tconst bytes = new Uint8Array(await probe.arrayBuffer());\n\t\tif (bytes.byteLength > cap) {\n\t\t\tthrow new ArchiveError(`Remote archive is too large to buffer without range support (> ${cap} bytes)`);\n\t\t}\n\t\treturn memoryByteSource(bytes);\n\t}\n\tif (probe.status !== 206) {\n\t\tawait probe.body?.cancel();\n\t\tthrow new ArchiveError(`Remote archive request failed (HTTP ${probe.status})`);\n\t}\n\tawait probe.body?.cancel();\n\t// `Content-Range: bytes 0-0/12345` carries the total size.\n\tconst contentRange = probe.headers.get(\"content-range\");\n\tconst total = contentRange ? Number(/\\/(\\d+)$/.exec(contentRange)?.[1]) : Number.NaN;\n\tif (!Number.isSafeInteger(total) || total < 0) {","sourceCodeStart":78,"sourceCodeEnd":114,"githubUrl":"https://github.com/can1357/oh-my-pi/blob/969062200754ea02cfac922e5ebb8c608c079e15/packages/utils/src/ar/source.ts#L78-L114","documentation":"When a remote server does not support HTTP range requests (probe returns 200, not 206), `httpByteSource` must buffer the whole body in memory. To keep that bounded, it first checks the Content-Length header against `options.maxFallbackBytes` (default `HTTP_FALLBACK_CAP`) and refuses archives whose declared size exceeds the cap, including the size in the message.","triggerScenarios":"Fetching a remote archive via the http ByteSource whose server ignores Range headers (200 response) and whose `Content-Length` exceeds the fallback cap — either the archive is genuinely large, or `options.maxFallbackBytes` was set below the archive's size.","commonSituations":"Pointing the library at large archives on static hosts without range support (some CDNs/CGI endpoints); configuring a tight `maxFallbackBytes`; archives that grew since the cap was chosen.","solutions":["Serve the archive from a host that supports HTTP range requests (Accept-Ranges: bytes / 206 responses) so full buffering is avoided","Raise `options.maxFallbackBytes` above the archive's size if buffering is acceptable","Download the archive to disk first and open it with `fileByteSource` instead of streaming from HTTP"],"exampleFix":"// before: small fallback cap\nconst src = await source(url, { maxFallbackBytes: 1024 * 1024 });\n// after: raise the cap or download locally first\nconst src = await source(url, { maxFallbackBytes: 256 * 1024 * 1024 });\n// or: const local = await Bun.write(\"tmp.tar\", await (await fetch(url)).arrayBuffer());\n//     const src = fileByteSource(\"tmp.tar\");","handlingStrategy":"fallback","validationCode":"const probe = await fetch(url, { method: \"HEAD\" });\nconst acceptsRanges = probe.headers.get(\"accept-ranges\");\nconst declared = Number(probe.headers.get(\"content-length\") ?? 0);\nif (!acceptsRanges && declared > cap) throw new Error(`too large to buffer: ${declared} > ${cap}`);","typeGuard":null,"tryCatchPattern":"try {\n\tconst src = await source(url, { maxFallbackBytes: cap });\n} catch (err) {\n\tif (err instanceof ArchiveError && err.message.includes(\"too large to buffer without range support\")) {\n\t\t// fallback: download to disk and open as a file source\n\t\tconst local = await Bun.write(Bun.tmpdir() + \"/archive.dl\", await (await fetch(url)).arrayBuffer());\n\t\treturn fileByteSource(local);\n\t}\n\tthrow err;\n}","preventionTips":["Serve archives from hosts that support range requests (Accept-Ranges: bytes)","Set maxFallbackBytes deliberately based on expected archive sizes","For large archives, download to disk first and use fileByteSource","HEAD-probe content-length before opening remote archives"],"tags":["network","http","limits","memory"],"backgroundTag":"remote-archive-too-large","analyzedSha":"969062200754ea02cfac922e5ebb8c608c079e15","analyzedAt":"2026-08-31T10:29:35.737Z","schemaVersion":2},"datasetVersion":"2026-08-31T14:17:45.589Z"}