can1357/oh-my-pi · error · ArchiveError

Remote archive is too large to buffer without range support

Error message

Remote archive is too large to buffer without range support (> ${cap} bytes)

What it means

httpByteSource() probes a remote archive with a Range request; when the server ignores Range and returns 200, it must download the entire body into memory as a fallback. This throw fires after that full download when the buffered byte count exceeds the cap (maxFallbackBytes, default 256 MiB). It exists to prevent unbounded memory use when a server lacks range support. The declared Content-Length variant (line 97) is checked first; this one catches servers that lie about or omit Content-Length.

Source

Thrown at packages/utils/src/ar/source.ts:102

 * one bounded full download. Wrap with {@link cachingByteSource} to coalesce
 * the many small header reads format parsers issue.
 */
export async function httpByteSource(url: string | URL, options: HttpByteSourceOptions = {}): Promise<ByteSource> {
	const doFetch = options.fetch ?? fetch;
	const headers = { ...options.headers, range: "bytes=0-0" };
	const probe = await doFetch(url, { headers });
	if (probe.status === 200) {
		// No range support: buffer the whole body once, bounded.
		const cap = options.maxFallbackBytes ?? HTTP_FALLBACK_CAP;
		const declared = Number(probe.headers.get("content-length") ?? 0);
		if (declared > cap) {
			throw new ArchiveError(
				`Remote archive is too large to buffer without range support (${declared} > ${cap} bytes)`,
			);
		}
		const bytes = new Uint8Array(await probe.arrayBuffer());
		if (bytes.byteLength > cap) {
			throw new ArchiveError(`Remote archive is too large to buffer without range support (> ${cap} bytes)`);
		}
		return memoryByteSource(bytes);
	}
	if (probe.status !== 206) {
		await probe.body?.cancel();
		throw new ArchiveError(`Remote archive request failed (HTTP ${probe.status})`);
	}
	await probe.body?.cancel();
	// `Content-Range: bytes 0-0/12345` carries the total size.
	const contentRange = probe.headers.get("content-range");
	const total = contentRange ? Number(/\/(\d+)$/.exec(contentRange)?.[1]) : Number.NaN;
	if (!Number.isSafeInteger(total) || total < 0) {
		throw new ArchiveError("Remote archive did not report a valid size in Content-Range");
	}
	return {
		size: total,
		async read(start, end) {
			assertValidRange(start, end);

View on GitHub (pinned to 9690622007)

Solutions

Host the archive on a server/CDN with HTTP Range support (S3, most CDNs, nginx with proper range handling) so responses are 206 instead of 200
Raise options.maxFallbackBytes if the memory cost is acceptable: httpByteSource(url, { maxFallbackBytes: 1024**3 })
Download the archive to disk first (fetch + Bun.write) and use fileByteSource(path) instead of httpByteSource
Verify with curl -r 0-0 -i <url> that the server returns 206 and a Content-Range header; fix proxy/gateway config if it returns 200

Example fix

// before: fails on a 400MB archive from a range-ignoring server
const src = await httpByteSource("https://mirror.example/big.tar");
// after: pre-download to disk, then read locally
const res = await fetch("https://mirror.example/big.tar");
await Bun.write("/tmp/big.tar", res);
const src = fileByteSource("/tmp/big.tar");

Defensive patterns

Strategy: validation

Validate before calling

const head = await fetch(url, { method: "HEAD" });
const size = Number(head.headers.get("content-length") ?? 0);
const acceptsRanges = head.headers.get("accept-ranges");
if (!acceptsRanges?.includes("bytes") && size > 256 * 1024 * 1024) {
	throw new Error(`archive too large (${size}B) for a server without range support; download to disk instead`);
}

Try / catch

try {
	const src = await httpByteSource(url);
} catch (err) {
	if (err instanceof ArchiveError && err.message.includes("too large to buffer")) {
		// fall back to full download + fileByteSource
	} else throw err;
}

Prevention

Prefer hosts with confirmed Accept-Ranges: bytes support for remote archives
HEAD-check content-length and accept-ranges before opening a remote archive
Set maxFallbackBytes explicitly to your app's memory budget rather than relying on the 256 MiB default
For archives over a few hundred MB, always download to disk first

When it happens

Trigger: Calling httpByteSource(url) against a server that responds 200 (not 206) to 'Range: bytes=0-0' and whose actual body exceeds maxFallbackBytes (default 268435456 bytes) after arrayBuffer() completes — including when Content-Length was absent or understated so the pre-download check at line 95 did not fire.

Common situations: Targeting archive mirrors/CDNs that strip or ignore Range headers (some S3-compatible proxies, misconfigured nginx without gzip/range handling); downloads routed through transforms that buffer responses; very large archives (hundreds of MB+) on legacy HTTP servers; developers assuming range support on a plain static host that streams full bodies.

Related errors

AI-assisted analysis of can1357/oh-my-pi@9690622007 (2026-08-31). Data as JSON: /api/errors/cc9c18e69e08b6c7. Report an issue: GitHub.