can1357/oh-my-pi · error · ArchiveError

Remote archive is too large to buffer without range support

Error message

Remote archive is too large to buffer without range support (${declared} > ${cap} bytes)

What it means

When a remote server does not support HTTP range requests (probe returns 200, not 206), `httpByteSource` must buffer the whole body in memory. To keep that bounded, it first checks the Content-Length header against `options.maxFallbackBytes` (default `HTTP_FALLBACK_CAP`) and refuses archives whose declared size exceeds the cap, including the size in the message.

Source

Thrown at packages/utils/src/ar/source.ts:96

const HTTP_FALLBACK_CAP = 256 * 1024 * 1024;

/**
 * A {@link ByteSource} over HTTP(S) range requests, so remote archives can be
 * indexed and read member-by-member without downloading the whole file.
 * Probes with `Range: bytes=0-0`; servers without range support fall back to
 * one bounded full download. Wrap with {@link cachingByteSource} to coalesce
 * the many small header reads format parsers issue.
 */
export async function httpByteSource(url: string | URL, options: HttpByteSourceOptions = {}): Promise<ByteSource> {
	const doFetch = options.fetch ?? fetch;
	const headers = { ...options.headers, range: "bytes=0-0" };
	const probe = await doFetch(url, { headers });
	if (probe.status === 200) {
		// No range support: buffer the whole body once, bounded.
		const cap = options.maxFallbackBytes ?? HTTP_FALLBACK_CAP;
		const declared = Number(probe.headers.get("content-length") ?? 0);
		if (declared > cap) {
			throw new ArchiveError(
				`Remote archive is too large to buffer without range support (${declared} > ${cap} bytes)`,
			);
		}
		const bytes = new Uint8Array(await probe.arrayBuffer());
		if (bytes.byteLength > cap) {
			throw new ArchiveError(`Remote archive is too large to buffer without range support (> ${cap} bytes)`);
		}
		return memoryByteSource(bytes);
	}
	if (probe.status !== 206) {
		await probe.body?.cancel();
		throw new ArchiveError(`Remote archive request failed (HTTP ${probe.status})`);
	}
	await probe.body?.cancel();
	// `Content-Range: bytes 0-0/12345` carries the total size.
	const contentRange = probe.headers.get("content-range");
	const total = contentRange ? Number(/\/(\d+)$/.exec(contentRange)?.[1]) : Number.NaN;
	if (!Number.isSafeInteger(total) || total < 0) {

View on GitHub (pinned to 9690622007)

Solutions

1. Serve the archive from a host that supports HTTP range requests (Accept-Ranges: bytes / 206 responses) so full buffering is avoided
2. Raise `options.maxFallbackBytes` above the archive's size if buffering is acceptable
3. Download the archive to disk first and open it with `fileByteSource` instead of streaming from HTTP

Example fix

// before: small fallback cap
const src = await source(url, { maxFallbackBytes: 1024 * 1024 });
// after: raise the cap or download locally first
const src = await source(url, { maxFallbackBytes: 256 * 1024 * 1024 });
// or: const local = await Bun.write("tmp.tar", await (await fetch(url)).arrayBuffer());
//     const src = fileByteSource("tmp.tar");

const probe = await fetch(url, { method: "HEAD" });
const acceptsRanges = probe.headers.get("accept-ranges");
const declared = Number(probe.headers.get("content-length") ?? 0);
if (!acceptsRanges && declared > cap) throw new Error(`too large to buffer: ${declared} > ${cap}`);

try {
	const src = await source(url, { maxFallbackBytes: cap });
} catch (err) {
	if (err instanceof ArchiveError && err.message.includes("too large to buffer without range support")) {
		// fallback: download to disk and open as a file source
		const local = await Bun.write(Bun.tmpdir() + "/archive.dl", await (await fetch(url)).arrayBuffer());
		return fileByteSource(local);
	}
	throw err;
}

When it happens

Trigger: Fetching a remote archive via the http ByteSource whose server ignores Range headers (200 response) and whose `Content-Length` exceeds the fallback cap — either the archive is genuinely large, or `options.maxFallbackBytes` was set below the archive's size.

Common situations: Pointing the library at large archives on static hosts without range support (some CDNs/CGI endpoints); configuring a tight `maxFallbackBytes`; archives that grew since the cap was chosen.

Related errors

• V2 remote compaction failed (${response.status} ${response.s
• sso-role
• HTTP request failed. status=${response.status}; url=${url};
• ${response.status} ${response.statusText}: ${text}
• HTTP ${response.status} from ${baseUrl}/api/tags

AI-assisted analysis of can1357/oh-my-pi@9690622007 (2026-08-31). Data as JSON: /api/errors/d66965a0e55675b9. Report an issue: GitHub.