{"record":{"id":"04e876d015d3d88a","repo":"can1357/oh-my-pi","slug":"response-exceeds-maxbytes-bytes","errorCode":null,"errorMessage":"response exceeds ${maxBytes} bytes","messagePattern":"response exceeds (.+?) bytes","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"packages/coding-agent/src/web/scrapers/utils.ts","lineNumber":51,"sourceCode":"\tif (!reader) return new Uint8Array(0);\n\n\tconst chunks: Buffer[] = [];\n\tlet totalBytes = 0;\n\n\ttry {\n\t\twhile (true) {\n\t\t\tif (signal?.aborted) {\n\t\t\t\tawait reader.cancel();\n\t\t\t\tthrow new ToolAbortError();\n\t\t\t}\n\t\t\tconst { done, value } = await reader.read();\n\t\t\tif (done) break;\n\t\t\tif (!value || value.byteLength === 0) continue;\n\n\t\t\ttotalBytes += value.byteLength;\n\t\t\tif (totalBytes > maxBytes) {\n\t\t\t\tawait reader.cancel();\n\t\t\t\tthrow new Error(`response exceeds ${maxBytes} bytes`);\n\t\t\t}\n\n\t\t\tchunks.push(Buffer.from(value));\n\t\t}\n\t} finally {\n\t\treader.releaseLock();\n\t}\n\n\treturn new Uint8Array(Buffer.concat(chunks, totalBytes));\n}\n\n/**\n * Fetch binary content from a URL\n */\nexport async function fetchBinary(url: string, timeout: number = 20, signal?: AbortSignal): Promise<BinaryFetchResult> {\n\tconst requestSignal = ptree.combineSignals(signal, timeout * 1000);\n\ttry {\n\t\tconst response = await fetch(url, {","sourceCodeStart":33,"sourceCodeEnd":69,"githubUrl":"https://github.com/can1357/oh-my-pi/blob/969062200754ea02cfac922e5ebb8c608c079e15/packages/coding-agent/src/web/scrapers/utils.ts#L33-L69","documentation":"readResponseWithLimit enforces a hard cap on how many bytes it will buffer from a response body. As soon as accumulated bytes exceed maxBytes it cancels the stream and throws 'response exceeds N bytes' rather than consuming unbounded memory. It's a deliberate safety guard against oversized downloads.","triggerScenarios":"Fetching a resource (web page, PDF, image) whose body is larger than the maxBytes passed to readResponseWithLimit (e.g. MAX_BYTES in fetchBinary).","commonSituations":"Scraping a huge HTML page or downloading a large PDF that exceeds the scraper's byte cap; endpoints that ignore Range/limits and return full bodies.","solutions":["Raise the maxBytes/MAX_BYTES limit if the content size is legitimate for your use case.","Pre-check Content-Length in the response headers and reject/short-circuit before reading.","Use a Range request to fetch only the first N bytes when you only need a preview.","Catch the error and surface a truncated/oversized result to the caller instead of failing the whole task."],"exampleFix":"// before\nconst buf = await readResponseWithLimit(response, 1024 * 1024);\n// after\nconst size = Number(response.headers.get(\"content-length\") ?? 0);\nif (size > 10 * 1024 * 1024) return { ok: false, error: \"too large\" };\nconst buf = await readResponseWithLimit(response, 10 * 1024 * 1024);","handlingStrategy":"validation","validationCode":"const len = Number(response.headers.get(\"content-length\") ?? NaN);\nif (Number.isFinite(len) && len > MAX_BYTES) {\n  return { ok: false, error: `content-length ${len} exceeds ${MAX_BYTES}` };\n}","typeGuard":"null","tryCatchPattern":"try { return await readResponseWithLimit(res, MAX_BYTES, signal); }\ncatch (e) {\n  if (e instanceof Error && /response exceeds \\d+ bytes/.test(e.message)) {\n    return { ok: false, error: \"response too large\", truncated: true };\n  }\n  throw e;\n}","preventionTips":["Pre-check Content-Length before reading the body.","Size MAX_BYTES to the largest payload your feature legitimately consumes.","For previews, use Range headers instead of raising the cap.","Report truncation/oversize to the caller rather than failing silently."],"tags":["limits","streaming","payload-size"],"backgroundTag":"response-body-too-large","analyzedSha":"969062200754ea02cfac922e5ebb8c608c079e15","analyzedAt":"2026-08-31T10:29:35.737Z","schemaVersion":2},"datasetVersion":"2026-08-31T14:17:45.589Z"}