{"record":{"id":"24537c6405f28e6f","repo":"santifer/career-ops","slug":"the-published-index-exceeds-max-index-bytes-byt-24537c","errorCode":null,"errorMessage":"the published index exceeds ${MAX_INDEX_BYTES} bytes: ${url}","messagePattern":"the published index exceeds (.+?) bytes: (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"plugins/h1b-sponsor/install-h1b-index.mjs","lineNumber":180,"sourceCode":"    // below turn it into the ordinary failure it is. once(out, 'drain') needs\n    // no extra wiring, it already rejects when 'error' fires mid-wait.\n    let writeError = null;\n    out.on('error', err => { writeError = err; });\n    const reader = res.body.getReader();\n    let total = 0;\n    try {\n      for (;;) {\n        if (writeError) {\n          await reader.cancel().catch(() => {});\n          throw writeError;\n        }\n        const { done, value } = await reader.read();\n        if (done) break;\n        if (!value) continue;\n        total += value.byteLength;\n        if (total > MAX_INDEX_BYTES) {\n          await reader.cancel().catch(() => {});\n          throw new Error(`the published index exceeds ${MAX_INDEX_BYTES} bytes: ${url}`);\n        }\n        hash.update(value);\n        // Respect backpressure: an 8 MiB body written without it queues the\n        // whole file in memory, which is the thing streaming was for.\n        if (!out.write(value)) await once(out, 'drain');\n      }\n    } finally {\n      await new Promise(resolve => out.end(resolve));\n    }\n    // end() above flushes the tail, so a failure during that flush, or one\n    // that raced the final read, is only visible here. The digest must not\n    // vouch for bytes that never landed on disk.\n    if (writeError) throw writeError;\n    return { digest: hash.digest('hex'), bytes: total };\n  });\n}\n\n/**","sourceCodeStart":162,"sourceCodeEnd":198,"githubUrl":"https://github.com/santifer/career-ops/blob/1696bec4d021768e7359f9aad6b329cba883da20/plugins/h1b-sponsor/install-h1b-index.mjs#L162-L198","documentation":"While streaming the index body to disk, downloadAsset() accumulates the byte count and aborts — cancelling the reader first — if the actual bytes exceed MAX_INDEX_BYTES. This is the streamed twin of the Content-Length check: it catches servers that omit or lie about the header, so the size cap cannot be bypassed by chunked encoding.","triggerScenarios":"Reading chunks in the body loop: total accumulates value.byteLength across chunks and crosses MAX_INDEX_BYTES before done, triggering reader.cancel() and this throw.","commonSituations":"A server sends no Content-Length (chunked transfer) and an oversized body; a hostile endpoint lies about Content-Length then streams more; a proxy pipelines a different, larger file; a misconfigured H1B_API_BASE serves an unbounded directory-style response.","solutions":["Confirm via curl -I / curl --raw what the endpoint actually streams at that URL — it should be the capped-size index.","Fix H1B_API_BASE so the URL resolves to the genuine release host rather than a proxy or wrong path.","If you publish the index, republish the correct artifact; do not raise MAX_INDEX_BYTES locally to work around a wrong download.","Retry later if a transiently misbehaving CDN is serving the wrong object."],"exampleFix":"// before\nH1B_API_BASE=https://proxy.internal/h1b   // chunked, unbounded response\n// after\nunset H1B_API_BASE  // default endpoint, correct content-length and size","handlingStrategy":"validation","validationCode":"// Pre-flight: confirm the endpoint streams a bounded, expected-size body\nconst res = await fetch(assetUrl);\nconst reader = res.body.getReader();\nlet total = 0;\nwhile (true) {\n  const { done, value } = await reader.read();\n  if (done) break;\n  total += value.byteLength;\n  if (total > 8 * 1024 * 1024) { reader.cancel(); throw new Error('endpoint streams an oversized body; wrong endpoint'); }\n}","typeGuard":null,"tryCatchPattern":"try {\n  await installH1BIndex();\n} catch (e) {\n  if (String(e.message).includes('exceeds') && e.message.includes('bytes')) {\n    console.error('Streamed body exceeded the cap (missing/lying content-length); check the endpoint or proxy.');\n  } else throw e;\n}","preventionTips":["Prefer endpoints that set accurate Content-Length; beware chunked proxies.","Configure H1B_API_BASE only against hosts you control or the official release host.","Never bypass the streamed cap — it is the defense against servers lying about size.","Probe unfamiliar endpoints with curl --raw to see what is actually streamed."],"tags":["network","validation","streams"],"backgroundTag":"unexpected-response-size","analyzedSha":"1696bec4d021768e7359f9aad6b329cba883da20","analyzedAt":"2026-09-01T19:19:23.111Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}