What causes "the published index exceeds ${MAX_INDEX_BYTES} bytes: ${url}" in santifer/career-ops?

Confirm via curl -I / curl --raw what the endpoint actually streams at that URL — it should be the capped-size index.

santifer/career-ops · error · Error

the published index exceeds ${MAX_INDEX_BYTES} bytes: ${url}

Error message

the published index exceeds ${MAX_INDEX_BYTES} bytes: ${url}

What it means

While streaming the index body to disk, downloadAsset() accumulates the byte count and aborts — cancelling the reader first — if the actual bytes exceed MAX_INDEX_BYTES. This is the streamed twin of the Content-Length check: it catches servers that omit or lie about the header, so the size cap cannot be bypassed by chunked encoding.

Source

Thrown at plugins/h1b-sponsor/install-h1b-index.mjs:180

    // below turn it into the ordinary failure it is. once(out, 'drain') needs
    // no extra wiring, it already rejects when 'error' fires mid-wait.
    let writeError = null;
    out.on('error', err => { writeError = err; });
    const reader = res.body.getReader();
    let total = 0;
    try {
      for (;;) {
        if (writeError) {
          await reader.cancel().catch(() => {});
          throw writeError;
        }
        const { done, value } = await reader.read();
        if (done) break;
        if (!value) continue;
        total += value.byteLength;
        if (total > MAX_INDEX_BYTES) {
          await reader.cancel().catch(() => {});
          throw new Error(`the published index exceeds ${MAX_INDEX_BYTES} bytes: ${url}`);
        }
        hash.update(value);
        // Respect backpressure: an 8 MiB body written without it queues the
        // whole file in memory, which is the thing streaming was for.
        if (!out.write(value)) await once(out, 'drain');
      }
    } finally {
      await new Promise(resolve => out.end(resolve));
    }
    // end() above flushes the tail, so a failure during that flush, or one
    // that raced the final read, is only visible here. The digest must not
    // vouch for bytes that never landed on disk.
    if (writeError) throw writeError;
    return { digest: hash.digest('hex'), bytes: total };
  });
}

/**

View on GitHub (pinned to 1696bec4d0)

Solutions

Confirm via curl -I / curl --raw what the endpoint actually streams at that URL — it should be the capped-size index.
Fix H1B_API_BASE so the URL resolves to the genuine release host rather than a proxy or wrong path.
If you publish the index, republish the correct artifact; do not raise MAX_INDEX_BYTES locally to work around a wrong download.
Retry later if a transiently misbehaving CDN is serving the wrong object.

Example fix

// before
H1B_API_BASE=https://proxy.internal/h1b   // chunked, unbounded response
// after
unset H1B_API_BASE  // default endpoint, correct content-length and size

Defensive patterns

Strategy: validation

Validate before calling

// Pre-flight: confirm the endpoint streams a bounded, expected-size body
const res = await fetch(assetUrl);
const reader = res.body.getReader();
let total = 0;
while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  total += value.byteLength;
  if (total > 8 * 1024 * 1024) { reader.cancel(); throw new Error('endpoint streams an oversized body; wrong endpoint'); }
}

Try / catch

try {
  await installH1BIndex();
} catch (e) {
  if (String(e.message).includes('exceeds') && e.message.includes('bytes')) {
    console.error('Streamed body exceeded the cap (missing/lying content-length); check the endpoint or proxy.');
  } else throw e;
}

Prevention

Prefer endpoints that set accurate Content-Length; beware chunked proxies.
Configure H1B_API_BASE only against hosts you control or the official release host.
Never bypass the streamed cap — it is the defense against servers lying about size.
Probe unfamiliar endpoints with curl --raw to see what is actually streamed.

When it happens

Trigger: Reading chunks in the body loop: total accumulates value.byteLength across chunks and crosses MAX_INDEX_BYTES before done, triggering reader.cancel() and this throw.

Common situations: A server sends no Content-Length (chunked transfer) and an oversized body; a hostile endpoint lies about Content-Length then streams more; a proxy pipelines a different, larger file; a misconfigured H1B_API_BASE serves an unbounded directory-style response.

Related errors

AI-assisted analysis of santifer/career-ops@1696bec4d0 (2026-09-01). Data as JSON: /api/errors/24537c6405f28e6f. Report an issue: GitHub.