apache/hadoop · error · IOException

block overrun

Error message

block overrun

What it means

CBZip2InputStream.getAndMoveToFrontDecode() expands RUNA/RUNB symbols into a run length s and writes s+1 copies of the current byte into the block buffer ll8. limitLast is blockSize100k * 100000, the maximum block payload declared in the stream header ('BZh<1-9>'). If the accumulated output position lastShadow reaches limitLast, the decoded runs claim more data than the block can legally hold. A conforming encoder never produces such a block, so this IOException signals a corrupted (bit-flipped) bzip2 stream, not a configuration problem.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/CBZip2InputStream.java:929

                    "unexpected end of stream");
              }
            }
            bsLiveShadow--;
            zvec = (zvec << 1)
                | ((bsBuffShadow >> bsLiveShadow) & 1);
          }
          nextSym = perm_zt[(int) (zvec - base_zt[zn])];
        }

        final byte ch = seqToUnseq[yy[0]];
        unzftab[ch & 0xff] += s + 1;

        while (s-- >= 0) {
          ll8[++lastShadow] = ch;
        }

        if (lastShadow >= limitLast) {
          throw new IOException("block overrun");
        }
      } else {
        if (++lastShadow >= limitLast) {
          throw new IOException("block overrun");
        }

        final char tmp = yy[nextSym - 1];
        unzftab[seqToUnseq[tmp] & 0xff]++;
        ll8[lastShadow] = seqToUnseq[tmp];

        /*
        * This loop is hammered during decompression, hence avoid
        * native method call overhead of System.arraycopy for very
        * small ranges to copy.
        */
        if (nextSym <= 16) {
          for (int j = nextSym - 1; j > 0;) {
            yy[j] = yy[--j];

View on GitHub (pinned to 2add963021)

Solutions

1. Validate the file externally (bzip2 -t file.bz2) — a real 'block overrun' means the archive itself is corrupt and must be restored from a good copy or backup.
2. Re-run the transfer with end-to-end checksums (MD5/CRC32 comparison producer vs consumer) to catch silent corruption.
3. Do not try to 'fix' it by changing reader-side blockSize settings — the limit comes from the stream's own header, so only replacing the input helps.
4. Quarantine the corrupt file in ingestion pipelines and report the source path so upstream can regenerate it.

// Cheap CRC gate before decompression when checksums accompany the data
if (fileCrc != null && crc32.getValue() != fileCrc) {
  throw new IOException("CRC mismatch for " + path + "; refusing to decompress");
}

try {
  decompressFully(path, in);
} catch (IOException e) {
  if ("block overrun".equals(e.getMessage())) {
    quarantine(path); // corrupt archive, cannot be fixed reader-side
  }
  throw e;
}

When it happens

Trigger: Any CBZip2InputStream/BZip2Codec read where a decoded run ('while (s-- >= 0) ll8[++lastShadow] = ch;') pushes lastShadow to or past blockSize100k*100000: corrupted compressed bytes after valid magic/header, a header blockSize byte inconsistent with the payload, or random/garbage data fed as a bzip2 stream that happens to parse a header.

Common situations: Bit-rot on archival storage; silent corruption from faulty network/disk hardware; files mangled by text-mode FTP or double-decompression; concatenating a non-bzip2 payload after bzip2 magic; processing attacker-supplied or unchecked archives.

Related errors

• stream corrupted
• blockSize({}) < 1
• blockSize({}) > 9
• closed
• offs({}) < 0.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/9c32fb5d18b57325. Report an issue: GitHub.