apache/hadoop · error · IOException

stream corrupted

Error message

stream corrupted

What it means

setupBlock() runs after a whole block is decoded: it builds the inverse Burrows-Wheeler transformation vector tt (size last+1) and then indexes it with origPtr, a 24-bit value read from the block header (this.origPtr = bsR(24) at the start of getAndMoveToFrontDecode). origPtr selects which rotation was the original input; it must lie in [0, tt.length). If it is negative or out of range, the stream's metadata contradicts its payload and 'stream corrupted' is thrown — an internally inconsistent (corrupt) bzip2 stream, as no valid encoder emits such a pointer.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/CBZip2InputStream.java:1060

    }

    final int[] cftab = this.data.cftab;
    final int[] tt = this.data.initTT(this.last + 1);
    final byte[] ll8 = this.data.ll8;
    cftab[0] = 0;
    System.arraycopy(this.data.unzftab, 0, cftab, 1, 256);

    for (int i = 1, c = cftab[0]; i <= 256; i++) {
      c += cftab[i];
      cftab[i] = c;
    }

    for (int i = 0, lastShadow = this.last; i <= lastShadow; i++) {
      tt[cftab[ll8[i] & 0xff]++] = i;
    }

    if ((this.origPtr < 0) || (this.origPtr >= tt.length)) {
      throw new IOException("stream corrupted");
    }

    this.su_tPos = tt[this.origPtr];
    this.su_count = 0;
    this.su_i2 = 0;
    this.su_ch2 = 256; /* not a char and not EOF */

    if (this.blockRandomised) {
      this.su_rNToGo = 0;
      this.su_rTPos = 0;
      setupRandPartA();
    } else {
      setupNoRandPartA();
    }
  }

  private void setupRandPartA() throws IOException {
    if (this.su_i2 <= this.last) {

View on GitHub (pinned to 2add963021)

Solutions

1. Verify the archive with bzip2 -t (or decompress it with the CLI) — if that fails too, restore the file from a good replica/backup.
2. Run checksum validation (CRC32/MD5 end-to-end) on transfer so corrupted files never reach decompression.
3. Investigate the storage path (failing disk, bad cable, checksum-less transfer mode like text-mode FTP) if multiple files show the same corruption.
4. Quarantine and report the exact path and byte offset; do not retry the same bytes.

catch (IOException e) {
  if ("stream corrupted".equals(e.getMessage())) {
    quarantineAndAlert(path); // metadata/payload inconsistency, restore from backup
  } else {
    throw e;
  }
}

When it happens

Trigger: Any CBZip2InputStream/BZip2Codec read where, for a fully-decoded block, origPtr < 0 or origPtr >= tt.length (tt sized last+1): flipped bits in the 24-bit origPtr field, a block whose decoded length (last) disagrees with its header, or hand-crafted/modified bzip2 data.

Common situations: Silent data corruption on disk/network; archives patched or truncated-then-recombined incorrectly; fuzzed input reaching a production path; files that pass the magic-byte check but are otherwise random bytes.

Related errors

• block overrun
• blockSize({}) < 1
• blockSize({}) > 9
• closed
• offs({}) < 0.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/08159040f7fd9ed5. Report an issue: GitHub.