{"record":{"id":"9c32fb5d18b57325","repo":"apache/hadoop","slug":"block-overrun","errorCode":null,"errorMessage":"block overrun","messagePattern":"block overrun","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/CBZip2InputStream.java","lineNumber":929,"sourceCode":"                    \"unexpected end of stream\");\n              }\n            }\n            bsLiveShadow--;\n            zvec = (zvec << 1)\n                | ((bsBuffShadow >> bsLiveShadow) & 1);\n          }\n          nextSym = perm_zt[(int) (zvec - base_zt[zn])];\n        }\n\n        final byte ch = seqToUnseq[yy[0]];\n        unzftab[ch & 0xff] += s + 1;\n\n        while (s-- >= 0) {\n          ll8[++lastShadow] = ch;\n        }\n\n        if (lastShadow >= limitLast) {\n          throw new IOException(\"block overrun\");\n        }\n      } else {\n        if (++lastShadow >= limitLast) {\n          throw new IOException(\"block overrun\");\n        }\n\n        final char tmp = yy[nextSym - 1];\n        unzftab[seqToUnseq[tmp] & 0xff]++;\n        ll8[lastShadow] = seqToUnseq[tmp];\n\n        /*\n        * This loop is hammered during decompression, hence avoid\n        * native method call overhead of System.arraycopy for very\n        * small ranges to copy.\n        */\n        if (nextSym <= 16) {\n          for (int j = nextSym - 1; j > 0;) {\n            yy[j] = yy[--j];","sourceCodeStart":911,"sourceCodeEnd":947,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/CBZip2InputStream.java#L911-L947","documentation":"CBZip2InputStream.getAndMoveToFrontDecode() expands RUNA/RUNB symbols into a run length s and writes s+1 copies of the current byte into the block buffer ll8. limitLast is blockSize100k * 100000, the maximum block payload declared in the stream header ('BZh<1-9>'). If the accumulated output position lastShadow reaches limitLast, the decoded runs claim more data than the block can legally hold. A conforming encoder never produces such a block, so this IOException signals a corrupted (bit-flipped) bzip2 stream, not a configuration problem.","triggerScenarios":"Any CBZip2InputStream/BZip2Codec read where a decoded run ('while (s-- >= 0) ll8[++lastShadow] = ch;') pushes lastShadow to or past blockSize100k*100000: corrupted compressed bytes after valid magic/header, a header blockSize byte inconsistent with the payload, or random/garbage data fed as a bzip2 stream that happens to parse a header.","commonSituations":"Bit-rot on archival storage; silent corruption from faulty network/disk hardware; files mangled by text-mode FTP or double-decompression; concatenating a non-bzip2 payload after bzip2 magic; processing attacker-supplied or unchecked archives.","solutions":["Validate the file externally (bzip2 -t file.bz2) — a real 'block overrun' means the archive itself is corrupt and must be restored from a good copy or backup.","Re-run the transfer with end-to-end checksums (MD5/CRC32 comparison producer vs consumer) to catch silent corruption.","Do not try to 'fix' it by changing reader-side blockSize settings — the limit comes from the stream's own header, so only replacing the input helps.","Quarantine the corrupt file in ingestion pipelines and report the source path so upstream can regenerate it."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Cheap CRC gate before decompression when checksums accompany the data\nif (fileCrc != null && crc32.getValue() != fileCrc) {\n  throw new IOException(\"CRC mismatch for \" + path + \"; refusing to decompress\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  decompressFully(path, in);\n} catch (IOException e) {\n  if (\"block overrun\".equals(e.getMessage())) {\n    quarantine(path); // corrupt archive, cannot be fixed reader-side\n  }\n  throw e;\n}","preventionTips":["End-to-end checksums (CRC32/MD5) on every hop so corruption is caught before decompression.","Investigate hardware/transport if multiple files corrupt the same way.","Do not attempt reader-side 'fixes' (bigger buffers, different blockSize) — the stream itself is bad."],"tags":["bzip2","compression","data-corruption","block-overrun","hadoop"],"backgroundTag":"compressed-data-corruption","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}