apache/hadoop · error · IOException

BlockReader: error in packet header {}

Error message

BlockReader: error in packet header {}

What it means

BlockReaderRemote receives a block as a sequence of packets; after parsing each header it calls PacketHeader.sanityCheck(lastSeqNo), which validates length, offset and sequence-number invariants against the previously seen packet. A header failing these checks produces this IOException — the packet stream itself is malformed.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderRemote.java:197

    buf.put(writeSlice);
    curDataSlice.position(writeSlice.position());

    return nRead;
  }

  private void readNextPacket() throws IOException {
    //Read packet headers.
    packetReceiver.receiveNextPacket(in);

    PacketHeader curHeader = packetReceiver.getHeader();
    curDataSlice = packetReceiver.getDataSlice();
    assert curDataSlice.capacity() == curHeader.getDataLen();

    LOG.trace("DFSClient readNextPacket got header {}", curHeader);

    // Sanity check the lengths
    if (!curHeader.sanityCheck(lastSeqNo)) {
      throw new IOException("BlockReader: error in packet header " +
          curHeader);
    }

    if (curHeader.getDataLen() > 0) {
      int chunks = 1 + (curHeader.getDataLen() - 1) / bytesPerChecksum;
      int checksumsLen = chunks * checksumSize;

      assert packetReceiver.getChecksumSlice().capacity() == checksumsLen :
          "checksum slice capacity=" +
              packetReceiver.getChecksumSlice().capacity() +
              " checksumsLen=" + checksumsLen;

      lastSeqNo = curHeader.getSeqno();
      if (verifyChecksum && curDataSlice.remaining() > 0) {
        // N.B.: the checksum error offset reported here is actually
        // relative to the start of the block, not the start of the file.
        // This is slightly misleading, but preserves the behavior from
        // the older BlockReader.

View on GitHub (pinned to 2add963021)

Solutions

1. Retry the read — DFSInputStream re-fetches block locations and usually picks a different replica.
2. Run 'hdfs fsck /file -files -blocks -locations' to rule out real block corruption.
3. If reproducible on one host/path, inspect the network (disable NIC offload, swap cables) and the datanode's GC/OOM behavior and logs.
4. Verify client and datanode Hadoop versions are compatible during rolling upgrades.

for (int attempt = 1; attempt <= 3; attempt++) {
  try (FSDataInputStream in = dfs.open(path)) {
    in.seek(lastGoodOffset);
    return IOUtils.readFullyAsBytes(in, remaining);
  } catch (IOException e) {
    if (attempt == 3 || !isPacketHeaderError(e)) throw e;
    lastGoodOffset = recomputeOffset(); // re-fetch locations on retry
  }
}

When it happens

Trigger: Any HDFS block read (open/read, MR/Spark splits, DistCp) where bytes on the data-transfer connection do not parse as a sane packet header: a corrupted TCP stream, a datanode with memory/GC problems emitting garbage, or protocol skew between client and datanode during a rolling upgrade.

Common situations: Faulty NICs or TCP-offload engines corrupting payloads; mixed client/datanode versions mid rolling-upgrade; long-lived connections traversing flaky interconnects, VPNs or LBs; datanodes under OOM/GC thrash.

Related errors

• Invalid payload length ${payloadLen}
• Invalid packet: data length in packet header exceeds data le
• Expected empty end-of-read packet! Header: {}
• Invalid header length ${headerLen}
• Incorrect value for packet payload size: ${payloadLen}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/70b1c533d621052f. Report an issue: GitHub.