What causes "Invalid header length ${headerLen}" in apache/hadoop?

Retry the block read/write to land on a different replica/node.

apache/hadoop · critical · IOException

Invalid header length ${headerLen}

Error message

Invalid header length ${headerLen}

What it means

After reading the payload length, PacketReceiver reads a 2-byte header length; since it is read into a short-based field, any value with the high bit set decodes negative and is rejected. A negative header length cannot be valid, so the packet framing is corrupt.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PacketReceiver.java:157

    // DATA       the actual block data
    Preconditions.checkState(curHeader == null || !curHeader.isLastPacketInBlock());

    curPacketBuf.clear();
    curPacketBuf.limit(PacketHeader.PKT_LENGTHS_LEN);
    doReadFully(ch, in, curPacketBuf);
    curPacketBuf.flip();
    int payloadLen = curPacketBuf.getInt();

    if (payloadLen < Ints.BYTES) {
      // The "payload length" includes its own length. Therefore it
      // should never be less than 4 bytes
      throw new IOException("Invalid payload length " +
          payloadLen);
    }
    int dataPlusChecksumLen = payloadLen - Ints.BYTES;
    int headerLen = curPacketBuf.getShort();
    if (headerLen < 0) {
      throw new IOException("Invalid header length " + headerLen);
    }

    LOG.trace("readNextPacket: dataPlusChecksumLen={}, headerLen={}",
        dataPlusChecksumLen, headerLen);

    // Sanity check the buffer size so we don't allocate too much memory
    // and OOME.
    int totalLen = payloadLen + headerLen;
    if (totalLen < 0 || totalLen > MAX_PACKET_SIZE) {
      throw new IOException("Incorrect value for packet payload size: " +
                            payloadLen);
    }

    // Make sure we have space for the whole packet, and
    // read it.
    reallocPacketBuf(PacketHeader.PKT_LENGTHS_LEN +
        dataPlusChecksumLen + headerLen);
    curPacketBuf.clear();

View on GitHub (pinned to 2add963021)

Solutions

Retry the block read/write to land on a different replica/node.
Align Hadoop versions across the write pipeline (client + all DataNodes).
Check DataNode and client logs for the first protocol error that led to desync; fix that root cause.
If recurring on specific nodes, suspect hardware and run diagnostics on that host.

Defensive patterns

Strategy: try-catch

Try / catch

try {
  receiver.readNextPacket();
} catch (IOException e) {
  if (e.getMessage().contains("Invalid header length")) {
    // stream misframed/corrupt — close reader, retry on another replica
  } else { throw e; }
}

Prevention

Align versions across write pipelines; mixed packet-header layouts desync streams.
Investigate the first protocol error in the logs — later 'invalid length' errors are downstream symptoms.
Replace/schedule diagnostics for hosts that repeatedly produce framing errors.

When it happens

Trigger: Corrupt or desynchronized data-transfer stream where the 2 bytes after the payload length are garbage (high bit set); same root causes as invalid payload length — version mismatch, misframed reads, transport corruption.

Common situations: Rolling upgrades mixing packet-header layouts, NIC/TCP offload corruption, reading from a mirror DataNode that aborted mid-stream, fuzzed or intercepted connections.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/806ae00f51d6a6b0. Report an issue: GitHub.