apache/hadoop · critical · IOException

Invalid payload length ${payloadLen}

Error message

Invalid payload length ${payloadLen}

What it means

PacketReceiver.readNextPacket() reads the 4-byte payload length prefix of a data-transfer packet; values below 4 bytes (Ints.BYTES) are impossible because the payload length includes its own 4-byte field, so the stream is corrupt or misframed. Thrown as IOException on the reading side (client BlockReader or DataNode mirror receiver).

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PacketReceiver.java:151

    // HLEN:      Header length
    //            = length(HEADER)
    //
    // HEADER:    the actual packet header fields, encoded in protobuf
    // CHECKSUMS: the crcs for the data chunk. May be missing if
    //            checksums were not requested
    // DATA       the actual block data
    Preconditions.checkState(curHeader == null || !curHeader.isLastPacketInBlock());

    curPacketBuf.clear();
    curPacketBuf.limit(PacketHeader.PKT_LENGTHS_LEN);
    doReadFully(ch, in, curPacketBuf);
    curPacketBuf.flip();
    int payloadLen = curPacketBuf.getInt();

    if (payloadLen < Ints.BYTES) {
      // The "payload length" includes its own length. Therefore it
      // should never be less than 4 bytes
      throw new IOException("Invalid payload length " +
          payloadLen);
    }
    int dataPlusChecksumLen = payloadLen - Ints.BYTES;
    int headerLen = curPacketBuf.getShort();
    if (headerLen < 0) {
      throw new IOException("Invalid header length " + headerLen);
    }

    LOG.trace("readNextPacket: dataPlusChecksumLen={}, headerLen={}",
        dataPlusChecksumLen, headerLen);

    // Sanity check the buffer size so we don't allocate too much memory
    // and OOME.
    int totalLen = payloadLen + headerLen;
    if (totalLen < 0 || totalLen > MAX_PACKET_SIZE) {
      throw new IOException("Incorrect value for packet payload size: " +
                            payloadLen);
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Verify all pipeline nodes and the client run compatible Hadoop versions (rolling-upgrade compatibility rules).
2. Check DataNode logs around the failure; retry the read/write — DFSClient will choose another replica.
3. Inspect network hardware/middleboxes on the data path if corruption recurs cluster-wide.
4. Capture the socket (tcpdump) to confirm where framing diverges if the issue persists.

try {
  receiver.readNextPacket();
} catch (IOException e) {
  if (e.getMessage().contains("Invalid payload length")) {
    // framing corrupt — abort this stream; upper layer retries on another replica
    throw new StreamCorruptException(e.getMessage(), e);
  }
  throw e;
}

When it happens

Trigger: A misaligned read on the data-transfer socket: earlier bytes consumed incorrectly, a peer speaking a different protocol version, memory/disk corruption producing garbage length fields, or a truncated first packet after a protocol error reply instead of data.

Common situations: Version mismatch between client and DataNode packet formats, TCP streams corrupted by faulty hardware or middleboxes, DataNode returning an out-of-band error before the pipeline stalled, or downstream mirror DataNodes with mismatched versions during rolling upgrades.

Related errors

• BlockReader: error in packet header {}
• Invalid header length ${headerLen}
• Invalid packet: data length in packet header exceeds data le
• Expected empty end-of-read packet! Header: {}
• Incorrect value for packet payload size: ${payloadLen}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/4821c92dd2ea3b58. Report an issue: GitHub.