apache/hadoop · critical · IOException

Invalid payload length

Error message

Invalid payload length ${payloadLen}

What it means

PacketReceiver.readNextPacket() reads the 4-byte payload length prefix of a data-transfer packet; values below 4 bytes (Ints.BYTES) are impossible because the payload length includes its own 4-byte field, so the stream is corrupt or misframed. Thrown as IOException on the reading side (client BlockReader or DataNode mirror receiver).

Solutions

1. Verify all pipeline nodes and the client run compatible Hadoop versions (rolling-upgrade compatibility rules).
2. Check DataNode logs around the failure; retry the read/write — DFSClient will choose another replica.
3. Inspect network hardware/middleboxes on the data path if corruption recurs cluster-wide.
4. Capture the socket (tcpdump) to confirm where framing diverges if the issue persists.

try {
  receiver.readNextPacket();
} catch (IOException e) {
  if (e.getMessage().contains("Invalid payload length")) {
    // framing corrupt — abort this stream; upper layer retries on another replica
    throw new StreamCorruptException(e.getMessage(), e);
  }
  throw e;
}

When it happens

Trigger: A misaligned read on the data-transfer socket: earlier bytes consumed incorrectly, a peer speaking a different protocol version, memory/disk corruption producing garbage length fields, or a truncated first packet after a protocol error reply instead of data.

Common situations: Version mismatch between client and DataNode packet formats, TCP streams corrupted by faulty hardware or middleboxes, DataNode returning an out-of-band error before the pipeline stalled, or downstream mirror DataNodes with mismatched versions during rolling upgrades.

Related errors

• BlockReader: error in packet header
• Invalid header length
• Invalid packet: data length in packet header exceeds data…
• Expected empty end-of-read packet! Header
• Incorrect value for packet payload size

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/4821c92dd2ea3b58. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PacketReceiver.java:151

    // HLEN:      Header length
    //            = length(HEADER)
    //
    // HEADER:    the actual packet header fields, encoded in protobuf
    // CHECKSUMS: the crcs for the data chunk. May be missing if
    //            checksums were not requested
    // DATA       the actual block data
    Preconditions.checkState(curHeader == null || !curHeader.isLastPacketInBlock());

    curPacketBuf.clear();
    curPacketBuf.limit(PacketHeader.PKT_LENGTHS_LEN);
    doReadFully(ch, in, curPacketBuf);
    curPacketBuf.flip();
    int payloadLen = curPacketBuf.getInt();

    if (payloadLen < Ints.BYTES) {
      // The "payload length" includes its own length. Therefore it
      // should never be less than 4 bytes
      throw new IOException("Invalid payload length " +
          payloadLen);
    }
    int dataPlusChecksumLen = payloadLen - Ints.BYTES;
    int headerLen = curPacketBuf.getShort();
    if (headerLen < 0) {
      throw new IOException("Invalid header length " + headerLen);
    }

    LOG.trace("readNextPacket: dataPlusChecksumLen={}, headerLen={}",
        dataPlusChecksumLen, headerLen);

    // Sanity check the buffer size so we don't allocate too much memory
    // and OOME.
    int totalLen = payloadLen + headerLen;
    if (totalLen < 0 || totalLen > MAX_PACKET_SIZE) {
      throw new IOException("Incorrect value for packet payload size: " +
                            payloadLen);
    }

View on GitHub (pinned to 2add963021)